Robots.txt Tester

How robots.txt matching works, step by step

A crawler that respects robots.txt goes through the same sequence for every URL, and this tool performs it in the same order.

1. Read the file line by line. Each line is a field name, a colon and a value. Field names are case-insensitive, so DISALLOW and disallow are the same field, and anything after # is a comment. Lines the protocol does not define are skipped; this tool lists them so typos stand out.
2. Build groups. One or more User-agent lines in a row open a group, and the Allow and Disallow lines after them belong to it. The next User-agent line after a rule starts a new group. Blank lines, comments, Sitemap and Crawl-delay lines do not break a run of User-agent lines.
3. Pick the crawler's group. The crawler compares its product token with each User-agent value, ignoring case. If several groups name it, their rules are merged into one. If none do, it uses the * group, and if there is no * group either, nothing is restricted. A crawler that finds its own group ignores the * group entirely — the two are never combined.
4. Prepare the path. Only the path and query are compared, octet by octet and case-sensitively, starting from the first slash. Characters outside ASCII are percent-encoded as UTF-8, and needless escapes of letters, digits, hyphen, period, underscore and tilde are decoded, so /café and /caf%C3%A9 compare equal, as do /%62az and /baz.
5. Collect every matching rule. A rule matches when the path begins with its pattern. An asterisk stands for any run of characters, slashes included, and a dollar sign at the very end requires the path to stop there.
6. Let the longest rule win. Among the matching rules, the one whose pattern has the most octets decides. When an Allow and a Disallow are the same length, Allow wins. When nothing matches, the URL is allowed. The file /robots.txt itself is always allowed.

The order of rules within a group never changes the outcome — only a rule's length and the group it sits in do. That surprises people who learned the rules from the 1990s drafts, where the first matching line won.

Worked examples

The example loaded in the form is a small shop's robots.txt. These results come from the same code that runs in the form.

Here is the full grid for the example file — five crawlers, eight URLs, with the line that decided each cell.

Wildcards, anchors and percent-encoding

Only two characters are special inside a pattern. The asterisk matches zero or more characters of any kind, so /*.pdf reaches PDFs in every folder. The dollar sign means "the URL ends here" only when it is the last character of the pattern; anywhere else it is an ordinary character. To match a literal asterisk or dollar sign in a URL, write it percent-encoded as %2A or %24.

Patterns should start with a slash or an asterisk. A rule such as Disallow: private/ can never match, because every path begins with /; the tester flags it. An empty Disallow: matches nothing and is the conventional way to say "everything is allowed", while Disallow: / blocks the whole site.

Escapes for reserved characters keep their meaning: %2F in a URL is not the same as /, and the tester compares them as written. Hex digits are compared in upper case, so %e3 and %E3 are equal. Matching is case-sensitive, so Disallow: /Private does not block /private; if your server answers both spellings, write a rule for each one that gets linked.

Why paste-and-test matters now

For close to ten years, the usual way to try a robots.txt edit was the tester built into Google Search Console. Google retired it in December 2023 and replaced it with a robots.txt report that lists the files Google has fetched for a property, when it last fetched them and any parsing problems. That report shows what Google sees on the live site, but it has no box for pasting a draft and trying URLs before you publish. The URL Inspection tool can still tell you whether one live URL is blocked.

The rules themselves are now settled. Google released its production robots.txt parser as open-source code in 2019, and the protocol was published as RFC 9309 in 2022. This tester implements the RFC text directly. Where Google documents extra behavior — such as letting Googlebot-Image fall back to a Googlebot group — the option is labeled so you can compare both readings.

AI crawler tokens: what blocking them does and does not do

Since 2023, many sites have edited robots.txt with AI crawlers in mind. Each company publishes the tokens it honors, and they do different jobs:

• GPTBot (OpenAI) and ClaudeBot (Anthropic) collect public pages that may be used to train models. Disallowing them asks those companies not to crawl your pages for that purpose.
• ChatGPT-User fetches a page when a person using ChatGPT asks for it. OpenAI says that because these visits are started by a user, robots.txt rules may not apply to them. OpenAI's separate OAI-SearchBot token governs appearance in ChatGPT search, and Anthropic likewise lists Claude-User and Claude-SearchBot alongside ClaudeBot.
• Google-Extended is a control token, not a crawler: no request ever arrives under that name. Google crawls with its usual Googlebot agents, and Google-Extended decides whether that content may be used to train Gemini models and to ground their answers. Google says it has no effect on inclusion or ranking in Google Search; AI features inside Search follow Googlebot's rules.
• Applebot-Extended works the same way for Apple: Applebot does the crawling, and the -Extended token only controls whether the content may train Apple's generative models.
• CCBot builds Common Crawl, a free public archive of the web from which many AI training datasets are drawn. PerplexityBot indexes pages for Perplexity's answer engine, and Bytespider is ByteDance's crawler.

What blocking does not do: it does not delete anything already collected, because robots.txt only governs future fetches. It is a request, not access control — a crawler that ignores the file is not stopped by it, and since the file is public, listing secret paths in it advertises them. It also does not remove pages from search: a URL that Googlebot may not crawl can still be indexed from links, usually without a description. To keep a page out of search, allow crawling and add a noindex robots meta tag or X-Robots-Tag header; for anything private, require a login.

One trap catches many AI rules. A crawler that has its own group ignores the * group, so adding a GPTBot group that disallows only /private/ quietly lifts every other * rule for GPTBot. With Disallow: /checkout/ under * and Disallow: /private/ under GPTBot, this tester reports /checkout/cart as allowed for GPTBot. Copy the * rules into every named group that should keep them. And never add Googlebot to an AI group to opt out of training — that removes the site from Google Search; Google-Extended is the token for that choice.

Limits of this tester

• It reads only what you paste. It does not fetch /robots.txt, so it cannot see redirects, HTTP status codes or outages. Under RFC 9309, a 4xx response for robots.txt lets crawlers fetch anything, while a 5xx error or an unreachable server means they must assume everything is disallowed.
• Each combination of scheme, host and port has its own file. Rules in one host's robots.txt say nothing about a subdomain or about the same host over another protocol.
• Reserved characters are compared as written: a literal : or / inside a query value does not equal %3A or %2F. Write rules the way the URLs appear in your links.
• The Googlebot-Image to Googlebot fallback follows Google's description of its own crawlers, so it applies only to tokens that begin with Googlebot-. Every other token is matched by its exact name, then the * group. Tick Exact token match only for the strict reading of Googlebot tokens too.
• Crawl-delay and Sitemap lines are reported, not applied. Misspelled fields such as Dissallow are flagged and ignored, although some crawlers accept them.
• Files over 500 KiB are cut at 500 KiB, the minimum RFC 9309 requires crawlers to read and the same limit Google documents.
• Robots.txt is voluntary, and crawlers may interpret edge cases differently. A result here tells you what a crawler that follows RFC 9309 would do.

Last updated October 11, 2026. Found a mistake? Tell us — corrections ship the same week.