# Robots.txt Tester

Paste a robots.txt file, enter a URL and a crawler name, and see whether that crawler may fetch the page — with the group it uses, the rule that decides and its line number. Ten common search and AI crawlers are checked at the same time.

- Runs in your browser
- Nothing fetched or uploaded
- RFC 9309 matching
- 10 search & AI crawlers

## Quick answer

Under RFC 9309 the **longest matching rule wins**, and Allow wins a tie. With `Disallow: /shop/` and `Allow: /shop/catalog/`, `/shop/catalog/item-1` is allowed and `/shop/cart` is blocked. A crawler uses only the group that names it, falling back to `User-agent: *` when none does.

## About the Robots.txt Tester

This tester answers one question: may a given crawler fetch a given URL under a given robots.txt file? Paste the file, type a URL or path and a crawler name, and it shows the verdict, the group of rules that crawler would follow, the exact line that decided the outcome, and every other rule that matched along the way. It follows RFC 9309, the Robots Exclusion Protocol the IETF published in September 2022, which turned almost thirty years of informal practice into a written standard.

Nothing is fetched. The file you paste stays in this browser tab, so you can test a draft before it goes live, check a staging site that is not public, or reproduce a crawler complaint without touching the production file. The same URL is also checked against ten common search and AI crawlers at once, so one table shows which of them your rules actually reach.

## How to use it

Paste the full contents of a robots.txt file into the large box — from the live site's /robots.txt, from your repository, or from the CMS setting that generates it. The tester reads it exactly as written, including comments and blank lines, and every line number in the results refers to the text in the box.

Enter the URL you want to check. A full address such as `https://shop.example/search?q=boots` works, and so does a bare path with its query string, like `/search?q=boots`. The scheme, host and anything after `#` are dropped, because robots.txt rules only ever see the path and query. A link that starts with two slashes, such as `//shop.example/checkout/cart`, is read the way a browser reads it: `shop.example` is the host, and the path tested is `/checkout/cart`.

Type the crawler's product token: the short name used on User-agent lines, such as Googlebot or GPTBot — not the long User-Agent header the bot sends. Capitalization does not matter. Leave **Exact token match only** unticked to let a Google sub-crawler like Googlebot-Image fall back to a Googlebot group, the way Google describes its own crawlers; tick it to see what a crawler that accepts only its exact name would do. The tester applies no fallback to other names, so here `Applebot-Extended` is not covered by an `Applebot` group and uses the `*` group instead.

Results update as you type. The verdict at the top names the deciding rule and its line. The table repeats the check for ten widely seen crawlers, with your crawler highlighted when it is one of them. **Effective group** shows the merged rules your crawler obeys, ready to copy, and **Show the math** lists each matching rule with its length so you can see why the longest one won. The robots.txt text is never written into the page address, so a copied link carries only the URL and the crawler name.

## Use cases

- **Checking an edit before you deploy it.** Paste the changed file and try the URLs that matter — product pages, category pages, the home page — so a stray `Disallow: /` never takes a site out of search.
- **Explaining a "blocked by robots.txt" report.** When a search console or a crawl tool flags a URL, find the line responsible and see whether a narrower Allow rule would fix it without opening the whole directory.
- **Deciding what AI crawlers may read.** See which AI tokens your current file actually reaches, and whether a group written for one bot accidentally shields or exposes another.
- **Auditing tracking parameters and faceted navigation.** Confirm that wildcard rules such as `/*?sessionid=` block the parameter combinations you meant, and only those.
- **Reviewing a pull request.** Teams that keep robots.txt in version control can paste the proposed file and check a short list of critical URLs as part of code review.
- **Learning the rules.** The step-by-step output shows how grouping, merging, wildcards and the longest-match rule interact, which is quicker than reading the RFC cold.

## How robots.txt matching works, step by step

A crawler that respects robots.txt goes through the same sequence for every URL, and this tool performs it in the same order.

1. **Read the file line by line.** Each line is a field name, a colon and a value. Field names are case-insensitive, so `DISALLOW` and `disallow` are the same field, and anything after `#` is a comment. Lines the protocol does not define are skipped; this tool lists them so typos stand out.
2. **Build groups.** One or more User-agent lines in a row open a group, and the Allow and Disallow lines after them belong to it. The next User-agent line after a rule starts a new group. Blank lines, comments, Sitemap and Crawl-delay lines do not break a run of User-agent lines.
3. **Pick the crawler's group.** The crawler compares its product token with each User-agent value, ignoring case. If several groups name it, their rules are merged into one. If none do, it uses the `*` group, and if there is no `*` group either, nothing is restricted. A crawler that finds its own group ignores the `*` group entirely — the two are never combined.
4. **Prepare the path.** Only the path and query are compared, octet by octet and case-sensitively, starting from the first slash. Characters outside ASCII are percent-encoded as UTF-8, and needless escapes of letters, digits, hyphen, period, underscore and tilde are decoded, so `/café` and `/caf%C3%A9` compare equal, as do `/%62az` and `/baz`.
5. **Collect every matching rule.** A rule matches when the path begins with its pattern. An asterisk stands for any run of characters, slashes included, and a dollar sign at the very end requires the path to stop there.
6. **Let the longest rule win.** Among the matching rules, the one whose pattern has the most octets decides. When an Allow and a Disallow are the same length, Allow wins. When nothing matches, the URL is allowed. The file `/robots.txt` itself is always allowed.

The order of rules within a group never changes the outcome — only a rule's length and the group it sits in do. That surprises people who learned the rules from the 1990s drafts, where the first matching line won.

## Worked examples

The example loaded in the form is a small shop's robots.txt. These results come from the same code that runs in the form.

**Longest match.** For Googlebot and `/search/about-us`, two rules in Googlebot's group match: `Disallow: /search` on line 11 (7 octets) and `Allow: /search/about` on line 12 (13 octets). The longer Allow wins, so the page is allowed even though everything else under /search is blocked.

**Merged groups.** The file names Googlebot twice, on lines 9 and 23 — the second time in lower case. Both groups merge, so `/drafts/fall-sale` is blocked for Googlebot by line 24\. Bingbot shares the first group but not the second, so for Bingbot the same URL is allowed.

**The catch-all group.** PerplexityBot has no group of its own, so it falls back to `*`. `/checkout/cart` is blocked by `Disallow: /checkout/` on line 3, but `/checkout/help/returns` is allowed by the longer `Allow: /checkout/help` on line 4.

**Wildcards and anchors.** Under the same group, `/menu.pdf` is blocked by `/*.pdf$` (line 6), yet `/menu.pdf?v=2` is allowed: the `$` requires the path to end right after .pdf, and the query string counts as part of the path. Likewise `/*?sessionid=` blocks `/shop?sessionid=42` but not `/shop?page=2&sessionid=42`, where the parameter is not first; `/*sessionid=` would catch both.

Here is the full grid for the example file — five crawlers, eight URLs, with the line that decided each cell.

| URL                       | Googlebot             | Bingbot               | GPTBot                | Google-Extended       | PerplexityBot        |
| ------------------------- | --------------------- | --------------------- | --------------------- | --------------------- | -------------------- |
| /search/about-us          | Allowed · line 12     | Allowed · line 12     | **Blocked** · line 18 | **Blocked** · line 21 | Allowed · no rule    |
| /search?q=boots           | **Blocked** · line 11 | **Blocked** · line 11 | **Blocked** · line 18 | Allowed · no rule     | Allowed · no rule    |
| /drafts/fall-sale         | **Blocked** · line 24 | Allowed · no rule     | **Blocked** · line 18 | Allowed · no rule     | Allowed · no rule    |
| /checkout/help/returns    | Allowed · no rule     | Allowed · no rule     | **Blocked** · line 18 | Allowed · no rule     | Allowed · line 4     |
| /menu.pdf                 | **Blocked** · line 13 | **Blocked** · line 13 | **Blocked** · line 18 | Allowed · no rule     | **Blocked** · line 6 |
| /menu.pdf?v=2             | Allowed · no rule     | Allowed · no rule     | **Blocked** · line 18 | Allowed · no rule     | Allowed · no rule    |
| /shop?sessionid=42        | Allowed · no rule     | Allowed · no rule     | **Blocked** · line 18 | Allowed · no rule     | **Blocked** · line 5 |
| /shop?page=2&sessionid=42 | Allowed · no rule     | Allowed · no rule     | **Blocked** · line 18 | Allowed · no rule     | Allowed · no rule    |

## Wildcards, anchors and percent-encoding

Only two characters are special inside a pattern. The asterisk matches zero or more characters of any kind, so `/*.pdf` reaches PDFs in every folder. The dollar sign means "the URL ends here" only when it is the last character of the pattern; anywhere else it is an ordinary character. To match a literal asterisk or dollar sign in a URL, write it percent-encoded as `%2A` or `%24`.

Patterns should start with a slash or an asterisk. A rule such as `Disallow: private/` can never match, because every path begins with `/`; the tester flags it. An empty `Disallow:` matches nothing and is the conventional way to say "everything is allowed", while `Disallow: /` blocks the whole site.

Escapes for reserved characters keep their meaning: `%2F` in a URL is not the same as `/`, and the tester compares them as written. Hex digits are compared in upper case, so `%e3` and `%E3` are equal. Matching is case-sensitive, so `Disallow: /Private` does not block `/private`; if your server answers both spellings, write a rule for each one that gets linked.

## Why paste-and-test matters now

For close to ten years, the usual way to try a robots.txt edit was the tester built into Google Search Console. Google retired it in December 2023 and replaced it with a robots.txt report that lists the files Google has fetched for a property, when it last fetched them and any parsing problems. That report shows what Google sees on the live site, but it has no box for pasting a draft and trying URLs before you publish. The URL Inspection tool can still tell you whether one live URL is blocked.

The rules themselves are now settled. Google released its production robots.txt parser as open-source code in 2019, and the protocol was published as RFC 9309 in 2022\. This tester implements the RFC text directly. Where Google documents extra behavior — such as letting Googlebot-Image fall back to a Googlebot group — the option is labeled so you can compare both readings.

## AI crawler tokens: what blocking them does and does not do

Since 2023, many sites have edited robots.txt with AI crawlers in mind. Each company publishes the tokens it honors, and they do different jobs:

- **GPTBot** (OpenAI) and **ClaudeBot** (Anthropic) collect public pages that may be used to train models. Disallowing them asks those companies not to crawl your pages for that purpose.
- **ChatGPT-User** fetches a page when a person using ChatGPT asks for it. OpenAI says that because these visits are started by a user, robots.txt rules may not apply to them. OpenAI's separate OAI-SearchBot token governs appearance in ChatGPT search, and Anthropic likewise lists Claude-User and Claude-SearchBot alongside ClaudeBot.
- **Google-Extended** is a control token, not a crawler: no request ever arrives under that name. Google crawls with its usual Googlebot agents, and Google-Extended decides whether that content may be used to train Gemini models and to ground their answers. Google says it has no effect on inclusion or ranking in Google Search; AI features inside Search follow Googlebot's rules.
- **Applebot-Extended** works the same way for Apple: Applebot does the crawling, and the -Extended token only controls whether the content may train Apple's generative models.
- **CCBot** builds Common Crawl, a free public archive of the web from which many AI training datasets are drawn. **PerplexityBot** indexes pages for Perplexity's answer engine, and **Bytespider** is ByteDance's crawler.

What blocking does not do: it does not delete anything already collected, because robots.txt only governs future fetches. It is a request, not access control — a crawler that ignores the file is not stopped by it, and since the file is public, listing secret paths in it advertises them. It also does not remove pages from search: a URL that Googlebot may not crawl can still be indexed from links, usually without a description. To keep a page out of search, allow crawling and add a noindex robots meta tag or X-Robots-Tag header; for anything private, require a login.

One trap catches many AI rules. A crawler that has its own group ignores the `*` group, so adding a GPTBot group that disallows only /private/ quietly lifts every other `*` rule for GPTBot. With `Disallow: /checkout/` under `*` and `Disallow: /private/` under GPTBot, this tester reports `/checkout/cart` as allowed for GPTBot. Copy the `*` rules into every named group that should keep them. And never add Googlebot to an AI group to opt out of training — that removes the site from Google Search; Google-Extended is the token for that choice.

## Limits of this tester

- It reads only what you paste. It does not fetch /robots.txt, so it cannot see redirects, HTTP status codes or outages. Under RFC 9309, a 4xx response for robots.txt lets crawlers fetch anything, while a 5xx error or an unreachable server means they must assume everything is disallowed.
- Each combination of scheme, host and port has its own file. Rules in one host's robots.txt say nothing about a subdomain or about the same host over another protocol.
- Reserved characters are compared as written: a literal `:` or `/` inside a query value does not equal `%3A` or `%2F`. Write rules the way the URLs appear in your links.
- The Googlebot-Image to Googlebot fallback follows Google's description of its own crawlers, so it applies only to tokens that begin with Googlebot-. Every other token is matched by its exact name, then the `*` group. Tick **Exact token match only** for the strict reading of Googlebot tokens too.
- Crawl-delay and Sitemap lines are reported, not applied. Misspelled fields such as `Dissallow` are flagged and ignored, although some crawlers accept them.
- Files over 500 KiB are cut at 500 KiB, the minimum RFC 9309 requires crawlers to read and the same limit Google documents.
- Robots.txt is voluntary, and crawlers may interpret edge cases differently. A result here tells you what a crawler that follows RFC 9309 would do.

## Sources

- [RFC 9309 — Robots Exclusion Protocol (IETF)](https://www.rfc-editor.org/rfc/rfc9309.html) — checked 2026-10-10

## Frequently asked questions

Does the order of rules in robots.txt matter?

Not within a group. Under RFC 9309 the matching rule with the longest pattern wins wherever it appears, and an Allow beats a Disallow of the same length. Order only matters for grouping: a User-agent line that comes after a rule starts a new group.

Why is my URL blocked when I wrote an Allow rule for it?

Usually for one of two reasons. A longer Disallow also matches the URL and outranks your Allow, or the Allow sits in a group the crawler does not use — for example in the \* group when the crawler has a group of its own. The Show the math list names every matching rule with its length and line.

Does Googlebot follow the rules in the User-agent: \* group?

Only when no group names Googlebot. As soon as a Googlebot group exists, the \* group is ignored for Googlebot completely, so repeat any \* rules that should also apply to it. The same goes for every crawler.

How do I block AI crawlers without affecting Google Search?

Add a group listing the AI tokens you want to exclude — for example GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended — followed by Disallow: /, and leave Googlebot and Bingbot out of it. Paste the result here and check that Googlebot still shows Allowed in the crawler table.

Is a page blocked by robots.txt removed from Google?

Not necessarily. A Disallow stops crawling, not indexing, so a blocked URL can still appear in results if other pages link to it, normally without a description. To remove a page, let it be crawled and serve a noindex robots meta tag or X-Robots-Tag header.

What does an empty Disallow line mean?

Disallow: with nothing after it matches no URL, so it allows everything for that group. It is the opposite of Disallow: /, which blocks the whole site. A group with no rules at all behaves the same as an empty Disallow.

Is the robots.txt I paste sent anywhere?

No. Parsing and matching run in this page's JavaScript, the file is not uploaded, and nothing is fetched from your site. Only the URL and crawler name are written into the page address, so a shared link never contains the file.

**Disclaimer.** Results follow RFC 9309 as written, with Google's documented token fallback as an option; individual crawlers may interpret robots.txt differently, and none is obliged to obey it.

Last updated October 11, 2026. Found a mistake? [Tell us](https://17x7.com/contact/) — corrections ship the same week.

---

Source: https://17x7.com/tools/robots-txt-tester/ · Last updated: 2026-10-11