Robots.txt Tester

How to use this robots.txt tester

This robots.txt checker answers one question quickly: is a URL blocked by robots.txt for a given crawler, and which line decides it?

1. Choose Fetch from site and enter a domain. The tool requests /robots.txt from that host and shows the HTTP status, size, redirects and what the status means for crawling. To check a draft before you upload it, choose Paste robots.txt.
2. Pick a crawler: Googlebot, Bingbot, one of the AI crawlers, or any custom user-agent token.
3. Enter URLs or paths, one per line. Each gets an Allowed or Blocked result, the rule that decided it with its line number, and the group of rules that was used.
4. Edit the file to try a fix; results update as you type. The warnings list works as a robots.txt validator for typos, unsupported fields and malformed lines.

Google retired the robots.txt Tester in Search Console in November 2023 and replaced it with the robots.txt report, which shows the robots.txt files Google found for your site, when it last crawled them, and any warnings or errors. For one URL on a site you own, the URL Inspection tool shows whether Google is allowed to crawl it.

Robots.txt syntax essentials

A robots.txt file is UTF-8 plain text at the root of a host. Each line is field: value, and anything after # is a comment. Rules come in groups: one or more User-agent lines followed by Allow and Disallow rules. Sitemap lines stand on their own and apply to the whole file.

User-agent: *
Disallow: /cart/
Disallow: /*?sessionid=
Allow: /cart/help

Sitemap: https://example.com/sitemap.xml

Field names and user-agent values are case-insensitive, but paths are case-sensitive and must start with /. An empty Disallow: blocks nothing. The sitemap URL must be absolute; if you need a sitemap to point to, the Sitemap Generator builds one from a list of URLs.

How Google decides between conflicting rules

Google compares the path and query string of the URL with every rule in the group. The longest matching rule wins, measured by the length of its path. When an Allow and a Disallow rule are equally long, Google uses the less restrictive one, so Allow wins the tie. The order of rules in the file does not matter. These examples come from Google's documentation:

User-agent groups and specificity

A crawler obeys exactly one group: the one whose user-agent matches it most specifically. Googlebot-News uses a googlebot-news group if there is one, then a googlebot group, and only then *. Googlebot-Image and Googlebot-Video fall back to Googlebot in the same way, and Apple says Applebot follows Googlebot rules when a file does not mention Applebot. Storebot-Google has no such fallback.

If several groups name the same crawler, their rules are merged. A named group is never combined with the * group, though: once you add a User-agent: Googlebot section, Googlebot ignores every rule under *, so repeat any rules it still needs. Version numbers are ignored, so Googlebot/2.1 is read as googlebot.

Wildcards: * and $

* matches any sequence of characters, including none, and $ anchors the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf but not /file.pdf?download=1, because the query string is part of what is matched. A trailing * changes nothing: /fish* equals /fish. To match a literal asterisk or dollar sign, write it percent-encoded as %2A or %24.

What robots.txt does not do

Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google results if other pages link to it, usually without a description, because Google cannot read the page. To keep a page out of search results, let it be crawled and add a noindex robots meta tag or an X-Robots-Tag HTTP header. You can confirm the header is actually sent with the HTTP Header Checker. Google stopped honouring noindex lines inside robots.txt in 2019.

Common mistakes

• Blocking CSS and JavaScript. Google renders pages much like a browser. If the files a page needs are disallowed, Google may not see the page the way visitors do.
• Trailing-slash errors. Disallow: /private also blocks /private-offers and /private.html, while Disallow: /private/ does not block /private itself.
• Wrong case. Disallow: /Admin/ does not block /admin/.
• Shipping the staging file. A User-agent: * plus Disallow: / file that protected a staging site can block the whole live site after a deploy. Fetch the live file after each release, and protect staging with a password instead.

AI crawler tokens

Each AI company publishes its own tokens, and each is controlled separately, so you can allow search crawlers while opting out of model training:

• OpenAI: GPTBot collects training data, OAI-SearchBot crawls for ChatGPT search, and ChatGPT-User fetches pages when a user asks; OpenAI says robots.txt rules may not apply to those user-initiated visits.
• Anthropic: ClaudeBot collects training data, Claude-SearchBot crawls for search, and Claude-User fetches pages for user requests.
• Google: Google-Extended is not a separate crawler. It controls whether content Google crawls may be used to train Gemini models and for grounding, and it does not affect Google Search.
• Apple: Applebot-Extended controls use for training Apple's foundation models. It does not crawl, and pages that disallow it can still appear in Apple's search results.
• Common Crawl: CCBot builds the open Common Crawl web archive.

How 4xx and 5xx robots.txt responses are treated

The status code of the robots.txt file matters as much as its contents. Google follows at least five redirects and then treats the file as not found. Any 4xx response except 429, including 401 and 403, means there is no robots.txt, so everything may be crawled. A 429, a 5xx or a network error such as a timeout or DNS failure is treated as a server error: Google stops crawling the site for the first 12 hours, then relies on its last good copy for up to 30 days. This tester applies the same logic to the status of a fetched file and tells you which case you are in.