robots.txt Tester

This robots.txt tester checks whether Googlebot, Bingbot, AI crawlers, or any bot you name may crawl a URL, and shows the exact line that decides it. Paste your robots.txt, list up to 1,000 URLs, and you get a results grid for every crawler, a line-by-line review of the file, and a robots.txt validator that flags syntax mistakes and can fix the common ones. Edit the file and the results update instantly, so you can test changes before you publish them. Everything runs in your browser.

How to test robots.txt

• Paste the file. Open yoursite.com/robots.txt, copy everything, and paste it in, or upload the file.
• Add URLs or paths, one per line. Full URLs and paths such as /blog/post/ both work.
• Choose crawlers. Googlebot, Googlebot-Image, Bingbot, and GPTBot are selected to start. Add others from the list, or type any crawler name or full user agent string.
• Select a result to see which group the crawler follows, every rule that matches, and the one that wins.

A replacement for Google’s robots.txt Tester

In November 2023, Google announced it was retiring the robots.txt Tester in Search Console and replacing it with the robots.txt report. The report shows which robots.txt files Google found, when it last fetched them, and any errors, and URL Inspection tells you whether one URL is blocked. Neither lets you edit a draft and test it before it goes live, or check many URLs and crawlers at once. This robots.txt checker does both, and it highlights the deciding line the way the old tester did.

The matching follows RFC 9309, the robots.txt standard, and Google’s open-source robots.txt parser, which is the code Google uses in production. Other crawlers follow the same standard, but small differences between them are possible.

How a crawler chooses its group

robots.txt is split into groups, each starting with one or more User-agent lines. A crawler follows only the most specific group that names it, and if no group does, it follows the * group. The two are never combined. That causes one of the most common mistakes: as soon as you add a group for Googlebot, Googlebot ignores everything in the * group, including rules such as Disallow: /wp-admin/. The checker warns you when a group skips rules from the * group.

Some crawlers have more than one token and fall back to a related group:

Crawler	Follows, in order
Googlebot-Image, Googlebot-Video, Googlebot-News	Its own group, then Googlebot, then *
Google-InspectionTool	Its own group, then Googlebot, then *
Storebot-Google, GoogleOther	Its own group, then *
AdsBot-Google, Mediapartners-Google	Only a group that names it; they ignore *
Bingbot	bingbot, then msnbot, then *
Applebot	Applebot, then Googlebot, then *
ClaudeBot	ClaudeBot, then the retired anthropic-ai and Claude-Web tokens, then *

Bot names are case-insensitive, and Google reads only the name at the start of the line, so Googlebot/2.1 counts as Googlebot. Wildcards don’t work in bot names: User-agent: *bot matches nothing.

How rules match a URL

Within a group, the robots.txt tester applies the same rules a crawler does:

• The longest matching rule wins, counted in characters. Allow: /private/press-kit/ beats Disallow: /private/.
• Allow wins a tie, because Google uses the least restrictive rule.
• * matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files but not /file.pdf?download=1.
• Paths are case-sensitive. Disallow: /Admin doesn’t block /admin.
• Rules match from the start of the path, including the query string. A rule without a leading / or * never matches.
• Non-ASCII characters are percent-encoded, so /café and /caf%C3%A9 are the same. Encoded characters aren’t decoded, though: /%62az doesn’t match /baz.
• An empty Disallow line blocks nothing, and /robots.txt itself is always allowed.

Google adds one special case: Allow: /folder/index.html also allows /folder/ itself. The tester includes it and tells you when it applies.

robots.txt syntax mistakes the validator finds

Google’s parser is forgiving: it accepts misspellings such as Dissallow, lines without a colon, and fields that only start with the right word. Other crawlers may not be as tolerant, so the robots.txt validator flags these and fixes them with one click:

• Misspelled fields and missing colons.
• Full URLs in Allow or Disallow lines, which never match. Use the path instead.
• Paths that don’t start with /.
• Version numbers or URLs in bot names.
• Rules before the first User-agent line, which crawlers ignore.
• Sitemap lines with a relative URL. Sitemap URLs must be complete, starting with https://.

It also warns about a * group that blocks the whole site, a blocked Googlebot or Bingbot, CSS and JavaScript that Googlebot can’t fetch, groups with no rules, and URLs on different hosts. Each robots.txt file only applies to its own host, so every subdomain needs its own.

Common robots.txt patterns to test

These patterns cover most sites. Paste them into the robots.txt checker with a few of your own URLs to confirm they do what you expect:

# Block a folder, but keep one page inside it
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Block internal search results and PDF files
Disallow: /*?s=
Disallow: /*.pdf$

Sitemap: https://www.example.com/sitemap.xml

Test both sides of every rule: a URL it should block, and a similar URL it shouldn’t, such as /wp-admin/admin-ajax.php next to /wp-admin/options.php. Mistakes with wildcards usually show up as a page blocked by accident, so it’s worth a moment to test robots.txt changes before they go live, because a wrong rule can stop crawling of a whole section.

What Google ignores in robots.txt

Google supports four fields: User-agent, Allow, Disallow, and Sitemap. It ignores Crawl-delay (Bing honours it), Noindex, and Host, and Yandex reads Clean-param. Google reads the first 500 KiB of the file and ignores the rest, and it expects UTF-8 text. The tester shows the file size against that limit and marks any lines past it.

robots.txt doesn’t remove pages from search

Blocking a page stops crawling, not indexing. If other pages link to a blocked URL, Google can still index it and show the address without a description. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header, so Google can see the instruction. Never block a page and add noindex to it at the same time.

Status codes and caching

How your server answers matters as much as the file. A 4xx response, except 429, means “no robots.txt”, so Google may crawl everything. A 5xx response or a 429 makes Google pause crawling for 12 hours, then use its last good copy for up to 30 days. Google usually caches robots.txt for up to 24 hours, so changes can take a day to apply. After publishing, you can ask Google to fetch the file again from the robots.txt report in Search Console.

Testing AI crawlers

The tester includes GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, and other AI crawlers, and notes the ones whose companies say they may ignore robots.txt. Many files still block retired tokens such as anthropic-ai; the tester shows that ClaudeBot follows those rules, so you can see what each AI company actually reads. To write rules for them, use the AI Crawler robots.txt Generator, then paste the result here to test it. To check how your pages look in search, try the SERP Snippet Preview and the Heading Structure Checker.