AI Crawler robots.txt Generator

This AI crawler robots.txt generator helps you block AI crawlers from training on your content while choosing, bot by bot, whether to stay visible in AI search answers. Paste your current robots.txt, pick a policy, and you get a complete file that keeps your existing rules and sitemaps, replaces any old AI rules, and can be tested URL by URL before you upload it. Everything runs in your browser.

How to block AI crawlers with robots.txt

• Paste your current robots.txt. Open yoursite.com/robots.txt, copy everything, and paste it into the first box. Bots it already blocks are switched on for you.
• Choose a policy. Block AI training, block all AI bots, allow them, or choose bot by bot.
• Review the checks. They flag syntax errors, blocked search engines, and bots that may ignore robots.txt.
• Test a URL, then copy or download the new file and replace the old one.

Training, search, and user bots do different jobs

Most AI companies now run more than one bot, and blocking the wrong one has real consequences. The generator groups them by what they do:

Type	What it does	What blocking it means
AI training	Collects pages to train AI models	Your new content isn’t used for future training. Content already collected isn’t removed.
AI search and answers	Indexes pages so assistants can find, quote, and link to them	Your pages stop appearing as sources in that assistant’s answers.
Visits requested by users	Fetches a page when someone asks an assistant about it	Several companies say these visits may ignore robots.txt, because a person made the request.

That’s why Block AI training is a common middle ground: it opts out of model training but leaves ChatGPT search, Perplexity, and Claude free to find and cite your pages.

Which policy should you choose?

There’s no single right answer. It depends on how your site earns its traffic:

• Publishers and creators whose content is the product often block AI crawlers used for training, and sometimes AI search bots too, accepting fewer AI referrals.
• Businesses, stores, and SaaS sites usually want to be recommended in AI answers, so they block training at most and allow search and user bots.
• Documentation and open projects often allow everything, because being quoted accurately helps their users.

Whichever you choose, this robots.txt generator writes the rules for you, and you can change them at any time.

The AI bots in this generator

Each bot is listed under the exact user-agent token its company documents, with a link to that documentation. The list was checked against each company’s own pages in October 2026, and it’s updated as companies add or rename bots.

Company	Training	Search and answers	User requests
OpenAI	GPTBot	OAI-SearchBot	ChatGPT-User
Anthropic	ClaudeBot	Claude-SearchBot	Claude-User
Google	Google-Extended (token)	Uses Googlebot	Google-Agent, Google-GeminiNotebook
Apple	Applebot-Extended (token)	Uses Applebot	–
Meta	Meta-ExternalAgent	Meta-WebIndexer	Meta-ExternalFetcher
Amazon	Amazonbot	Amzn-SearchBot	Amzn-User
Mistral AI	MistralAI-Training	MistralAI-Index	MistralAI-User
Perplexity	–	PerplexityBot	Perplexity-User
Others	CCBot, AI2Bot, Bytespider	DuckAssistBot	–

Google-Extended and Applebot-Extended aren’t crawlers. They’re tokens that tell Google and Apple whether pages their normal crawlers fetch may be used for AI. You can add any other bot under More options.

How to block GPTBot

To block GPTBot, OpenAI’s training crawler, a robots.txt file needs just two lines:

User-agent: GPTBot
Disallow: /

Blocking GPTBot doesn’t remove you from ChatGPT search, which uses OAI-SearchBot. OpenAI says a page blocked from OAI-SearchBot can still appear as a plain navigation link, and that robots.txt rules may not apply to ChatGPT-User, which visits pages users ask about. The same pattern holds for Anthropic, Meta, Amazon, and Mistral, so decide on each bot type separately.

Keep your existing robots.txt rules

A bot that has its own group in robots.txt ignores the User-agent: * group completely. So if you allow an AI bot into some folders only, it could also crawl folders you’ve closed to everyone. When you choose Only some folders, the generator copies your * rules into the AI groups to prevent that.

It also removes old rules for the bots you manage, including rules it added before, so you never end up with conflicting groups. It keeps everything else exactly as it was. Retired tokens such as anthropic-ai are replaced with ClaudeBot, which Anthropic says follows rules written for them.

Google AI Overviews and Bing Copilot

Google-Extended controls whether your pages train Gemini models and ground answers in Gemini apps and Vertex AI. Google says it doesn’t affect Google Search or rankings. AI Overviews and AI Mode are part of Search and use Googlebot, so robots.txt can only remove you from them by removing you from Google entirely. To limit what they show, use the nosnippet, data-nosnippet, or max-snippet controls.

Microsoft Copilot uses Bingbot, which has no separate AI token. Bing’s noarchive meta tag keeps a page out of Copilot answers and Microsoft model training, and nocache limits answers to the URL, title, and snippet.

Testing your AI bots robots.txt

Before you upload your AI bots robots.txt, enter a page URL or path under Test a URL to see whether each bot may crawl it, and which line decides. The test follows RFC 9309, the robots.txt standard: the longest matching rule wins, Allow wins a tie, * matches any characters, $ marks the end of the URL, and paths are case-sensitive.

The checks also flag common mistakes: rules before any User-agent line, misspelled fields, full URLs in Disallow lines, version numbers in bot names, a blocked Googlebot, and files over the 500 KiB that crawlers read.

What robots.txt can’t do

• It’s a request, not a lock. Reputable bots follow it, but nothing forces them to. To enforce a block, add a firewall or CDN rule for the bot’s user agent.
• It doesn’t remove past data. Blocking a training crawler stops future collection only.
• User-requested visits may ignore it. Google, Perplexity, Meta, Amazon, and OpenAI say some of their user-triggered fetchers may not follow robots.txt.
• Newer signals are optional. Cloudflare’s Content Signals Policy adds a Content-Signal line that states your preferences for search, AI input, and AI training. The IETF is still drafting a standard vocabulary for AI preferences.

Is blocking AI crawlers legally meaningful?

In the EU, the copyright rules on text and data mining let rightholders reserve their rights in a machine-readable way, and robots.txt is a common way to do that. Companies that signed the EU’s 2025 code of practice for general-purpose AI models commit to crawlers that read and follow robots.txt. Elsewhere, robots.txt is mainly a technical convention. This is general information, not legal advice.

Installing your new robots.txt

robots.txt must sit at the root of your domain, at yoursite.com/robots.txt. On WordPress, paste the file into Rank Math (General Settings › Edit robots.txt) or Yoast (Tools › File editor). A physical robots.txt file in your site’s root folder overrides both. On Shopify, edit the robots.txt.liquid template. If Cloudflare’s managed robots.txt is turned on, it adds its own rules in front of yours, so check the live file after publishing.

Crawlers cache robots.txt, usually for up to a day, so changes take a little time to apply. Once the file is live, check how your pages appear in Google with the SERP Snippet Preview, review page structure with the Heading Structure Checker, and test your structured data with the JSON-LD Validator.