Block AI training bots in robots.txt and keep AI search open

An AI-aware robots.txt is a robots.txt file that names each AI company's crawlers by the job they do, so a site can refuse model training and still be read by the search and answer bots behind ChatGPT, Claude, Perplexity and Google's AI features. You need one the day you decide your content should not train models but should still be cited. The common mistake is a single rule that blocks every AI user agent at once, and with them the visibility the site was built to earn.
The fix is one file and one decision per company. Below: which bot does what in November 2026, the file we ship, what robots.txt cannot stop, and the new no-training line the IETF is drafting.
Why one "block all AI" rule costs you citations
Every large AI company now runs more than one bot, and each bot answers to its own line in robots.txt. OpenAI says it plainly: each setting is independent, so a site can allow OAI-SearchBot to appear in search results and disallow GPTBot to keep its content out of training (OpenAI, Overview of OpenAI crawlers). Block OAI-SearchBot as well and the site drops out of ChatGPT search answers; OpenAI says it can still show up there only as a navigational link.
Anthropic describes the same split. Turning off Claude-SearchBot may reduce a site's visibility and accuracy in search results, and turning off Claude-User stops Claude from reading your pages when a user asks about them (Anthropic, robots.txt controls).
The urge to block has a real cause. Cloudflare reports that 52% of the crawler requests it saw in June 2026 were for AI training, up from 22% in spring 2025, while pure search crawling is a small and shrinking share (Cloudflare, Content Independence Day, one year on). Training traffic costs bandwidth and sends few visitors back. Search traffic is the part that cites you. A blanket block treats them as one thing.
Which AI bots train, which search, and which fetch for a user
Each company splits its bots across three jobs: collecting training data, building a search index, and fetching a page because a person asked a question. The names to know, from each company's own documentation:
- OpenAI. GPTBot collects content that may train its foundation models. OAI-SearchBot indexes pages for ChatGPT search. ChatGPT-User visits a page when a user's question needs it, and OpenAI says robots.txt rules "may not apply" to it.
- Anthropic. ClaudeBot collects training data. Claude-SearchBot indexes pages for search answers. Claude-User fetches pages for a user's question, and Anthropic lists it among the bots that honor robots.txt.
- Google. Google-Extended is a control token with no crawler of its own: Google crawls with its usual user agents, and the token decides whether that content may train Gemini models and ground answers in Gemini Apps and Vertex AI (Google, common crawlers). AI Overviews and AI Mode are part of Search and follow Googlebot.
- Perplexity. PerplexityBot indexes pages for Perplexity's search results and, per Perplexity, does not collect training data. Perplexity-User fetches pages for a user's question and "generally ignores robots.txt rules" (Perplexity, crawlers).
Two of these are traps. Google-Extended sounds like the switch for Google's AI answers, and it isn't. And the user-triggered fetchers sit partly outside robots.txt, so the file can't promise what a site owner often expects from it.
The robots.txt that blocks training and keeps AI search open
RFC 9309, the robots.txt standard, lets one group carry several user-agent lines, so the file stays short (RFC 9309, Robots Exclusion Protocol). This is the version we start from:
# Training: no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
# Search and answer bots: yes
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
# Everyone else, Googlebot included
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThree details matter. A crawler follows the group that names it and ignores the wildcard, so GPTBot reads the first group and skips the asterisk. Rules are per host: Anthropic asks for the file on every subdomain you want covered. And OpenAI says a change takes about 24 hours to reach its search systems; Perplexity says up to 24 hours.
One trade-off to decide on purpose. Disallowing Google-Extended also stops grounding in Gemini Apps, which is an answer surface of its own. If Gemini citations matter more to you than staying out of Gemini training, take Google-Extended out of the first group.
What robots.txt cannot do for AI answers
The standard is explicit: "These rules are not a form of access authorization." A robots.txt line is a request that declared crawlers choose to honor. Four limits follow.
- User-triggered fetchers. ChatGPT-User and Perplexity-User may fetch a page despite a disallow, because a person asked for it. Stopping them takes a firewall rule or a login, and that also takes the page out of those answers.
- Google's AI Overviews. Google says robots.txt rules for Googlebot are the control for Search, AI features included. To limit what appears there, it points to nosnippet, data-nosnippet, max-snippet and noindex (Google, AI features and your website). Disallowing Googlebot removes you from Search entirely.
- A firewall in front of the file. If your CDN blocks AI bots at the edge, the allow in robots.txt never gets read. Cloudflare blocks known AI crawlers by default on new domains; our guide to fixing Cloudflare's AI crawler block covers the override.
- Spoofed user agents. Anyone can send "GPTBot" in a header. OpenAI, Anthropic and Perplexity publish their IP ranges, and a log line counts only when its IP is on the list.
The new no-training line: __usage_ai-training
Naming bots one by one has a limit: every new training crawler needs a new line. On 30 October 2026 an IETF draft in the AI Preferences working group proposed a general signal, written as an ordinary robots.txt group with a reserved token (IETF, Machine-Readable Training Preference Signal):
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/That group says no to AI training everywhere except the press kit. It states a usage preference: the draft says the token "is not a directive for crawling behavior", and a wildcard group never counts as matching it. Anthropic began honoring the signal the same day and treats a rule that names ClaudeBot as more specific. Google says it intends to support it in early 2027.
Our advice for now: add the token group, and keep the named training groups too. The draft is at version 00 and may change, and one company reads it today.
What this looks like in practice
Take a SaaS with a public docs site and a blog. It wants Claude and ChatGPT to cite its docs when a buyer asks how to integrate the product, and it does not want its writing in the next training set. We would:
- Pull a week of access logs and count requests per AI user agent, keeping only those whose IP is on the vendor's published list (openai.com/gptbot.json, openai.com/searchbot.json and the others).
- Check the CDN's bot settings, so the edge does not block a bot the file allows.
- Ship the file above with the __usage_ai-training group added, on the apex domain and on the docs subdomain.
- Fetch /robots.txt from both hosts, then watch the logs: GPTBot and ClaudeBot should stop at robots.txt, while OAI-SearchBot and Claude-SearchBot keep reading pages.
The change is one file per host and an afternoon of log reading. The hard part is the decision behind each line, and that belongs to whoever owns the content. For the other half of the work, getting the pages you allow actually cited, see how to get cited in Google AI Overviews and ChatGPT.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
- Google: List of Google common crawlers (Google-Extended)
- Google Search Central: AI features and your website
- Perplexity: Perplexity crawlers
- RFC 9309: Robots Exclusion Protocol
- IETF draft-kelley-aipref-training-preference-signal-00
- Cloudflare: Content Independence Day, one year on
Frequently asked questions
No. GPTBot only collects content that may train OpenAI's models. Whether your pages appear in ChatGPT search answers depends on OAI-SearchBot, which you can keep allowed while GPTBot is blocked. OpenAI documents the two settings as independent and says a change takes about 24 hours to reach its search systems. ChatGPT-User, the fetcher that opens a page when a user asks, may read it either way.
No. AI Overviews and AI Mode are part of Google Search and are crawled by Googlebot, and Google states that Google-Extended does not affect inclusion in Search. Google-Extended controls Gemini model training and grounding in Gemini Apps and Vertex AI. To limit what AI Overviews show from a page, Google points to nosnippet, data-nosnippet, max-snippet and noindex, which limit the page in regular results too.
Yes. robots.txt rules are path-based, so a group can disallow / and allow a folder such as /press-kit/ or /docs/. The longest matching path wins, as RFC 9309 defines. The new __usage_ai-training token from the October 2026 IETF draft uses the same path rules to state a training preference per folder. Anthropic reads that token today; other companies still need their training bots named.
It is a request, and the standard says so: its rules are not a form of access authorization. The documented crawlers from OpenAI, Anthropic, Google and Perplexity say they follow it, but user-triggered fetchers may not, and a header can claim to be any bot. Content behind a login or a firewall rule is the only content a crawler cannot read. Whether training on public content is lawful in your market is a legal question for a lawyer, and robots.txt does not answer it.
Related articles
May 2, 2026 · 12 min read
October 28, 2026 · 9 min read
August 9, 2026 · 6 min read