SEO SentinelBeta
Menu

Draft under editorial review. This page is not yet publication-ready and is excluded from search engines.

Crawler names and purposes change. Re-verify every row of the crawler table against each operator’s documentation on the day of publication, then update the date below.

GEO & AI search

AI crawlers and robots.txt: what each token controls

Which robots.txt tokens control AI search, AI training and user-triggered fetchers (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended) and how to test them.

By the SEO Sentinel team Updated 10 min read

“Should we block AI crawlers?” has become a routine client question. The useful answer is that there is no single AI crawler. Operators run separate agents for search, model training and user-triggered fetching, and each one is controlled by its own robots.txt token. Blocking the wrong one can remove a site from an AI search product while doing nothing about training, or the other way round.

Last verified: 5 November 2026, against each operator’s own documentation (linked in the table). Operators add and rename agents; check the linked pages before changing a client’s robots.txt.

The main tokens, by purpose

Token Operator Purpose Documentation
OAI-SearchBot OpenAI AI search: surfaces websites in ChatGPT search features OpenAI bots
GPTBot OpenAI Training: content may be used to train OpenAI foundation models OpenAI bots
ChatGPT-User OpenAI User-triggered: visits pages for user actions in ChatGPT and Custom GPTs OpenAI bots
Claude-SearchBot Anthropic AI search: improves search result quality for Claude users Anthropic
ClaudeBot Anthropic Training: content that could contribute to model training Anthropic
Claude-User Anthropic User-triggered: fetches pages when Claude users ask questions Anthropic
PerplexityBot Perplexity AI search: surfaces and links websites in Perplexity results Perplexity bots
Perplexity-User Perplexity User-triggered: visits pages to answer a user question Perplexity bots
Googlebot Google Search engine, including Google’s AI features in Search Google crawlers
Google-Extended Google Control token only: use of content for training and grounding in some other Google AI systems (e.g. Gemini) Google crawlers
Applebot Apple Search engine for Spotlight, Siri and Safari; data may also be used for Apple AI features Applebot
Applebot-Extended Apple Control token only: opt out of Apple generative-AI training Applebot
CCBot Common Crawl Dataset: the open Common Crawl corpus, widely used in research and model training CCBot

Three points practitioners often miss:

  • Google-Extended does not control Google Search or its AI features. Those use Googlebot. Blocking Google-Extended doesn’t remove a site from AI Overviews; blocking Googlebot removes it from Google Search entirely.
  • Control tokens aren’t crawlers. You will never see “Google-Extended” or “Applebot-Extended” in your logs. They are read by the main crawler.
  • User-triggered fetchers may not follow robots.txt. OpenAI and Perplexity both state that robots.txt rules may not apply to fetches a user initiates. Treat robots.txt as ineffective for them, and use authentication or server-side controls for content that must not be fetched.

How robots.txt precedence actually works

Under RFC 9309, the Robots Exclusion Protocol standard:

  1. A crawler looks for the group whose User-agent line matches its token. If one exists, it ignores the * group completely. Rules are not merged.
  2. Within that group, the longest matching path rule wins. When an Allow and a Disallow match with equal length, Allow wins.
  3. Matching is case-sensitive for paths, case-insensitive for the user-agent token.

That first rule causes most mistakes. Consider:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /blog/

GPTBot follows only its own group. It is blocked from /blog/ but now allowed into /admin/ and /checkout/. If you add a dedicated group for a bot, repeat the generic rules it should still obey.

Some crawlers document a fallback. Applebot, for example, follows Googlebot’s rules when no Applebot group exists.

Common configurations

Allow AI search, opt out of training:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /admin/

Several User-agent lines can share one group. Search agents such as OAI-SearchBot, Claude-SearchBot and PerplexityBot have no group of their own here, so they follow *.

Block everything AI-related except Google Search: list every AI search, training and token agent in a disallow group, and make sure Googlebot isn’t included. Remember this also removes the site from those products’ search features. That’s a business decision, so get it in writing from the client.

Testing your rules

  1. Parse it like a crawler. For each token and a set of important URLs (homepage, key sections, a few templates), work out which group applies and which rule wins. Record the deciding line number as evidence.
  2. Check the response, not just the file. robots.txt must return 200 on every host and protocol. A 5xx on robots.txt can make crawlers stop crawling the site.
  3. Check the network layer. CDNs, firewalls and bot-protection products frequently block AI crawlers by user agent or IP, whatever robots.txt says. If you see 401/403/429/503 responses for bots in your logs, robots.txt isn’t the reason.
  4. Verify bots in logs properly. User-agent strings are easy to fake. Operators publish IP ranges or verification methods; use them before drawing conclusions from log data.

Allowing access is not the same as being cited

Letting AI search crawlers in makes a site eligible to appear in those products. It doesn’t mean the site will be mentioned or cited. That depends on the question, the content and the system. Treat crawler access as one part of a broader GEO audit, and measure visibility separately, by recording real answers over time.