AI crawlers: GPTBot, ClaudeBot and what to allow
An AI crawler fetches pages for an AI company: to train models, to build an AI search index, or for a person's live request. Tokens, and what to allow.
An AI crawler is a bot that fetches web pages for an AI company, either to train models, to build the index an AI search product cites from, or to answer a person’s request in real time.
Each class has its own robots.txt user-agent token. OpenAI uses GPTBot for training, OAI-SearchBot for ChatGPT search and ChatGPT-User for actions a user starts, and says each setting is independent of the others. Anthropic splits the same way with ClaudeBot, Claude-SearchBot and Claude-User. Google’s Google-Extended is a robots.txt token only: it controls whether content Google already crawls can be used for training, and sends no requests of its own. Archive crawlers such as Common Crawl’s CCBot sit upstream of many training sets.
Category: Agent Experience
Why it matters
A rule aimed at one token never covers another, so “block AI crawlers” is several decisions. Blocking a training crawler is a policy choice about one vendor. Blocking an AI search indexer keeps the site out of that product’s answers, though OpenAI notes a blocked site can still appear as a navigational link. Blocking a user-action fetcher at the edge breaks a request a person made inside an assistant; OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person starts those requests. A CCBot block is wider than it looks: it keeps future pages out of Common Crawl, which many training sets are built from. robots.txt is also only a request, and a WAF can still challenge a bot that robots.txt allows.
How squirrelscan checks it
ax/ai-crawlers reads robots.txt and resolves the groups that apply to each known AI crawler token. It lists every crawler blocked from the whole site (Disallow: / with no Allow: /), and warns only when an AI search indexer, a user-action fetcher or an archive crawler is blocked; a training-only block never costs score. ax/agent-blocking fetches the homepage as a browser, as GPTBot and as Claude-User, and flags a bot challenge, a 401, 403 or 5xx, or a body under a fifth of the browser’s size served to an agent while the browser gets a 2xx. ax/archive-indexing looks the domain up in Common Crawl and the Wayback Machine, and needs a cloud audit.
| Rule | What it flags | Severity |
|---|---|---|
| AI crawlers in robots.txt: which bots you block, by class | robots.txt blocking Claude-User or PerplexityBot alongside the training bots. squirrel classifies all 24 known AI crawlers and names what each block costs. | warning |
| Blocking AI agents at the edge: what robots.txt hides | A WAF or bot-management rule serving GPTBot or Claude-User a challenge the browser never sees. squirrel probes the homepage as all three. | error |
| robots.txt: what to put in it and how squirrel checks it | A missing robots.txt, a Disallow: / left in production, or a syntax error changes what gets crawled. squirrel checks existence, syntax and rules. | error |
| Common Crawl and Wayback: is your site in the archives | A site missing from the Common Crawl index or the Wayback Machine, or archived only with a capture over a year old. squirrel checks both. | warning |
How to fix it
Decide per class, then write one group per token (see the AI crawler access fix):
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
This opts out of OpenAI’s training crawl and lets in its search crawler and the fetches a person starts in ChatGPT. Then check the edge: allow-list ChatGPT-User, Claude-User and Perplexity-User in the WAF or bot-management rules, whatever the preset does (see the agent blocking fix).
Run the check
squirrel audit https://example.com --rule-include ax/ai-crawlers,ax/agent-blocking,crawl/robots-txt,ax/archive-indexingA signed-out local audit costs nothing. The archive lookup runs in cloud audits only.
Related terms
- robots.txt: A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).
- Content Signals: Content-Signal lines in robots.txt that state whether content may be used for search, AI input or AI training.
- Agent experience (AX): How well a site serves AI agents that read it, find their way around it and act on it, the way UX serves people.
- noai and noimageai: Meta robots values some sites use to ask AI systems not to train on their text or images; not part of any standard search engines honour.