AI crawlers in robots.txt: which bots you block, by class

robots.txt blocking Claude-User or PerplexityBot alongside the training bots. squirrel classifies all 24 known AI crawlers and names what each block costs.

What an AI crawler block is

“Block AI crawlers” is not one decision. The named bots serve four different purposes, each with its own robots.txt user-agent token, so a rule aimed at one never covers another. Blocking ClaudeBot, a training crawler, does not block Claude-User, a live fetch a person triggered inside an assistant.

Training crawlers, such as GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot and Bytespider, collect pages for a vendor’s future model training.
AI-search indexers, such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot and YouBot, build the index an answer engine cites from.
User-action fetchers, such as ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User and Meta-ExternalFetcher, fetch a page because a person asked for it right now.
Archive crawlers, CCBot and the Internet Archive’s ia_archiver and archive.org_bot, feed the Common Crawl corpus and the Wayback Machine, which are the shared sources training sets are built from.

Blocking a training crawler is a policy choice about one vendor. Blocking an archive crawler is wider, because the corpus it feeds is shared: a CCBot block keeps future pages out of Common Crawl, which many training sets are built from. robots.txt is a request that cooperative crawlers honour, and it does not remove captures that already exist.

What squirrel checks

The rule runs once per crawl against the parsed robots.txt and evaluates 24 known crawler tokens. For each one it resolves the groups that apply, preferring a group naming the token over User-agent: *, and treats the bot as blocked when those groups carry Disallow: / with no Allow: /. Token matching is case-insensitive. It emits one to three checks:

ai-crawler-access: always emitted, at info status. With no robots.txt, or one that does not exist, the message is No robots.txt — all AI crawlers are allowed by default and nothing else runs. Otherwise the message is N/24 known AI crawlers allowed, followed by ; M blocked (<tokens>) only when something is blocked, listing the tokens in the order the crawler list declares them. The blocked tokens are also listed as items showing vendor and class.
ai-answer-access: emitted only when an AI-search indexer or a user-action fetcher is blocked. It warns with the message robots.txt blocks <clauses>. These are separate user-agents from the training bots — blocking ClaudeBot (training) does not block Claude-User (user-action); check the block was intended. The clauses are AI-search indexers (<tokens>) — lost AI citations and user-action fetchers (<tokens>) — breaks live requests users made inside an assistant, joined with ; , and a class with nothing blocked is left out.
archive-crawler-access: emitted only when an archive crawler is blocked. It warns with the message robots.txt blocks archive crawlers — this removes the site from the Wayback Machine and/or the Common Crawl corpus, the archives AI training sets are built from. Blocking CCBot opts you out of every model trained on Common Crawl, not just one vendor.

Blocking training crawlers alone produces no warning. The summary check reports it and the score is untouched, because opting out of one vendor’s training is a legitimate choice. Ambiguous data harvesters such as Diffbot are classified as training for the same reason: that is the class whose blocking never penalises.

How to fix it

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Put the groups in robots.txt at the site root. This example opts out of OpenAI’s training crawl while keeping the site citable in ChatGPT search and reachable when a person asks for it.

The common false positive is a block that lives at the edge rather than in robots.txt. A WAF rule or a bot-management toggle can serve a challenge to the same user-agent that robots.txt allows, and this rule reads robots.txt only. ax/agent-blocking probes the actual responses and catches that case.


Rule ID	`ax/ai-crawlers`
Category	Agent Experience
Scope	Site-wide
Severity	warning
Weight	1/10

Enable / disable

Disable this rule

[rules]
disable = ["ax/ai-crawlers"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/ai-crawlers"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

ax/agent-blocking: the same question asked of the live responses, which catches edge blocks robots.txt cannot show.
ax/content-signals: per-purpose policy declared in robots.txt, cross-checked against these blocks.
ax/archive-indexing: whether the archive crawlers reached the site in the end.
crawl/robots-txt: whether robots.txt exists and parses at all.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See AI crawlers for how an agent works through a report.

Glossary: Agent experience (AX), AI crawler, Generative engine optimization (GEO), robots.txt.

References

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. Every blocked bot is listed with its vendor and class. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.

Install squirrel

One command to install, one to run your first audit.

See a sample report

What the Agent Experience section looks like on a real site.

Edit this page