Blocking AI agents at the edge: what robots.txt hides

What edge blocking is

robots.txt states a policy. It does not enforce one. A site can allow an AI user-agent in robots.txt and still block it in front of the origin, through a WAF rule, a CDN bot-management toggle, or a managed rule set with a “block AI bots” switch. The crawler gets a challenge page, a 403, or a stripped-down interstitial, and nothing in robots.txt shows it.

The two cases differ in cost. Blocking a training crawler such as GPTBot is often the intent behind flipping that switch. Blocking Claude-User is not: that user-agent fetches a page because a person asked for it inside an assistant and is waiting for the answer.

What squirrel checks

The rule runs once per crawl. It uses three homepage fetches made during the prefetch phase, one with a browser user-agent, one as GPTBot and one as Claude-User, and compares each AI response against the browser baseline. It emits one or two checks: at most one per AI probe, or a single check covering all of them.

• agent-blocking, skipped with the message Agent access not checked: the discovery probe was not sent and the value not-checked when the probes did not run or the browser probe was never sent, or Browser baseline did not return 2xx (status <status>) — cannot compare agent access when the browser fetch itself failed, where <status> is the status it returned or n/a when the probe is missing entirely. Without a healthy baseline the site may be down for everyone, so the rule declines to blame the agents.
• agent-access-payment: reports at info status with the message <agent> hit a pay-per-crawl wall (<signal>) — reported by ax/pay-per-crawl, not a block when the response is a 402 or carries a payment signal. The parenthesised signal is omitted when the response named none. A deliberate payment wall is not an access block.
• agent-access: at info status with <agent> not checked: the discovery probe was not sent and the value not-checked when that probe was never sent, and with <agent> probe was inconclusive (<reason>) when it was sent and never completed. At warn status with <agent> got the same status as the browser but only N% of the body — likely a soft-block / interstitial served to the agent when the agent response was itself 2xx, the browser body was not empty, and the agent body came back under a fifth of its size. At warn status with <agent> is blocked (<reason>) while the browser gets 200 — often an intentional training opt-out, but confirm it was deliberate for a bot challenge, a 401, a 403 or a 5xx. The same condition on Claude-User is a fail with Claude-User is blocked (<reason>) while the browser gets 200 — this breaks a live fetch a real user triggered inside an assistant, the worst case for agent access.
• agent-blocking: passes with GPTBot and Claude-User reached the homepage like a browser, naming the probes it ran, when none of them was classified as blocked, soft-blocked, inconclusive or payment-walled. An unchallenged 3xx or 404 counts as clean, since the rule is comparing access rather than status.

A probe counts as a bot challenge when the response carries cf-mitigated: challenge, or when the first 10 KB of its body holds Cloudflare’s challenge script (/cdn-cgi/challenge-platform/h/…/orchestrate/) or a __cf_chl_ or _cf_chl_opt marker, or the first 4 KB holds a <title> that starts with Just a moment. The reason in the message names which: cf-mitigated, challenge-platform, __cf_chl_ or just-a-moment. The JavaScript Detections script that Cloudflare adds to ordinary pages loads from /cdn-cgi/challenge-platform/scripts/jsd/ and does not count, and neither does the phrase in page text.

The soft-block threshold is a fifth of the browser body size. That is the shape of an interstitial: a 2xx with a fraction of the content. The rule’s severity is error so that a blocked user-action fetcher badges as an error; a training-crawler block emits a warn check, which the report downgrades to a warning on its own.

A probe goes unsent when discovery probes are turned off, when no rule that reads it was enabled for the crawl, or when the crawl’s preamble time budget ran out first. A probe that went out and got no answer, a refused connection or a timeout, counts the same. None of those says anything about the site, so the result is not-checked, never absent.

How to fix it

Allow-list the user-action fetchers in the edge rules, whatever the bot-management preset does by default:

ChatGPT-User
Claude-User
Perplexity-User

These three fetch on behalf of a person who is waiting. Blocking a training crawler alongside them is a separate decision, and this rule reports it without penalising the score.

The common false positive is a rate limit that happens to catch the probe. A single homepage fetch is unlikely to trip one, but a site already under load can return a 5xx to the agent and a 200 to the browser by luck of timing. Re-run the audit after any WAF, CDN or bot-management change and check whether the finding is stable.

Enable / disable

Disable this rule

[rules]
disable = ["ax/agent-blocking"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/agent-blocking"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• ax/ai-crawlers: the declared policy in robots.txt, which this rule tests against reality.
• ax/pay-per-crawl: the 402 responses this rule hands off rather than reporting as blocks.
• ax/content-without-js: content an agent cannot see even when it does reach the page.
• security/csp: the Content-Security-Policy header and its directives, set in the same place.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See AI crawlers for how an agent works through a report.

References

• Robots Exclusion Protocol, RFC 9309
• 403 Forbidden, MDN
• OpenAI crawlers, OpenAI platform docs

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. Each probe shows its status and body size against the browser. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.