Agent Blocking

robots.txt only tells you what a site says it allows. This rule checks what actually happens: it fetches the homepage with a plain browser user-agent, then again as GPTBot, then again as Claude-User, and compares the results. A site can declare ChatGPT-User: Allow: / in robots.txt and still WAF-block that same user-agent at the edge — AI Crawler Access wouldn’t catch that; this rule does.

What it checks

The audit issues three requests to the homepage: one with a standard browser user-agent, one as GPTBot, and one as Claude-User. It flags the site when the browser request returns 200 but an agent request comes back as:

• an HTTP 403,
• a Cloudflare challenge response (cf-mitigated: challenge header, or a body containing __cf_chl_), or
• another bot-challenge pattern that a browser wouldn’t hit.

Both cases are reported as a warning-level check:

• GPTBot blocked: often intentional (opting out of training) but worth surfacing if it wasn’t a deliberate choice.
• Claude-User blocked: the more serious case — this is a live user-action fetcher, so blocking it breaks a request a real person initiated inside an AI assistant, not a background crawl. The finding message calls this out explicitly, even though the rule’s severity badge stays warning.

Why it matters

Cloudflare has said it will flip AI-crawler blocking on by default for many zones from September 15, 2026, and most bot-management products bundle training crawlers, AI-search bots, and user-action fetchers into one “AI bots” toggle. That bundling is the risk: a site owner who intends to opt out of model training can end up blocking the fetcher behind their own customers’ AI-assistant requests without knowing it, because a single dashboard switch controls all three categories at once. Whatever policy a site chooses, it should be the one that’s actually in effect at the edge, not whatever a bot-management default silently applied.

Solution

• Check your WAF / bot-management rules for a blanket “block AI bots” or “block AI crawlers” toggle and confirm it’s scoped to the crawler classes you actually want to block (see AI Crawler Access for the training vs. search vs. user-action breakdown).
• Explicitly allow-list ChatGPT-User, Claude-User, and Perplexity-User in your edge rules if you want AI-assistant users to be able to browse your site through their tools.
• Re-run this probe after any change to bot-management, WAF, or CDN configuration.

Enable / Disable

Disable this rule

[rules]
disable = ["ax/agent-blocking"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/agent-blocking"]
disable = ["*"]