HTTP 402 for crawlers: pay-per-crawl and x402 detection

What pay-per-crawl is

HTTP 402 Payment Required has been reserved in the HTTP specification for years and has only recently found a use: charging automated clients for access. Cloudflare’s Pay Per Crawl returns a 402 with a crawler-price header naming what a fetch costs. x402 is a separate scheme that returns a 402 with a JSON body describing how to pay.

Charging for crawler access is a business decision, not a defect, and this rule reports it as such. The one case worth calling out is scope. A training crawler pulling thousands of pages and a person’s assistant fetching one page they asked about are different clients, and a 402 aimed at the first will hit the second unless the rule is scoped.

What squirrel checks

The rule runs once per crawl and reads the three homepage probes made during prefetch: one with a browser user-agent, one as GPTBot and one as Claude-User. It emits one or two checks:

• pay-per-crawl: reports at info status with the message Monetized agent access not checked: the discovery probe was not sent and the value not-checked when the probes did not run, or when no probe required payment and at least one was never sent or got no answer (named in details.notChecked; the message then ends got no answer if any of them went out). With no probe requiring payment it otherwise reports at info status with No monetized agent access (pay-per-crawl / x402) detected and the value absent. Otherwise it reports at info status with Monetized agent access configured via <mechanisms> for <user-agents>, joining the distinct mechanisms with and and the charged user-agent names with , . A crawler-price or crawler-charged signal reads as Cloudflare Pay Per Crawl, an x402 body reads as x402, and anything else reads as an unnamed HTTP 402 payment wall.
• pay-per-crawl-user-action: emitted only when Claude-User is charged while the browser probe returned a 2xx. It warns with the message Claude-User (a live user-action fetch) is charged HTTP 402 while browsers get a normal 2xx response — this bills a real person's in-the-moment question, not a background crawl.

Detection is a report, not a judgement. Only the second check can affect anything, and it fires on one specific asymmetry: a person’s live request being billed while an ordinary browser is not.

A probe goes unsent when discovery probes are turned off, when no rule that reads it was enabled for the crawl, or when the crawl’s preamble time budget ran out first. A probe that went out and got no answer, a refused connection or a timeout, counts the same. None of those says anything about the site, so the result is not-checked, never absent.

How to fix it

Scope the 402 to the crawler classes you meant to charge, and exclude the user-action fetchers:

ChatGPT-User
Claude-User
Perplexity-User

These three fetch because a person asked, so they belong outside the payment wall even on a site that charges for bulk crawling.

There is no common false positive. A 402 is unambiguous, and the rule hands the case off from ax/agent-blocking so a payment wall is never reported as an access block.

Enable / disable

Disable this rule

[rules]
disable = ["ax/pay-per-crawl"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/pay-per-crawl"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• ax/agent-blocking: access blocks, which this rule takes the payment cases out of.
• ax/ai-crawlers: the declared robots.txt policy for the same user-agents.
• ax/rsl-license: machine-readable licensing terms, including crawl pricing.
• ax/content-signals: per-purpose AI policy declared in robots.txt.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See AI crawlers for how an agent works through a report.

References

• 402 Payment Required, MDN
• Pay per crawl, Cloudflare docs

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. The mechanism and every charged user-agent are listed. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.