Pay Per Crawl

Detects monetized agent access: an HTTP 402 Payment Required response carrying a crawler-price header (Cloudflare Pay Per Crawl) or an x402 JSON payment body, served to AI crawler user-agents. This is the most literal expression of the “Actionable” pillar — a site that has decided agent access to its content is a paid transaction, not a free crawl.

What it checks

The audit re-requests representative pages using AI crawler user-agents and inspects the response for:

• Cloudflare Pay Per Crawl: a 402 status with a crawler-price response header stating the price to fetch the resource.
• x402: a 402 status with a JSON body following the x402 protocol shape (payment requirements, accepted schemes, resource identifier).

When found, it reports the monetization scheme, the declared price where present, and which user-agent classes (training / AI-search / user-action, per AI Crawler Access) the 402 applies to.

Solution

If you’re monetizing crawler access, scope the 402 to the crawler classes you intend to charge — typically training and bulk-ingestion crawlers — and keep it off user-action fetchers so a real user’s in-the-moment request through an AI assistant isn’t the thing that gets billed:

GET / HTTP/1.1
User-agent: GPTBot

HTTP/1.1 402 Payment Required
crawler-price: $0.01
Content-Type: text/plain

Payment required for automated access. See https://example.com/crawler-pricing

If you have no intention of charging for access, no action is needed — this rule is purely informational unless a 402 shows up somewhere it shouldn’t.

Enable / Disable

Disable this rule

[rules]
disable = ["ax/pay-per-crawl"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/pay-per-crawl"]
disable = ["*"]