Content Signals

Parses Content-Signal directives in robots.txt: a machine-readable way to declare AI policy per-purpose rather than per-crawler. Instead of allow/disallow-ing named bots one at a time, a site declares intent directly — search=yes, ai-input=no, ai-train=no — and lets crawlers that honor the standard resolve their own behavior against it.

What it checks

The audit parses any Content-Signal line in the site’s robots.txt and validates its key=value syntax against the three defined signals:

• search — usable for search-result display and answer-engine citation
• ai-input — usable as input to an AI system at inference time (e.g. RAG, browsing tools)
• ai-train — usable to train or fine-tune a model

Cloudflare’s managed robots.txt also emits an auxiliary use=<immediate|reference|full> key describing how AI systems may consume collected content. It carries no crawl-permission semantics, but it is valid syntax — the audit recognizes it rather than flagging Cloudflare-managed files as invalid.

Each takes yes or no. The rule reports:

• Presence and syntax: whether a Content-Signal line exists, at the wildcard (User-agent: *) group or scoped to specific crawlers, and whether it parses cleanly.
• Declared policy: a plain-language summary of what the site has opted into or out of.
• Contradictions: cases where the declared signal disagrees with the crawler-level rules covered by AI Crawler Access within the signal’s own user-agent scope — for example, Content-Signal: ai-train=yes under User-agent: * while Disallow: / is set for GPTBot specifically. A contradiction surfaces as a warning, since it leaves crawlers to guess which directive wins. A signal scoped to one crawler’s group is not compared against other crawlers’ blocks.

Solution

Add a Content-Signal line to robots.txt that matches your actual crawler-blocking policy:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

If you block specific training crawlers outright, make sure the signal doesn’t claim to allow training generally, or scope the signal to the same user-agent group as the block:

User-agent: GPTBot
Content-Signal: ai-train=no
Disallow: /

Enable / Disable

Disable this rule

[rules]
disable = ["ax/content-signals"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/content-signals"]
disable = ["*"]