Content-Signal in robots.txt: search, ai-input, ai-train

A Content-Signal line with a typo, or one declaring ai-train=yes while robots.txt blocks the training crawlers. squirrel parses and cross-checks both.

What Content-Signal is

Content-Signal is a robots.txt directive that states policy per purpose rather than per bot. It takes comma-separated key=value pairs, and three keys are defined: search covers search-result display and answer-engine citation, ai-input covers use as input to an AI system at inference time, and ai-train covers training or fine-tuning a model. Each takes yes or no.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

It sits inside a normal robots.txt group, so a directive under User-agent: * speaks for every crawler and one under a named group speaks only for those agents. Cloudflare’s managed robots.txt also emits a use= key with the values immediate, reference or full, describing how a collected page may be consumed. Declaring a signal is optional.

What squirrel checks

The rule runs once per crawl against the raw robots.txt. It strips comments, tracks which user-agent group each Content-Signal: line sits under, and emits one to three checks:

content-signals: reports at info status with the message No Content-Signal declared in robots.txt (optional per-purpose AI policy) and the value absent when no directive is present. No further checks run.
content-signals-policy: always emitted when a directive is present, at info status. The message is Content-Signal declares: followed by each declared key with its description, joined with ; , and then any recognised auxiliary key as <key>=<value>. The three descriptions are search=yes (search-result display & answer-engine citation), ai-input=yes (input to an AI system at inference time) and ai-train=no (training / fine-tuning a model). When a directive is present but declares nothing recognised, the message is Content-Signal present but declares no recognized signal.
content-signals-syntax: warns when a token’s key is not one of the three signals or use, when a signal’s value is anything other than yes or no, when a use value is not immediate, reference or full, or when a token has no = at all. The message is Content-Signal has invalid syntax — <notes>. Valid form is comma-separated key=value using search/ai-input/ai-train with yes/no., where the notes are unknown key(s): <keys> and malformed token(s): <tokens> joined with ; , and a note with nothing in it is left out.
content-signals-contradiction: warns when a signal is declared yes while robots.txt fully disallows at least one crawler in the class that signal implies access for. search maps to the AI-search indexers, ai-input to the user-action fetchers, and ai-train to the training crawlers. The message is Content-Signal <signal>=yes but robots.txt fully Disallow-s <tokens> — a crawler can't tell which directive wins, one clause per contradiction joined with ; .

A contradiction is only reported inside the directive’s own scope. A Content-Signal under User-agent: *, or outside any group, is checked against every blocked crawler; one under a named group is checked only against blocks on those agents. Full blocking means Disallow: / with no Allow: / in the groups that apply to that bot. One blocked crawler in the class is enough to raise the contradiction, and each signal is reported at most once.

How to fix it

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

User-agent: GPTBot
Disallow: /

Put the directive in robots.txt and keep it in step with the blocks below it. The example is consistent: ai-train=no and a disallowed training crawler say the same thing. Declaring ai-train=yes above that block is what the contradiction check reports.

There is no common false positive. Absence is reported at info status and never affects the score.


Rule ID	`ax/content-signals`
Category	Agent Experience
Scope	Site-wide
Severity	warning
Weight	2/10

Enable / disable

Disable this rule

[rules]
disable = ["ax/content-signals"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/content-signals"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

ax/ai-crawlers: the crawler blocks this rule cross-checks the declaration against.
ax/noai-signals: the per-page equivalent, declared in meta tags and the X-Robots-Tag header.
ax/rsl-license: machine-readable licensing terms, including AI-training permissions and pricing.
crawl/robots-txt: whether robots.txt exists and parses at all.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See AI crawlers for how an agent works through a report.

References

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. The declared policy and any contradiction are listed together. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.

Install squirrel

One command to install, one to run your first audit.

See a sample report

What the Agent Experience section looks like on a real site.

Edit this page