robots.txt License: directive and RSL document validation

What RSL is

Really Simple Licensing is an XML format for stating machine-readable terms for a site’s content, including whether it may be used for AI training and what a crawl costs. The document is referenced from robots.txt with a License: directive, or from a response with a Link: <...>; rel="license" header.

It answers a question robots.txt cannot. A Disallow says a crawler may not fetch a path. A licence says what may be done with content that was fetched. Publishing one is optional and this rule never asks for one. A reference that points at nothing is the defect it exists to catch.

What squirrel checks

The rule runs once per crawl. It looks for a License: directive in robots.txt, a Link: rel=license header, and any licence URLs those produce, then reads the result of fetching each URL. It emits one or two checks:

• rsl-license: reports at info status with the message RSL licensing not checked: the discovery probe was not sent or got no answer and the value not-checked when the robots.txt read did not run or got no answer.
• rsl-license-present: reports at info status with No machine-readable licensing declared (no robots.txt License: directive or Link: rel=license) and the value absent when there is no signal at all. No further checks run. Otherwise it reports at info status with License reference declared, appending via robots.txt License: and via Link: rel=license for whichever source was present, then (N URL) or (N URLs) for the count it found.
• rsl-license-valid: reports at info status with Valid RSL document found at <url> when a referenced document returned 200, parsed as XML and read as RSL, adding (+N more) when several did. When nothing resolved, nothing was broken, and at least one document is recorded as never requested, it reports at info status with License declared but the referenced document was not checked (N URLs) and the value unknown. Otherwise it warns with License declared but the referenced document didn't resolve as valid RSL (<url>: <error or status N>), listing each broken URL with its error or its status.

The distinction in the last bullet is deliberate. A URL the crawl tried and could not fetch is a broken reference and warns. A URL the crawl never requested proves nothing, and the check says it does not know rather than reporting a defect the audit did not establish. Both look like a status of 0, so the recorded reason is the only thing separating them. A declared licence with no recorded documents at all falls through to the warning.

A probe goes unsent when discovery probes are turned off, when no rule that reads it was enabled for the crawl, or when the crawl’s preamble time budget ran out first. A probe that went out and got no answer, a refused connection or a timeout, counts the same. None of those says anything about the site, so the result is not-checked, never absent.

How to fix it

User-agent: *
License: https://example.com/license.xml

Put the directive in robots.txt and serve a valid RSL XML document at that URL. A Link: <https://example.com/license.xml>; rel="license" response header does the same job for consumers that never read robots.txt.

The common false positive is a licence document behind a redirect chain or a WAF that answers the audit differently from a browser. Check the reported status before assuming the file is missing.

Enable / disable

Disable this rule

[rules]
disable = ["ax/rsl-license"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/rsl-license"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• ax/content-signals: per-purpose AI policy in robots.txt, the lighter declaration.
• ax/pay-per-crawl: crawl pricing enforced at the edge rather than described in a licence.
• ax/ai-crawlers: the robots.txt blocks a cooperative crawler reads before fetching.
• legal/terms-of-service: the human-readable terms the licence sits alongside.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See AI crawlers for how an agent works through a report.

References

• Really Simple Licensing
• Robots Exclusion Protocol, RFC 9309

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. Each licence URL is listed with the status it returned. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.