llms.txt: what the file is and how squirrel validates it

What llms.txt is

llms.txt is a Markdown file at a site’s root that points AI agents and answer engines at the content worth reading. The convention from llmstxt.org is an H1 with the site or project name, a short summary, then sections of Markdown links to key pages. A companion llms-full.txt inlines that content rather than linking to it.

It is a proposal, not a standard, and no crawler is obliged to fetch it. It costs one static file and it gives an agent a curated entry point instead of a sitemap of every URL you have. Absence is never a defect here.

What squirrel checks

The rule runs once per crawl against the root /llms.txt, with /.well-known/llms.txt and /docs/llms.txt as fallbacks probed by the crawler. It emits one or two checks:

• llms-txt: reports at info status with the message /llms.txt not checked: the discovery probe was not sent or got no answer and the value not-checked when the root request did not run or got no answer. When the site refused the request (a 401, 403 or 429, or a bot challenge) the message is /llms.txt not checked: the site refused the request (HTTP 403 (Cloudflare)) and the value is refused. The rule is skipped when the crawl fetched no page and a root request was refused or failed. If one of the alternate paths is a real hit, it reports llms-txt-present at info status with Found llms.txt at <path> (/llms.txt at the root was not checked) instead.
• llms-txt-present: fails with the message /llms.txt returned 200 but the body is an HTML page (SPA fallback) — not a real llms.txt when the file exists and its first 512 characters, trimmed and lowercased, start with <!doctype html or <html. With no root file but a real hit at one of the two alternate paths, it reports at info status with No /llms.txt at the root, but found one at <path>. With nothing anywhere it warns with No /llms.txt found — consider adding one so AI agents can discover your key content. With a real root file it reports at info status with /llms.txt found (N bytes), adding ; /llms-full.txt also present when that exists too.
• llms-txt-format: emitted only when a root file was found. It passes at info status with Valid llms.txt format (N links), or warns with llms.txt format notes: followed by the problems joined with ; . The three possible notes are missing H1 title (should start with `# Name`), no Markdown links found and N malformed link URL(s).

An alternate-path hit counts only when the probe returned 200, the body did not sniff as HTML, and the content read as Markdown. The format check tests that the first non-empty line matches an H1 with text after the hash, counts every [text](url) link, and tries to parse each link URL against the site’s base URL. The rule’s severity is info, so a warn check surfaces as a recommendation and does not move the score. The SPA-fallback result is the exception: a file that claims to be curated content and is not is a fail.

A probe goes unsent when discovery probes are turned off, when no rule that reads it was enabled for the crawl, or when the crawl’s preamble time budget ran out first. A probe that went out and got no answer, a refused connection or a timeout, counts the same. None of those says anything about the site, so the result is not-checked, never absent.

How to fix it

# Acme Docs

> Deployment and configuration guides for Acme.

## Guides
- [Quickstart](https://example.com/docs/quickstart): install and first deploy.
- [Configuration](https://example.com/docs/config): every option, with defaults.

Serve it at https://example.com/llms.txt with a text/plain or text/markdown content type.

The common false positive is a single-page application whose server returns index.html for every unmatched path. The file appears to exist with a 200 status, which is why the rule sniffs the body and reports the fallback rather than reporting a valid file.

Enable / disable

Disable this rule

[rules]
disable = ["ax/llms-txt"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/llms-txt"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• ax/agents-md: the repository-facing equivalent, which is only recommended to sites that already publish llms.txt.
• ax/markdown-response: serving Markdown for the pages themselves rather than a curated index.
• ax/well-known-agent: the machine-readable manifests an agent looks for under .well-known.
• crawl/sitemap-exists: the equivalent entry point for search crawlers.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See llms.txt for how an agent works through a report.

References

• The /llms.txt file proposal
• text/markdown media type, RFC 7763

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. The file size, link count and any format note are listed together. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.