What is llms.txt? Format, adoption and what it is not
llms.txt is a proposed Markdown file at /llms.txt that points language models at a site's key content. The format, who reads it, and what to check.
llms.txt is a proposed Markdown file, served at /llms.txt, that gives language models and AI agents a curated map of a site’s most useful content.
The proposal at llmstxt.org was published in September 2024. A file starts with an H1 naming the site or project, the only required section, followed by a blockquote summary and sections of Markdown links to key pages. The file can also sit under a path, such as /docs/llms.txt. A companion llms-full.txt inlines the content instead of linking to it.
Category: Agent Experience
Why it matters
llms.txt is a proposal, not a standard, and no crawler has to fetch it. It costs one static file and gives an agent a short list of entry points instead of a sitemap of every URL. The proposal’s changelog says documentation platforms now generate one automatically and coding agents use them. Google Search does not use it: Google says you do not need AI text files or Markdown to appear in Search or its AI features, and that publishing one for other services is fine. A missing llms.txt is never a defect.
How squirrelscan checks it
ax/llms-txt requests /llms.txt at the root, with /.well-known/llms.txt and /docs/llms.txt as fallbacks. It fails a file that returns 200 but is really an HTML page, the fallback a single-page app serves for every unknown path. For a real root file it checks the format: an H1 title on the first non-empty line, at least one Markdown link, and link URLs that parse. It also notes when /llms-full.txt exists. The rule is info severity, so a missing file or a format note does not move the score.
| Rule | What it flags | Severity |
|---|---|---|
| llms.txt: what the file is and how squirrel validates it | No /llms.txt at the root, an SPA fallback served in its place, or a file with no H1 and no links. squirrel checks the file and its format. | info |
How to fix it
Write a short file by hand, or generate one from your docs, and serve it at the root with a text/plain or text/markdown content type (see the llms.txt fix):
# Acme Docs
> Deployment and configuration guides for Acme.
## Guides
- [Quickstart](https://example.com/docs/quickstart): install and first deploy.
- [Configuration](https://example.com/docs/config): every option, with defaults.If your server returns index.html for unmatched paths, add a real file or a route, or the audit will report the HTML fallback.
Run the check
squirrel audit https://example.com --rule-include ax/llms-txtA signed-out local audit costs nothing.
Related terms
- Agent experience (AX): How well a site serves AI agents that read it, find their way around it and act on it, the way UX serves people.
- AGENTS.md: A Markdown file of instructions for coding agents working in a repository or against a docs site.
- Markdown for agents: Serving a clean Markdown version of a page, through Accept: text/markdown or a .md URL, so agents read fewer tokens.