Agent experience (AX) audit rules: llms.txt, AI crawlers

Agent experience is how well a site works for something other than a browser with a person behind it: a chat assistant, an answer engine, or an autonomous agent. These rules ask whether such a client can read the content, reach the site at all, discover an entry point, understand the policy the site declares, and act on what it finds. Agent experience is scored as one of the four groups on every report, alongside SEO, performance and security.

The five pillars

AX splits into five questions, each covered by one or more rules.

Readable

Readable asks whether an agent that fetches a URL gets the content. Most agent fetchers, such as ChatGPT-User and Claude-User, are plain HTTP clients with no JavaScript engine, so anything that only exists after hydration is invisible to them. Readable also covers token weight: a page bloated with markup relative to its text costs an agent more tokens to parse for the same information.

• Content Without JavaScript: flags main content that only renders client-side
• Token Weight: per-page token estimate and text-to-HTML ratio
• Markdown Response: serving clean Markdown via content negotiation or a .md variant

Reachable

Reachable asks whether legitimate agents are let in. robots.txt is the front door, and it has three different kinds of visitors to reason about: crawlers that train models (GPTBot, ClaudeBot, CCBot, Google-Extended), crawlers that back AI search results (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and fetchers acting on a live user request (ChatGPT-User, Claude-User, Perplexity-User). These are separate user-agent tokens with separate policies: blocking ClaudeBot does not block Claude-User. Beyond declared policy, a WAF or bot-challenge page that 403s a verified agent breaks reachability regardless of what robots.txt says.

• AI Crawler Access: what robots.txt declares for each crawler class
• Agent Blocking: behavioral probe for WAF/challenge responses served to agent user-agents

Discoverable

Discoverable asks whether the site exposes a machine-readable entry point instead of making an agent guess. This is llms.txt and its neighbors: sitemaps, feeds, structured data, AGENTS.md for coding agents working in a repo or against docs, and the newer .well-known agent descriptors (MCP server cards, A2A agent cards, agent-skills manifests).

• llms.txt: curated Markdown map of the site’s key content
• AGENTS.md: repo and docs-oriented instructions file for coding agents
• Well-Known Agent Files: MCP server cards, A2A agent cards, agent-skills manifests

Declared

Declared asks whether the site has said anything explicit about AI access. A Content-Signal line in robots.txt, an RSL license, or a noai tag are all ways of declaring a position. Even “no” is good AX: an explicit block is a clear answer an agent can act on. Ambiguity, or worse, contradiction (declaring ai-train=yes while blocking GPTBot), is bad AX.

• Content Signals: Content-Signal policy lines in robots.txt
• noai Signals: noai/noimageai and AI-search-quoting opt-outs
• RSL License: machine-readable licensing terms via the RSL standard

Actionable

Actionable asks whether an agent can go beyond reading and transact: call an API, use an MCP endpoint, authenticate, or pay for access. This is the newest part of the category, covering OpenAPI specs, OAuth discovery documents, and HTTP 402 monetized-access responses.

• API & OAuth Discovery: api-catalog, OpenAPI, OAuth discovery documents
• Pay Per Crawl: HTTP 402 monetized agent access (Cloudflare Pay Per Crawl, x402)
• Archive Indexing: presence in the Wayback Machine and Common Crawl, the archives AI training corpora are built from (cloud)

Run only these rules

squirrel audit https://example.com --rule-include ax

ax also selects the four ai/ rules, which belong to this category. Three of them are cloud rules that run only when you are signed in, and AI content detection is retired and never runs. Content Without JavaScript and Archive Indexing need a cloud audit too.

Where to start

These carry the highest severity and weight in the category.

• Blocking AI agents at the edge: what robots.txt hides: A WAF or bot-management rule serving GPTBot or Claude-User a challenge the browser never sees. squirrel probes the homepage as all three.
• Page type mismatch: when schema disagrees with the page: A page that reads as a listing but declares Product schema. squirrel compares the classified type against the declared JSON-LD.
• Content-Signal in robots.txt: search, ai-input, ai-train: A Content-Signal line with a typo, or one declaring ai-train=yes while robots.txt blocks the training crawlers. squirrel parses and cross-checks both.
• Text to HTML ratio: the token cost of a page for an agent: Visible text under 15% of the HTML, or a page over a 100,000-token budget. squirrel estimates both from the raw response.
• AI crawlers in robots.txt: which bots you block, by class: robots.txt blocking Claude-User or PerplexityBot alongside the training bots. squirrel classifies all 24 known AI crawlers and names what each block costs.

Rules

Key terms

• Agent experience (AX): How well a site serves AI agents that read it, find their way around it and act on it, the way UX serves people.
• AGENTS.md: A Markdown file of instructions for coding agents working in a repository or against a docs site.
• AI crawler: A bot that fetches pages for AI companies, in three classes: training crawlers, AI search indexers and user-triggered fetchers, each with its own user-agent token.
• AI-generated content: Text written by a language model; Google’s position is that how content is made matters less than whether it is helpful. The related rule is retired and does not run.
• Client-side rendering: Building page content in the browser with JavaScript; agents and many fetchers that do not run JavaScript see an empty page.
• Content Signals: Content-Signal lines in robots.txt that state whether content may be used for search, AI input or AI training.
• Generative engine optimization (GEO): Making content more likely to be retrieved, cited and quoted by AI answer engines; the term comes from a 2023 research paper.
• llms.txt: A proposed Markdown file at /llms.txt that gives language models a curated map of a site’s key content.
• Markdown for agents: Serving a clean Markdown version of a page, through Accept: text/markdown or a .md URL, so agents read fewer tokens.
• noai and noimageai: Meta robots values some sites use to ask AI systems not to train on their text or images; not part of any standard search engines honour.
• Pay per crawl: Charging AI crawlers for access with HTTP 402 Payment Required responses, as in Cloudflare Pay Per Crawl and the x402 protocol.
• robots.txt: A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).
• RSL (Really Simple Licensing): A standard for publishing machine-readable licensing terms for AI use of content, referenced from robots.txt.
• Well-known URIs: Fixed paths under /.well-known/ (RFC 8615) where sites publish machine-readable files: security.txt, api-catalog, OAuth metadata and agent manifests.

Related categories

Crawlability checks robots.txt, sitemaps and indexing for all crawlers. Performance covers page speed, and Structured data covers the JSON-LD that agents and search engines parse.

Learn more

• AI crawlers guide
• llms.txt guide
• Generative engine optimization guide
• llms.txt validator
• robots.txt generator
• Sample report, with the Agent Experience section on a real site

Roadmap

More AX checks are planned: WebMCP / agent-accessibility parity with Lighthouse agentic browsing.

Disable all Agent Experience rules

[rules]
disable = ["ax/*", "ai/*"]