Agent Experience

Agent Experience (AX) measures how ready a site is for AI agents — chat assistants, answer engines, and autonomous browsers — to read it, find their way around it, and act on it. AX is to agents what UX is to people and DX is to developers: a first-class audit pillar alongside SEO, Performance, and Security, not an afterthought bolted onto them.

Two things pushed AX from “nice to have” to something worth measuring in 2026: Cloudflare shipped an Agent Readiness score across its network, and Vercel published an Agent Readability spec. Both formalize what this category already tracked informally — whether a page is legible and reachable by something other than a browser with a human behind it.

The five pillars

AX splits into five questions, each covered by one or more rules.

Readable

Can an agent that fetches a URL actually get the content? Most agent fetchers — ChatGPT-User, Claude-User, and friends — are plain HTTP clients with no JavaScript engine, so anything that only exists after hydration is invisible to them. Readable also covers token weight: a page bloated with markup relative to its text costs an agent more tokens to parse for the same information.

• Content Without JavaScript — flags main content that only renders client-side
• Token Weight — per-page token estimate and text-to-HTML ratio
• Markdown Response — serving clean Markdown via content negotiation or a .md variant

Reachable

Are legitimate agents actually let in? robots.txt is the front door, and it has three different kinds of visitors to reason about: crawlers that train models (GPTBot, ClaudeBot, CCBot, Google-Extended), crawlers that back AI search results (OAI-SearchBot, Claude-SearchBot, PerplexityBot), and fetchers acting on a live user request (ChatGPT-User, Claude-User, Perplexity-User). These are separate user-agent tokens with separate policies — blocking ClaudeBot does not block Claude-User. Beyond declared policy, a WAF or bot-challenge page that 403s a verified agent breaks reachability regardless of what robots.txt says.

• AI Crawler Access — what robots.txt declares for each crawler class
• Agent Blocking — behavioral probe for WAF/challenge responses served to agent user-agents

Discoverable

Does the site expose a machine-readable entry point instead of making an agent guess? This is llms.txt and its neighbors: sitemaps, feeds, structured data, AGENTS.md for coding agents working in a repo or against docs, and the newer .well-known agent descriptors (MCP server cards, A2A agent cards, agent-skills manifests).

• llms.txt — curated Markdown map of the site’s key content
• AGENTS.md — repo/docs-oriented instructions file for coding agents
• Well-Known Agent Files — MCP server cards, A2A agent cards, agent-skills manifests

Declared

Has the site said anything explicit about AI access, or is it silent? A Content-Signal line in robots.txt, an RSL license, or a noai tag are all ways of declaring a position. Even “no” is good AX — an explicit block is a clear answer an agent can act on. Ambiguity, or worse, contradiction (declaring ai-train=yes while blocking GPTBot), is bad AX.

• Content Signals — Content-Signal policy lines in robots.txt
• noai Signals — noai/noimageai and AI-search-quoting opt-outs
• RSL License — machine-readable licensing terms via the RSL standard

Actionable

Can an agent go beyond reading and actually transact — call an API, use an MCP endpoint, authenticate, or pay for access? This is the frontier of the category: OpenAPI specs, OAuth discovery documents, and Cloudflare Pay Per Crawl / x402 monetized-access responses.

• API & OAuth Discovery — api-catalog, OpenAPI, OAuth discovery documents
• Pay Per Crawl — HTTP 402 monetized agent access (Cloudflare Pay Per Crawl, x402)
• Archive Indexing — presence in the Wayback Machine and Common Crawl, the archives AI training corpora are built from (cloud)

Rules

Roadmap

More AX checks are planned: WebMCP / agent-accessibility parity with Lighthouse agentic browsing.

Disable All Agent Experience Rules

[rules]
disable = ["ax/*", "ai/*"]