Text to HTML ratio: the token cost of a page for an agent

What text-to-HTML ratio is

An agent that fetches a page pays for every byte of it in tokens, including the parts a reader never sees: inline scripts, inline styles, wrapper divs, and long utility-class strings. The text-to-HTML ratio is the share of the response that is visible text. A page at 5% is spending nineteen tokens on markup for every one on content.

The second number is absolute size. A response large enough to fill a meaningful part of a context window is a problem regardless of its ratio, because the agent has to hold all of it to find the part it wanted.

What squirrel checks

The rule runs on every crawled page against the raw HTML and emits two checks:

• token-weight-ratio: warns with the message Visible text is under 15% of the page HTML — agents pay token cost mostly for markup, scripts, and styles when the ratio is below 0.15. The per-page number is in the item, which reads ~N% of HTML is visible text (~N est. tokens). Otherwise it passes with ~N% of this page's HTML is visible text, above the 15% floor.
• token-weight-budget: warns with the message Raw HTML exceeds a generous 100,000-token budget for a single page fetch — too heavy for an agent's context window when the estimate is over 100,000 tokens, with the item reading ~N estimated tokens (N bytes). Otherwise it passes with ~N estimated tokens (N bytes), within budget.

Instead of both, a single token-weight check is emitted as skipped with the message Page too small (or not a real content page) to assess token weight when the page has no parsed document or its HTML is under 1,024 bytes. The rule is also skipped on a page detected as a soft 404, whose numbers describe the error template rather than a real page.

The token estimate is the UTF-8 byte length of the HTML divided by four and rounded. It is a heuristic, not any model’s tokenizer, and it is there to flag order-of-magnitude bloat rather than to bill anyone. The two warning messages carry the fixed thresholds but no per-page numbers: the report groups checks by message, and differing digits would collapse the shared headline into a literal N. The per-page figures live in the item instead.

How to fix it

Move inline scripts and styles into external files, which also lets them cache:

<link rel="stylesheet" href="/app.css" />
<script src="/app.js" defer></script>

Then look at the markup around the content itself. Deeply nested wrapper divs and hundred-character class strings on every paragraph are the other half of the ratio. Serving a Markdown representation, covered by ax/markdown-response, removes the markup from the response entirely.

The common false positive is a page that is mostly a data table or a media gallery, where the markup-to-text ratio is inherent to the content. The finding is still true about token cost, but there may be nothing to cut.

Enable / disable

Disable this rule

[rules]
disable = ["ax/token-weight"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/token-weight"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• ax/markdown-response: serving Markdown, which removes the markup from the response.
• ax/content-without-js: the opposite failure, where the heavy HTML carries no content at all.
• perf/total-byte-weight: the same size question measured for a browser rather than an agent.
• crawl/html-size: the crawl-side view of oversized HTML documents.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See Generative engine optimization for how an agent works through a report.

References

• How large DOM sizes affect interactivity, web.dev
• text/markdown media type, RFC 7763

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. Each page lists its ratio and estimated tokens. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.