HTML document size: when a page is too big to crawl fully

What the HTML size limit is

Crawlers do not read an unlimited amount of markup. Google documents that Googlebot fetches up to 2 MB from any individual URL other than a PDF, response headers included, and stops there. The bytes past the cutoff are not fetched, not rendered and not indexed. Referenced resources such as stylesheets and scripts have their own separate per-URL counter and do not consume the document’s allowance.

Most pages are nowhere near it. A page gets there through inline base64 images, large blocks of inline CSS or JavaScript, or a serialised framework payload, and what falls off the end is whatever came last in the file.

squirrel’s default failure threshold is that same 2 MB, with a warning at 1 MB so a page is flagged before it starts losing content.

What squirrel checks

The rule runs on every crawled page. It measures the UTF-8 byte length of the response HTML and emits a single check named html-size:

• Fail at or above error_bytes, 2 MB by default. Message: HTML is 2.4MB — exceeds Googlebot 2MB limit, content will be truncated, with the byte count as the value and the threshold as the expected value. Severity error, weight 5.
• Warn at or above warn_bytes, 1 MB by default. Message: HTML is 1180KB — approaching Googlebot 2MB limit.
• Pass otherwise. Message: HTML size OK: 84KB.

Both thresholds are configurable. The default failure threshold matches the 2 MB per-URL limit Google documents for Googlebot.

How to fix it

const catalogue = await fetch("/data/catalogue.json").then((r) => r.json());

Move large inline blobs out of the document and fetch them instead. A <script type="application/json"> block only holds inline data, so an external file has to be requested rather than referenced with src. Serialised state, base64 images, inline SVG sprites and inline CSS are the usual culprits. Paginate or lazy-load long lists rather than rendering ten thousand rows.

Keep the title, meta description, headings and main body near the top of the file. If anything is ever cut, it should be the parts that do not describe the page.

Options

Configuration example

[rule_options."crawl/html-size"]
warn_bytes = 512000
error_bytes = 1048576

Enable / disable

Disable this rule

[rules]
disable = ["crawl/html-size"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/html-size"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/pdf-size: the same measurement applied to linked PDF documents.
• content/word-count: how much of that markup is actually text.
• crawl/soft-404: pages whose body is an error shell rather than content.
• perf/dom-size: the parsed cost of the same document.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Inside Googlebot: crawling, fetching and the bytes we process, Google Search Central
• Managing crawl budget for large sites, Google Search Central

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Every oversized page is listed with its exact byte count. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.