PDF size: large linked documents and what gets indexed

What the PDF size question is

Search engines index PDFs as documents in their own right, extracting text, title and links. Like HTML, a PDF is read up to a limit rather than in full, and content past that point does not contribute to what the document ranks for.

Google documents a 64 MB fetch limit for PDFs, against 2 MB for any other individual URL. squirrel’s thresholds sit just under it: 30 MB for a warning and 60 MB for a failure, both configurable. Either way, a 60 MB PDF is a slow download for a person as well as a crawler, and is usually a scanned document that could be compressed or split.

What squirrel checks

The rule is site-wide and reads PDF sizes the crawl fetched ahead of time. When no size data is present it first looks for internal PDF links, meaning links ending in .pdf whose origin matches the site base URL.

• Skipped when no PDF sizes were fetched and no pages were crawled. Message: No pages crawled. A non-empty size list is processed regardless of the page count.
• Pass when there is no size data and no internal PDF link exists. Message: No internal PDF links found.
• Skipped when internal PDF links exist but no sizes were fetched. Message: PDF size data not available.
• pdf-size fails at or above error_bytes, 60 MB by default. Message: 2 PDF(s) exceed the 60MB error threshold, each listed with its size in megabytes. Severity error, weight 4.
• pdf-size-warn warns at or above warn_bytes, 30 MB by default. Message: 3 PDF(s) exceed 30MB, approaching the 60MB error threshold.
• pdf-size passes when every measured PDF is under the warning threshold. Message: 14 PDF(s) checked, all under 30MB.
• pdf-size-errors reports info for PDFs whose size could not be fetched. Message: Could not check 2 PDF(s).

The messages name the 60 MB error threshold, which is the rule’s own default and not a Google figure. The limit Google documents for PDFs is 64 MB.

How to fix it

X-Robots-Tag: noindex

Compress the images inside the document, split a long report into chapters, or serve an HTML version and keep the PDF as a download. Where a PDF does not need to appear in search at all, send X-Robots-Tag: noindex on its response, which works for non-HTML files that cannot carry a meta tag.

Options

Configuration example

[rule_options."crawl/pdf-size"]
warn_bytes = 10485760
error_bytes = 26214400

Enable / disable

Disable this rule

[rules]
disable = ["crawl/pdf-size"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/pdf-size"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/html-size: the same measurement applied to HTML documents.
• crawl/indexability: the X-Robots-Tag header this rule’s fix uses.
• links/broken-links: PDF links that no longer resolve.
• content/mime-type: documents served with the wrong content type.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Inside Googlebot: crawling, fetching and the bytes we process, Google Search Central
• Robots meta tag and X-Robots-Tag, Google Search Central

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Every measured PDF over the threshold is listed with its size. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.