Indexable pages missing from the sitemap: measuring the gap

What sitemap coverage is

Coverage is the overlap between the pages a crawl reaches and the URLs a sitemap lists. Two gaps are possible and they mean different things.

A crawled, indexable page that is not in the sitemap relies on internal links to be found. That is usually fine for a well-linked page and a real problem for one buried several levels deep. A sitemap URL the crawl never reached is the reverse: either the page is orphaned from the site’s own navigation, or the crawl stopped before reaching it.

What squirrel checks

The rule is site-wide. It uses the sitemap processor’s precomputed missing and orphan lists, and falls back to computing the missing list itself when that is empty. Pages marked noindex, by robots meta tag or X-Robots-Tag header, are never listed as missing, because a noindex page should not be in the sitemap. This applies only when the site itself is indexable (the homepage is not noindex), so a staging host that is noindex everywhere is still reported from the sitemap processor’s list (the fallback has always left meta-noindex pages out, and still does); see pages kept out of search. The fallback also skips pages whose status is not 200, then checks both the page URL and its final URL against the normalised sitemap URLs.

• Skipped when no sitemap was discovered. Message: No sitemap to compare.
• Skipped when no pages were crawled. Message: No pages to compare.
• sitemap-coverage warns when indexable pages are missing. Message: 23 indexable page(s) not in sitemap (18%), where the percentage is of all crawled pages. It passes otherwise with All indexable pages are in sitemap. Severity warning, weight 5.
• sitemap-orphans warns for sitemap URLs the crawl did not reach. Message: N sitemap URL(s) were not crawled.
• sitemap-orphans reports info instead when the crawl hit its page cap and the sitemap holds more URLs than that cap. Message: N sitemap URL(s) not audited this run (crawl capped at M pages).

N is the exact count, even past the 10,000-URL list the audit engine hands the rule. The check lists the first 50 URLs; details.total holds the count, and past 50, additional and pagesTruncated record the rest.

The capped case matters because a shallow crawl profile against a large sitemap would otherwise report a crawl-budget artefact as a coverage defect.

How to fix it

Generate the sitemap from the same route data the site renders from, filtered to indexable pages that return 200, and regenerate it on deploy.

For sitemap URLs the crawl could not reach, check whether anything on the site links to them. A page that exists only in the sitemap is discoverable but not navigable, which is a different problem from a missing sitemap entry.

Enable / disable

Disable this rule

[rules]
disable = ["crawl/sitemap-coverage"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/sitemap-coverage"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/sitemap-exists: whether there is a sitemap to compare against.
• crawl/noindex-in-sitemap: the reverse mistake, submitting pages you have blocked.
• links/orphan-pages: pages with too few inbound internal links to be found.
• crawl/sitemap-4xx: listed URLs that resolve to an error.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Build and submit a sitemap, Google Search Central
• Sitemaps overview, Google Search Central

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Both gaps are listed by URL, with the missing-page share as a percentage. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.