4XX pages in sitemap: dead URLs you are still submitting
A sitemap entry that returns 404 wastes a crawl and teaches search engines to trust the file less. squirrel reports the status of every checked entry.
What a 4XX sitemap entry is
A sitemap lists URLs you are asking a crawler to fetch. An entry that answers with a 4XX status is a request you asked for and cannot serve: 404 for a page that no longer exists, 403 for one behind auth, 410 for one deliberately removed.
Each one costs a fetch that could have gone to a real page, and a sitemap with many of them is a weaker signal overall. The usual causes are a generator reading a stale cache, a content deletion that never propagated to the sitemap, and URLs built from data that has since changed.
What squirrel checks
The rule is site-wide and reads the status codes the crawl recorded for sitemap URLs. It emits up to three checks:
- Skipped when no sitemap was discovered. Message:
No sitemap to compare. sitemap-4xxpasses when no sitemap URL statuses were collected. Message:No sitemap URL checks available.sitemap-4xxwarns when entries returned a status from 400 to 499. Message:9 sitemap URL(s) return 4XX, with each URL listed alongside its status and any fetch error. Severity warning, weight 6.sitemap-4xxpasses otherwise. Message:No 4XX pages found in sitemap URLs checked.sitemap-rate-limitedreports info for entries the host throttled. Message:5 sitemap URL(s) rate-limited - status unverifiable.sitemap-4xxis skipped when asquirrel audit --offlinerun had only sitemap URLs on another site to check. Message:No sitemap URL on the audited site to check.sitemap-off-site-not-checkedreports info when asquirrel audit --offlinerun left sitemap URLs on another site unchecked. Message:{count} sitemap URL(s) on another site not checked this audit.
Rate-limit responses are separated deliberately. A 429, a 430 or a 503 carrying Retry-After sits inside or near the 4XX band but says the host throttled the audit, not that the URL is dead, so those entries are excluded from the warn list and reported on their own.
How to fix it
Remove the dead entries and regenerate from live data. Where a URL moved, redirect it and list the destination instead: a sitemap should hold final URLs, not redirects.
A rate-limited run leaves entries unverified rather than failing them. Re-run with a lower concurrency if you need a definite answer for those URLs.
| Rule ID | crawl/sitemap-4xx |
| Category | Crawlability |
| Scope | Site-wide |
| Severity | warning |
| Weight | 6/10 |
Enable / disable
Disable this rule
[rules]
disable = ["crawl/sitemap-4xx"]Disable all Crawlability rules
[rules]
disable = ["crawl/*"]Enable only this rule
[rules]
enable = ["crawl/sitemap-4xx"]enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.
Related rules
- crawl/sitemap-coverage: the reverse gap, pages missing from the sitemap.
- crawl/sitemap-valid: structural problems in the same files.
- links/broken-links: internal links that point at the same dead URLs.
- crawl/soft-404: pages that return 200 while showing an error, which this check cannot see.
Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.
References
Check your site
Run squirrel audit https://example.com and open the Crawlability section of the report. Every failing entry is listed with the status code it returned. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.