squirrelscan
GitHub

Server errors: pages that answer 5xx

A page that answers 5xx is broken for visitors and drops out of search if it stays that way. squirrel reports each crawled page whose final response is a server error.

What a server error is

A 5xx status means the server failed to answer a request it should have been able to serve: an unhandled exception, an exhausted database or upstream, a bad deploy. Visitors see an error page, and search engines treat a persistent 5xx as a page that is gone and eventually drop it from the index.

What squirrel checks

The rule is site-wide and reads every page the crawl fetched.

  • Skipped when no pages were crawled. Message: No pages crawled.
  • Pass when no crawled page answered 5xx and the crawl reached every page it knew of. Message: No crawled page answered with a 5xx status.
  • Not checked when no crawled page answered 5xx but the crawl stopped at its page cap or with URLs still queued: a page it never fetched may still be failing, so the audit does not call the site clean, and an earlier finding stays open. Message: Not checked: the crawl stopped at its 500-page cap, so a page not crawled may still answer 5xx.
  • Not checked when no crawled page answered 5xx but the crawl gave up on some pages (a timeout, a refused connection, a body it could not read): one of those may still be failing, so an earlier finding stays open. Message: Not checked: the crawl could not fetch 3 page(s), and any of them may still answer 5xx.
  • server-errors fails with one item per page, labelled with the URL and its status code. Message: 2 crawled pages answer with a 5xx server error. Severity error, weight 8.

A redirecting URL is a redirect, not a page, so it is not counted here.

Retries and intermittent errors

squirrel retries a failing fetch before it gives up. A page that errors once and then answers is stored as the healthy page it became and is not reported, and the rule does not label anything intermittent. Only a page that is still 5xx after those retries is a finding. A throttled response (429, or a 503 that carries Retry-After) is the host asking squirrel to slow down, not a broken page, and is never reported here.

How to fix it

Read the server or application logs for the failing URL, fix the fault behind it, then re-run the audit to confirm the page returns 200. A page that is gone on purpose should answer 404 or 410, not 500.

Rule IDcrawl/server-errors
CategoryCrawlability
ScopeSite-wide
Severityerror
Weight8/10

Enable / disable

Disable this rule

[rules]
disable = ["crawl/server-errors"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/server-errors"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Check names by outcome

Each check name this rule can emit, by what it means to a tool comparing two audits. A problem check is a finding. A clean check says the rule ran over what it covers and found nothing; it clears the problem checks listed beside it, so a finding is fixed when the rule ran over the same scope and a clean check took its place. A not checked check says the rule did not evaluate, and never clears a finding.

Check nameOutcomeClears
server-errorsProblem
server-errorsCleanserver-errors
server-errorsNot checked

Type to search…

↑↓ navigate↵ openesc close