Custom 404 page: status, branding, navigation and noindex
A missing URL should answer 404 with a branded page that leads back into the site. On an active run squirrel requests one URL that cannot exist and checks the status and the page.
What a good 404 page is
When a visitor, a crawler or an agent requests a URL that does not exist, the site decides what happens next. A good answer has four parts: a real 404 (or 410) status, a page in the site’s own template rather than the web server’s stock text, a way back into the site, and a noindex signal.
The failures are different in kind. A missing URL that answers 200 is a soft 404: crawlers keep requesting it, it eats crawl budget and it can be indexed as thin duplicate content. A redirect of every missing URL to the homepage reads the same way. A bare 404 Not Found page from nginx or Apache is a dead end: no navigation, no search, nothing to click. A 404 template with no noindex is one misconfiguration away from being indexed.
What squirrel checks
The crawl cannot judge this. It stores no body for a 4xx response, and it only meets a 404 where the site links to a dead page. So the rule asks the site directly, with one GET to a random path that cannot exist, such as /squirrelscan-missing-page-3f9a1c0b2d7e4a68. It is a quiet probe: it runs when probing intensity is active or aggressive, which is the default for surface and full audits. The request goes to the audited origin only, carries no cookies, never follows a redirect, and reads at most 256 KB of the answer.
Soft 404 covers the other half: pages the crawl fetched that show a “not found” body with a 200. This rule covers the site’s answer to a path that was never linked, and the quality of the real 404 page.
Results
| Check | Status | When |
|---|---|---|
custom-404-soft | fail | the missing URL answered 2xx |
custom-404-redirect | warn | the missing URL answered 3xx |
custom-404-default | warn | the 404 page is a stock server page (nginx, Apache, IIS, Express, Spring Boot, Tomcat), or a page with no navigation, no search, no site links, no stylesheet and almost no text |
custom-404-no-route | warn | the 404 page is styled but has no link to another page on the site |
custom-404-indexable | warn | the 404 page has no noindex in a robots or googlebot meta tag or an X-Robots-Tag header |
custom-404-ok | pass | a 404 or 410 with a page that has a route back and is noindex |
custom-404-not-checked | skipped | the 404 handling was not observed |
A soft 404 or a redirect ends the check: there is no 404 page to judge. A bare default page is reported once, as a default page, not again as having no route back.
Not checked
The rule reports skipped, never pass, when no 404 response was observed:
- Probing is
passive, with reasonprobe-level-too-low. squirrel analyzere-reads stored data and sends nothing (analyze-offline).- The probe budget ran out, the request failed, or the site answered with throttling, a bot wall (401, 403, 429) or a server error (
probe-incomplete). - The 404 body is not HTML, such as a JSON API error.
A site whose error handling was never exercised is not credited.
How to fix it
- Return a real status for missing pages. In most frameworks that means throwing the framework’s not-found error from the route, not rendering a “not found” component with a
200. - Replace the server’s stock page. For nginx,
error_page 404 /404.html;with a page in your template. For Apache,ErrorDocument 404 /404.html. - Put the site header and navigation on the 404 page, add a search box if the site has search, and link to the homepage and the main sections.
- Add
<meta name="robots" content="noindex">to the 404 template, or sendX-Robots-Tag: noindex. - Check it:
curl -I https://example.com/some-missing-pageshould showHTTP/2 404.
| Rule ID | crawl/custom-404 |
| Category | Crawlability |
| Scope | Site-wide |
| Severity | warning |
| Weight | 4/10 |
Enable / disable
Disable this rule
[rules]
disable = ["crawl/custom-404"]Disable all Crawlability rules
[rules]
disable = ["crawl/*"]Enable only this rule
[rules]
enable = ["crawl/custom-404"]enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.
Check names by outcome
Each check name this rule can emit, by what it means to a tool comparing two audits. A problem check is a finding. A clean check says the rule ran over what it covers and found nothing; it clears the problem checks listed beside it, so a finding is fixed when the rule ran over the same scope and a clean check took its place. A not checked check says the rule did not evaluate, and never clears a finding.
| Check name | Outcome | Clears |
|---|---|---|
custom-404-soft | Problem | |
custom-404-redirect | Problem | |
custom-404-default | Problem | |
custom-404-no-route | Problem | |
custom-404-indexable | Problem | |
custom-404-ok | Clean | custom-404-soft, custom-404-redirect, custom-404-default, custom-404-no-route, custom-404-indexable |
custom-404-not-checked | Not checked |
Related rules
- crawl/soft-404: crawled pages that serve error content with a 200.
- crawl/indexability: pages excluded from search by a robots directive.
- crawl/redirect-chain: redirect hops and loops.