All non-indexed pages: the full blocked list for review

What this list is for

Individual noindex directives are easy to justify one at a time and hard to audit in aggregate. A staging directive that survived a deploy, a robots.txt rule that is broader than intended, or a CMS default applied to a whole post type all look normal on the page and only become visible when you list every blocked URL together.

This rule is that list. It is informational by default, and it becomes a real finding only for URL patterns you nominate as ones that must stay indexable.

What squirrel checks

The rule is site-wide. For each crawled page it runs the shared indexability test over the parsed head, the response headers, the URL and robots.txt, collecting a reason list such as meta:noindex, header:noindex or robots.txt:disallowed. Every non-indexable page is bucketed by the patterns you configured:

• critical-noindex fails for pages matching errorOnPatterns. Message: 2 critical page(s) blocked from indexing (match error patterns), listing up to five as path plus reasons.
• important-noindex warns for pages matching warnOnPatterns. Message: 4 important page(s) blocked from indexing (match warning patterns), listing up to five.
• all-noindex reports info for everything else that is blocked. Message: 18 page(s) blocked from indexing, listing up to ten.
• all-noindex passes when every page is indexable. Message: All pages are indexable.
• all-noindex is skipped when there are no pages. Message: No pages to check.

Both pattern lists are empty by default, so an unconfigured run produces the info bucket when anything is blocked, a pass when nothing is, and a skip when there are no pages. Severity for the rule is info, weight 2.

How to fix it

[rule_options."crawl/all-noindex-pages"]
errorOnPatterns = ["/products/", "/blog/"]
warnOnPatterns = ["/guides/"]

Nominate the sections that must stay indexable, then treat any failure as a release blocker. For the pages themselves, remove the directive the reason names: meta:noindex is in the head, header:noindex is in the response, robots.txt:disallowed is a crawl rule rather than an index rule.

Options

Configuration example

[rule_options."crawl/all-noindex-pages"]
warnOnPatterns = ["/blog/"]
errorOnPatterns = ["/products/"]

Enable / disable

Disable this rule

[rules]
disable = ["crawl/all-noindex-pages"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/all-noindex-pages"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/indexability: the per-page version, with the directive named.
• crawl/indexability-conflicts: pages robots.txt blocks that carry no noindex.
• crawl/noindex-in-sitemap: blocked pages you are still submitting in a sitemap.
• crawl/robots-txt: the file that produces the robots.txt:disallowed reason.
• Pages kept out of search: the title, description, H1, duplicate and orphan rules that skip the meta:noindex and header:noindex pages listed here. A page blocked only by robots.txt is not skipped.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Robots meta tag and X-Robots-Tag, Google Search Central
• Introduction to robots.txt, Google Search Central

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Every blocked page is listed by path with the reason it is blocked. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.