Redundant noindex: a blocked page nobody reads it on

What a redundant noindex is

When robots.txt disallows a path, a compliant crawler does not request it. Any directive in that page’s HTML, including noindex, is therefore never read.

The combination is common because both look like ways to keep a page out of search. They are not interchangeable. Disallow prevents the fetch; noindex removes the page from the index after a fetch. Applying both means the second has no effect, and the URL can still appear in results as a bare link discovered elsewhere.

What squirrel checks

The rule is site-wide. For each crawled page it tests whether robots.txt blocks the URL, then looks for noindex in the robots meta tag or the X-Robots-Tag response header.

squirrel decides whether robots.txt blocks a URL the way Google does: it reads the group for Googlebot, or the * group when there is none, takes the longest Allow or Disallow rule whose path matches (an Allow wins a tie), treats * in a rule as any run of characters and a trailing $ as the end of the URL, and matches against the path with its query string, with percent-encoding normalised on both sides (/café matches /caf%C3%A9). crawl/robots-meta-conflict and crawl/indexability-conflicts share this one decision, so they always agree on which pages are blocked.

• Skipped when there is no robots.txt or no pages. Message: Insufficient data to check for conflicts.
• redundant-noindex reports info for pages that are both blocked and carry noindex, from the meta tag or the header. Message: {count} page(s) blocked in robots.txt also have noindex, with noindex won't be seen if page is blocked from crawling as the value. The pages are listed on the finding.
• robots-conflict passes. Message: No robots meta/robots.txt conflicts detected.

The robots-conflict check always passes in the current implementation: its warn branch is never reached, so the only finding this rule produces is the informational one. Severity for the rule is warning, weight 5.

A page robots.txt allows that carries noindex is not reported by this rule: that is the supported way to keep a page out of search results. (crawl/all-noindex-pages still counts noindex pages across the site.)

How to fix it

# robots.txt: remove the Disallow so the noindex is readable
User-agent: *
Allow: /archive/

Pick one. If the goal is keeping the page out of results, allow the crawl and keep noindex. If the goal is saving crawl budget on a section nobody should fetch, keep the Disallow and drop the meta tag.

Enable / disable

Disable this rule

[rules]
disable = ["crawl/robots-meta-conflict"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/robots-meta-conflict"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/indexability-conflicts: pages robots.txt blocks that carry no noindex.
• crawl/robots-txt: the file this rule reads its rules from.
• crawl/all-noindex-pages: every blocked page with the reason it is blocked.
• crawl/noindex-in-sitemap: noindexed pages you are still submitting.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Robots meta tag and X-Robots-Tag, Google Search Central
• How Google interprets the robots.txt specification

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Each page carrying both directives is counted and named. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.