robots.txt blocked pages without noindex

What an indexability conflict is

robots.txt controls crawling and a robots meta tag controls indexing. They are different mechanisms applied at different moments.

A page robots.txt allows that carries noindex is not a conflict. It is the correct way to keep a page out of the index: the crawler has to fetch the page to read the directive, so the crawl must be allowed. squirrel does not report it, including when the directive comes from an X-Robots-Tag header.

A page robots.txt blocks that carries no noindex is reported. The crawler never fetches it, so nothing on the page can influence indexing, and the URL can still be listed from links elsewhere with no description.

A page robots.txt blocks that also carries noindex is a real conflict, because the directive is never seen. That case is reported only by crawl/robots-meta-conflict, so nothing is reported twice.

What squirrel checks

The rule is site-wide. For each crawled page it tests robots.txt against the URL and runs the shared indexability test over the page’s head and headers.

squirrel decides whether robots.txt blocks a URL the way Google does: it reads the group for Googlebot, or the * group when there is none, takes the longest Allow or Disallow rule whose path matches (an Allow wins a tie), treats * in a rule as any run of characters and a trailing $ as the end of the URL, and matches against the path with its query string, with percent-encoding normalised on both sides (/café matches /caf%C3%A9). crawl/robots-meta-conflict and crawl/indexability-conflicts share this one decision, so they always agree on which pages are blocked.

• Skipped when there is no robots.txt or no pages. Message: Insufficient data (no robots.txt or pages).
• robots-block-without-noindex reports info for pages robots.txt blocks that are otherwise indexable. Message: 12 page(s) blocked by robots.txt without noindex. The affected URLs are listed on the finding.
• conflicts passes when no page is blocked without noindex. Message: No indexability conflicts detected.

Severity is warning and the weight is 4, but the only check this rule reports is informational.

How to fix it

# robots.txt: let the crawler in so it can read the directive
User-agent: *
Allow: /private-preview/

<meta name="robots" content="noindex">

To remove a page from results, allow the crawl and add noindex. A Disallow on the same path defeats the directive.

When the goal is only saving crawl budget on pages nobody should fetch, Disallow alone is the right tool, and the URL can still be listed without a description. Pick based on which of the two you want: noindex removes a page from results, Disallow stops the fetch.

Enable / disable

Disable this rule

[rules]
disable = ["crawl/indexability-conflicts"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/indexability-conflicts"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/robots-meta-conflict: the blocked-plus-noindex case, which is reported only there.
• crawl/all-noindex-pages: the full list of blocked pages with their reasons.
• crawl/robots-txt: the syntax and rules of the file being compared.
• crawl/indexability: the per-page verdict for one URL.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Robots meta tag and X-Robots-Tag, Google Search Central
• Introduction to robots.txt, Google Search Central

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Each conflicting page is listed by path, split by which conflict it is. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.