noindex in a robots meta tag or X-Robots-Tag header: what squirrel reports per page

What the robots meta tag does

<meta name="robots"> controls how a search engine treats one page. Absent, the default is index, follow: index the page and follow its links. noindex keeps the page out of the index. nofollow asks a search engine not to follow the page’s links or pass signals through them. Google documents it as a hint rather than a directive.

The X-Robots-Tag response header carries the same directives without touching the HTML, so it also works for PDFs and other files, and it can address one crawler by name (X-Robots-Tag: googlebot: noindex). It is often set by a server, CDN or framework rather than the page template.

Both directives have legitimate uses. A thank-you page after a form submission should be noindex. A page of user-submitted links might be nofollow. What matters is whether the directive is on the pages you meant, which is the question a crawl answers and a single page view does not.

What squirrel checks

The rule runs on every crawled page and reads two sources: the parsed robots meta value and the X-Robots-Tag response header. Both are lowercased before matching. It emits one check named robots-meta:

• Passing with the message No robots meta tag (defaults to index, follow) and a null value when there is neither a tag nor a header.
• Warning with the message Page is set to noindex and nofollow via <source> when both appear, or Page is set to noindex via <source> when only noindex does.
• At info status with the message Page is set to nofollow via <source> (links won't pass equity) when only nofollow appears. The wording is the report’s; Google treats the directive as a hint.
• Passing with the message Robots meta tag allows indexing, X-Robots-Tag header allows indexing or Robots meta tag and X-Robots-Tag header allow indexing when the sources are present and carry neither.

The source is robots meta tag, X-Robots-Tag header, or both joined with and. When noindex and nofollow come from different sources, each gets its own clause, as in Page is set to noindex via X-Robots-Tag header, nofollow via robots meta tag.

A header can scope its directives to one crawler, as in X-Robots-Tag: googlebot: noindex. The prefix applies to the directives after it until the next prefix, and the source then names the crawler: Page is set to noindex via X-Robots-Tag header for googlebot. Up to five crawlers are named, with a count of the rest. An unscoped directive in the same header covers every crawler, so the scoped ones are not listed. Prefixes such as max-snippet: and unavailable_after: are directive values, not crawler names.

The value is reported alongside the message. With no header it is the meta tag’s content, as before. A header-only page reports X-Robots-Tag: <header value>, and a page with both reports robots meta: <content>; X-Robots-Tag: <header value>.

The match is a substring test on the directives, so nofollow is also found inside nofollow-something and noindex is found inside any directive containing it. Crawler names in a header are not searched: X-Robots-Tag: noindexbot: index is not a noindex. crawl/indexability searches the whole header value instead, so it would count that one. Only <meta name="robots"> is read; a crawler-specific meta tag such as <meta name="googlebot"> is not. When a server sends X-Robots-Tag more than once, the values arrive joined with commas, so an unscoped header that follows a scoped one is read as part of the scoped group.

How to fix it

Remove the directive from pages that should be indexed:

<meta name="robots" content="index, follow" />

Omitting the tag entirely does the same thing and is one less place for a template to go wrong. Where the page really should stay out of the index, leaving it is correct, and the finding is a report rather than a defect.

When the report names the X-Robots-Tag header, the directive is in your server, CDN or framework configuration, not in the page source. Check the response headers with curl -I https://example.com/page and remove it where it is set.

The common false positive is a staging environment audited by mistake, where a site-wide noindex is exactly what should be there.

Enable / disable

Disable this rule

[rules]
disable = ["core/robots-meta"]

Disable all Core SEO rules

[rules]
disable = ["core/*"]

Enable only this rule

[rules]
enable = ["core/robots-meta"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/indexability: a noindex test over the meta tag and the whole header value, reported at info status.
• Pages kept out of search: the title, description, H1, duplicate and orphan rules that skip the noindex pages this rule reports.
• crawl/robots-meta-conflict: a robots meta that contradicts another signal on the page.
• crawl/noindex-in-sitemap: a noindex page listed in the sitemap.
• core/nosnippet: the snippet directives in the same tag.

Core SEO findings ship in every audit next to the performance, security and agent experience rules. See Fix SEO issues with AI for how an agent works through a report.

References

• Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, Google Search Central
• meta name=robots, MDN

Check your site

Run squirrel audit https://example.com and open the Core SEO section of the report. Each page reports the robots directives it declared. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.