What does noindex do? Meta tag, header and conflicts
noindex is a robots rule, in a meta tag or the X-Robots-Tag header, that keeps a page out of search results. How it works and why robots.txt breaks it.
noindex is a robots rule, set in a <meta name="robots"> tag or an X-Robots-Tag response header, that asks search engines to keep a page out of their results.
Google documents both forms. The header also works for files that have no HTML, such as PDFs, and Google does not support a noindex line in robots.txt. A noindexed page can still be crawled, and its links can still be followed unless nofollow is added.
Category: Crawlability
Why it matters
For noindex to work, the crawler has to fetch the page and read it. Google says the page must not be blocked by robots.txt: if it is, the rule is never seen, and the URL can still appear in results when other pages link to it. Blocking a page in robots.txt and adding noindex is a common way to get this wrong.
The other failure is a noindex nobody meant to ship. A staging setting left in a production template, or a header set at the CDN, removes pages from search without any visible change to the page.
How squirrelscan checks it
core/robots-meta reads the robots meta tag and the X-Robots-Tag header on every page and names the directive and where it was set, including a header scoped to one crawler such as googlebot. crawl/indexability reports each non-indexable page at info level with its source. crawl/all-noindex-pages lists every blocked page across the site, and fails for URL patterns you nominate as ones that must stay indexable.
Three rules catch contradictions. crawl/robots-meta-conflict lists pages that robots.txt blocks and that also carry noindex, so the rule is never read. crawl/noindex-in-sitemap lists noindexed pages that a sitemap still submits; it reads the meta tag only. crawl/schema-noindex-conflict flags rich result structured data on a page that cannot be indexed.
| Rule | What it flags | Severity |
|---|---|---|
| noindex in a robots meta tag or X-Robots-Tag header: what squirrel reports per page | A page declaring noindex or nofollow in its robots meta tag or X-Robots-Tag header. squirrel reads both on every crawled page and names the directive and where it was set. | warning |
| Page is blocked from indexing: finding the directive | A page can be excluded from search by a robots meta tag or an X-Robots-Tag header. squirrel reports which one, per page. | info |
| All non-indexed pages: the full blocked list for review | One list of every page a search engine will not index, with the reason. squirrel can raise severity for URL patterns that must never be blocked. | info |
| Noindexed page in sitemap: why it is a contradiction | A sitemap says crawl this, noindex says do not index it. squirrel cross-references every crawled page’s robots meta against your sitemap URLs. | warning |
| Rich result schema on a noindexed page: wasted markup | Structured data on a page search engines cannot index will never produce a rich result. squirrel finds pages carrying both. | error |
| Redundant noindex: a blocked page nobody reads it on | A page disallowed in robots.txt is never fetched, so its noindex is never seen. squirrel lists the pages carrying both. | warning |
How to fix it
To keep a page out of search, let it be crawled and say noindex:
<meta name="robots" content="noindex" />For a PDF or another file, send X-Robots-Tag: noindex from the server. Remove any Disallow that covers the same path (the conflict fix), and drop the URL from the sitemap (the sitemap fix). When a page should be indexed and the report names the header, the directive is in your server or CDN configuration, so check it with curl -I (the robots meta fix). Use errorOnPatterns to make sections that must stay indexable fail loudly (the list fix).
Run the check
squirrel audit https://example.com --rule-include core/robots-meta,crawl/indexability,crawl/all-noindex-pages,crawl/noindex-in-sitemap,crawl/schema-noindex-conflict,crawl/robots-meta-conflictA signed-out local audit costs nothing.
Related terms
- Robots meta tag: A
<meta name="robots">tag carrying page-level directives such as noindex, nofollow, nosnippet and max-snippet. - X-Robots-Tag: An HTTP response header that carries the same directives as the robots meta tag, and works for non-HTML files such as PDFs.
- robots.txt: A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).
- Indexability: Whether a crawled URL is eligible to be indexed: reachable, a 200 response, not noindexed, not blocked and not canonicalised elsewhere.