URL: /glossary/noindex

---
title: "What does noindex do? Meta tag, header and conflicts"
description: "noindex is a robots rule, in a meta tag or the X-Robots-Tag header, that keeps a page out of search results. How it works and why robots.txt breaks it."
sidebarTitle: "noindex"
---

**noindex** is a robots rule, set in a `<meta name="robots">` tag or an `X-Robots-Tag` response header, that asks search engines to keep a page out of their results.

Google documents both forms. The header also works for files that have no HTML, such as PDFs, and Google does not support a `noindex` line in robots.txt. A noindexed page can still be crawled, and its links can still be followed unless `nofollow` is added.

{/* glossary:hub:start */}

Category: [Crawlability](/rules/crawl)

{/* glossary:hub:end */}

## Why it matters

For `noindex` to work, the crawler has to fetch the page and read it. Google says the page must not be blocked by robots.txt: if it is, the rule is never seen, and the URL can still appear in results when other pages link to it. Blocking a page in robots.txt and adding `noindex` is a common way to get this wrong.

The other failure is a `noindex` nobody meant to ship. A staging setting left in a production template, or a header set at the CDN, removes pages from search without any visible change to the page.

## How squirrelscan checks it

[core/robots-meta](/rules/core/robots-meta) reads the robots meta tag and the `X-Robots-Tag` header on every page and names the directive and where it was set, including a header scoped to one crawler such as `googlebot`. [crawl/indexability](/rules/crawl/indexability) reports each non-indexable page at info level with its source. [crawl/all-noindex-pages](/rules/crawl/all-noindex-pages) lists every blocked page across the site, and fails for URL patterns you nominate as ones that must stay indexable.

Three rules catch contradictions. [crawl/robots-meta-conflict](/rules/crawl/robots-meta-conflict) lists pages that robots.txt blocks and that also carry `noindex`, so the rule is never read. [crawl/noindex-in-sitemap](/rules/crawl/noindex-in-sitemap) lists noindexed pages that a sitemap still submits; it reads the meta tag only. [crawl/schema-noindex-conflict](/rules/crawl/schema-noindex-conflict) flags rich result structured data on a page that cannot be indexed.

{/* glossary:rules:start */}

| Rule | What it flags | Severity |
|---|---|---|
| [noindex in a robots meta tag or X-Robots-Tag header: what squirrel reports per page](/rules/core/robots-meta) | A page declaring noindex or nofollow in its robots meta tag or X-Robots-Tag header. squirrel reads both on every crawled page and names the directive and where it was set. | warning |
| [Page is blocked from indexing: finding the directive](/rules/crawl/indexability) | A page can be excluded from search by a robots meta tag or an X-Robots-Tag header. squirrel reports which one, per page. | info |
| [All non-indexed pages: the full blocked list for review](/rules/crawl/all-noindex-pages) | One list of every page a search engine will not index, with the reason. squirrel can raise severity for URL patterns that must never be blocked. | info |
| [Noindexed page in sitemap: why it is a contradiction](/rules/crawl/noindex-in-sitemap) | A sitemap says crawl this, noindex says do not index it. squirrel cross-references every crawled page's robots meta against your sitemap URLs. | warning |
| [Rich result schema on a noindexed page: wasted markup](/rules/crawl/schema-noindex-conflict) | Structured data on a page search engines cannot index will never produce a rich result. squirrel finds pages carrying both. | error |
| [Redundant noindex: a blocked page nobody reads it on](/rules/crawl/robots-meta-conflict) | A page disallowed in robots.txt is never fetched, so its noindex is never seen. squirrel lists the pages carrying both. | warning |

{/* glossary:rules:end */}

## How to fix it

To keep a page out of search, let it be crawled and say `noindex`:

```html
<meta name="robots" content="noindex" />
```

For a PDF or another file, send `X-Robots-Tag: noindex` from the server. Remove any `Disallow` that covers the same path ([the conflict fix](/rules/crawl/robots-meta-conflict#how-to-fix-it)), and drop the URL from the sitemap ([the sitemap fix](/rules/crawl/noindex-in-sitemap#how-to-fix-it)). When a page should be indexed and the report names the header, the directive is in your server or CDN configuration, so check it with `curl -I` ([the robots meta fix](/rules/core/robots-meta#how-to-fix-it)). Use `errorOnPatterns` to make sections that must stay indexable fail loudly ([the list fix](/rules/crawl/all-noindex-pages#how-to-fix-it)).

## Run the check

{/* glossary:run:start */}

```bash
squirrel audit https://example.com --rule-include core/robots-meta,crawl/indexability,crawl/all-noindex-pages,crawl/noindex-in-sitemap,crawl/schema-noindex-conflict,crawl/robots-meta-conflict
```

{/* glossary:run:end */}

A signed-out local audit costs nothing.

{/* glossary:related:start */}

## Related terms

- [Robots meta tag](/glossary/robots-meta-tag): A `<meta name="robots">` tag carrying page-level directives such as noindex, nofollow, nosnippet and max-snippet.
- [X-Robots-Tag](/glossary#x-robots-tag): An HTTP response header that carries the same directives as the robots meta tag, and works for non-HTML files such as PDFs.
- [robots.txt](/glossary/robots-txt): A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).
- [Indexability](/glossary/indexability): Whether a crawled URL is eligible to be indexed: reachable, a 200 response, not noindexed, not blocked and not canonicalised elsewhere.

{/* glossary:related:end */}

## References

- [Block Search indexing with noindex, Google Search Central](https://developers.google.com/search/docs/crawling-indexing/block-indexing)
- [Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- [Introduction to robots.txt, Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/intro)