Robots meta tag: directives, defaults and conflicts
The robots meta tag sets page-level rules for search engines: noindex, nofollow, nosnippet, max-snippet. The defaults, the header form and conflicts.
The robots meta tag is a <meta name="robots"> element in a page’s head that gives search engines page-level rules, such as noindex, nofollow, nosnippet and max-snippet.
With no tag, the default is no restriction: Google’s documentation names it all, the same as index, follow. Rules are case-insensitive and can be combined in one comma-separated value. name="robots" addresses every crawler, and a name such as googlebot addresses one. The X-Robots-Tag response header carries the same rules for any response, including files that have no HTML.
Category: Core SEO
Why it matters
The tag decides what search does with a page after crawling it. noindex keeps it out of results. nofollow asks the crawler not to follow its links, and Google treats that as a hint. nosnippet removes the text snippet under the result, and max-snippet:0 does the same by setting the allowed length to zero. A positive max-snippet only caps the length.
The tag only works when the page can be fetched. robots.txt decides whether a crawler requests the page at all, so a page blocked there never has its meta tag read. The two are different tools: robots.txt controls crawling and the meta tag controls indexing.
How squirrelscan checks it
core/robots-meta reads the robots meta tag and the X-Robots-Tag header on every page. It warns on noindex, notes nofollow at info level, and says which source set each directive. It does not read crawler-specific tags such as <meta name="googlebot">. core/nosnippet warns on nosnippet and on the exact form max-snippet:0, and notes any other max-snippet value. It reads the meta tag only, not the header.
Two rules compare the tag with robots.txt, deciding blocking the way Google does, with the longest matching rule winning. crawl/robots-meta-conflict lists pages that are blocked and also carry noindex. crawl/indexability-conflicts lists blocked pages that carry no noindex, which can still be listed in results from links elsewhere.
| Rule | What it flags | Severity |
|---|---|---|
| noindex in a robots meta tag or X-Robots-Tag header: what squirrel reports per page | A page declaring noindex or nofollow in its robots meta tag or X-Robots-Tag header. squirrel reads both on every crawled page and names the directive and where it was set. | warning |
| nosnippet and max-snippet:0: pages with no search snippet | A robots directive that stops search engines showing any description under the result. squirrel reads the robots meta on every page. | warning |
| Redundant noindex: a blocked page nobody reads it on | A page disallowed in robots.txt is never fetched, so its noindex is never seen. squirrel lists the pages carrying both. | warning |
| robots.txt blocked pages without noindex | Allow plus noindex is the correct way to deindex a page. squirrel only reports pages robots.txt blocks that carry no noindex, at info level. | warning |
How to fix it
Leave the tag out on pages that should be indexed with a normal snippet, or state the defaults:
<meta name="robots" content="index, follow" />
<meta name="robots" content="max-snippet:160" />The second line caps the snippet instead of removing it (the snippet fix). Remove noindex from pages meant for search, and look for it in server or CDN headers when the report names X-Robots-Tag (the robots meta fix). To remove a page from results, allow the crawl in robots.txt and add noindex (the robots.txt conflict fix).
Run the check
squirrel audit https://example.com --rule-include core/robots-meta,core/nosnippet,crawl/robots-meta-conflict,crawl/indexability-conflictsA signed-out local audit costs nothing.
Related terms
- noindex: A robots directive, in a meta tag or the X-Robots-Tag header, that asks search engines to keep a page out of their index.
- X-Robots-Tag: An HTTP response header that carries the same directives as the robots meta tag, and works for non-HTML files such as PDFs.
- robots.txt: A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).