What is indexability? Why a page is not indexed
Indexability is whether a crawled URL can be indexed: reachable, a 200 response, not noindexed, not blocked. Why a page is not indexed and how to check.
Indexability is whether a URL a search engine has found can go into its index, which depends on whether the crawler may fetch it, the status code it answers with, any noindex it carries and where its canonical points.
Each condition has its own mechanism. robots.txt decides whether the URL is fetched, though a blocked URL can still be indexed from links without its content. The status code decides whether there is content to consider: Google ignores the body of a 4xx or 5xx response. A robots meta tag or X-Robots-Tag header with noindex removes the page after the fetch, and a rel="canonical" pointing elsewhere asks for another URL to be indexed instead. Indexable is not the same as indexed. Google says the content of a 200 page may be indexed, but that is not guaranteed.
Category: Crawlability
Why it matters
A page that cannot be indexed cannot rank, however good it is. Most non-indexable pages are deliberate: thank-you pages, internal search results, filter combinations, staging routes. The problems are the unplanned ones, such as a noindex left in a template, a robots.txt rule broader than intended, or a CMS default applied to a whole post type. Each looks normal on one page and only shows when you list every blocked URL together.
How squirrelscan checks it
crawl/indexability checks every page for noindex in the robots meta tag and then the X-Robots-Tag header, and reports a blocked page at info level with the source. It describes rather than penalises. crawl/all-noindex-pages collects every non-indexable page in one list with its reasons, such as meta:noindex, header:noindex or robots.txt:disallowed. Configure errorOnPatterns or warnOnPatterns and blocked pages under those paths fail or warn.
Two rules compare robots.txt with the page. crawl/robots-meta-conflict lists pages blocked in robots.txt that also say noindex, which is never read. crawl/indexability-conflicts lists blocked pages with no noindex, which can still be listed from links elsewhere with no description. Status codes and canonicals have rules of their own, on the HTTP status code and canonical URL pages.
| Rule | What it flags | Severity |
|---|---|---|
| Page is blocked from indexing: finding the directive | A page can be excluded from search by a robots meta tag or an X-Robots-Tag header. squirrel reports which one, per page. | info |
| robots.txt blocked pages without noindex | Allow plus noindex is the correct way to deindex a page. squirrel only reports pages robots.txt blocks that carry no noindex, at info level. | warning |
| All non-indexed pages: the full blocked list for review | One list of every page a search engine will not index, with the reason. squirrel can raise severity for URL patterns that must never be blocked. | info |
| Redundant noindex: a blocked page nobody reads it on | A page disallowed in robots.txt is never fetched, so its noindex is never seen. squirrel lists the pages carrying both. | warning |
How to fix it
Decide per section which pages belong in search, then make the signals agree:
[rule_options."crawl/all-noindex-pages"]
errorOnPatterns = ["/products/", "/blog/"]That turns a stray noindex on a product or post into a failing finding (the list fix). Remove the directive from the source the report names (the indexability fix). To keep a page out of results, allow the crawl and use noindex. Keep Disallow for paths that should not be fetched at all (the conflict fix).
Run the check
squirrel audit https://example.com --rule-include crawl/indexability,crawl/indexability-conflicts,crawl/all-noindex-pages,crawl/robots-meta-conflictA signed-out local audit costs nothing.
Related terms
- noindex: A robots directive, in a meta tag or the X-Robots-Tag header, that asks search engines to keep a page out of their index.
- robots.txt: A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).
- Canonical URL: The URL a page declares, with
<link rel="canonical">or a Link header, as the preferred address for its content. - HTTP status code: The three-digit code a server returns with every response; 2xx success, 3xx redirect, 4xx client error, 5xx server error.