Crawlability audit rules: robots.txt, sitemaps, indexing
Audit rules for crawling and indexing: robots.txt, XML sitemaps, noindex, canonical and redirect chains, soft 404s and pagination.
Crawlability rules cover the layer before ranking: whether a search engine can reach a page, is permitted to index it, and is being told the truth about it. They read robots.txt, every discovered sitemap, the robots directives in each page’s head and headers, and the redirect chains the crawl recorded. Most findings here are configuration mismatches rather than content problems, which makes them cheap to fix and easy to ship by accident.
Run only these rules
squirrel audit https://example.com --rule-include crawlWhere to start
These carry the highest severity and weight in the category.
- No XML sitemap found: where squirrel looks and what to add: Without a sitemap, discovery depends entirely on internal links. squirrel checks the common locations and robots.txt, and counts the URLs it found.
- Cross-domain sitemap URL: why search engines ignore it: A sitemap can only vouch for URLs on its own host. squirrel compares every sitemap entry’s hostname against your base URL.
- Rich result schema on a noindexed page: wasted markup: Structured data on a page search engines cannot index will never produce a rich result. squirrel finds pages carrying both.
- robots.txt: what to put in it and how squirrel checks it: A missing robots.txt, a Disallow: / left in production, or a syntax error changes what gets crawled. squirrel checks existence, syntax and rules.
- Sitemap validation: XML errors and the 50,000 entry limit: A sitemap that fails to parse or exceeds the protocol limits is ignored past the break. squirrel reports errors per file and checks entry counts.
Rules
| Rule | What it flags | Severity | Scope |
|---|---|---|---|
| 4XX pages in sitemap: dead URLs you are still submitting | A sitemap entry that returns 404 wastes a crawl and teaches search engines to trust the file less. squirrel reports the status of every checked entry. | warning | Site-wide |
| All non-indexed pages: the full blocked list for review | One list of every page a search engine will not index, with the reason. squirrel can raise severity for URL patterns that must never be blocked. | info | Site-wide |
| Canonical URL resolves through a redirect: how to spot it | A canonical that redirects sends search engines on an extra hop before the real URL. squirrel checks the tag against the page’s own redirect chain. | warning | Per-page |
| Cross-domain sitemap URL: why search engines ignore it | A sitemap can only vouch for URLs on its own host. squirrel compares every sitemap entry’s hostname against your base URL. | error | Site-wide |
| HTML document size: when a page is too big to crawl fully | A very large HTML document risks being read only in part. squirrel measures the response body and warns at 1MB, failing at 2MB by default. | error | Per-page |
| Indexable pages missing from the sitemap: measuring the gap | A page you want indexed that is in no sitemap depends entirely on link discovery. squirrel diffs the crawl against the sitemap in both directions. | warning | Site-wide |
| No XML sitemap found: where squirrel looks and what to add | Without a sitemap, discovery depends entirely on internal links. squirrel checks the common locations and robots.txt, and counts the URLs it found. | error | Site-wide |
| Noindexed page in sitemap: why it is a contradiction | A sitemap says crawl this, noindex says do not index it. squirrel cross-references every crawled page’s robots meta against your sitemap URLs. | warning | Site-wide |
| Page is blocked from indexing: finding the directive | A page can be excluded from search by a robots meta tag or an X-Robots-Tag header. squirrel reports which one, per page. | info | Per-page |
| Pagination canonicals: why page 2 is not page 1 | Canonicalizing every paginated page to page 1 hides the rest of the sequence. squirrel detects paginated URLs and checks where their canonical points. | info | Per-page |
| PDF size: large linked documents and what gets indexed | A very large PDF may only be read in part by a crawler. squirrel measures every internal PDF the crawl fetched and reports the outliers. | error | Site-wide |
| Redirect chains: when one hop turns into three | Each redirect hop costs latency and crawl budget, and search engines stop following after a few. squirrel counts the hops on every crawled URL. | warning | Site-wide |
| Redundant noindex: a blocked page nobody reads it on | A page disallowed in robots.txt is never fetched, so its noindex is never seen. squirrel lists the pages carrying both. | warning | Site-wide |
| Rich result schema on a noindexed page: wasted markup | Structured data on a page search engines cannot index will never produce a rich result. squirrel finds pages carrying both. | error | Site-wide |
| robots.txt blocked pages without noindex | Allow plus noindex is the correct way to deindex a page. squirrel only reports pages robots.txt blocks that carry no noindex, at info level. | warning | Site-wide |
| robots.txt: what to put in it and how squirrel checks it | A missing robots.txt, a Disallow: / left in production, or a syntax error changes what gets crawled. squirrel checks existence, syntax and rules. | error | Site-wide |
| Sitemap lastmod disagrees with the page’s dateModified | When lastmod and the page’s own dateModified disagree, one of them is wrong. squirrel compares them per URL and separates the two directions. | warning | Site-wide |
| Sitemap lastmod stamped at build time: how to detect it | If every lastmod carries the same date, the field records your deploy, not your content. squirrel counts distinct days across the sitemap. | warning | Site-wide |
| Sitemap validation: XML errors and the 50,000 entry limit | A sitemap that fails to parse or exceeds the protocol limits is ignored past the break. squirrel reports errors per file and checks entry counts. | error | Site-wide |
| Soft 404: error content served with an HTTP 200 status | A not-found page that answers 200 gets crawled and can be indexed as thin content. squirrel detects the error shell and re-checks it before reporting. | warning | Per-page |
Key terms
- AI crawler: A bot that fetches pages for AI companies, in three classes: training crawlers, AI search indexers and user-triggered fetchers, each with its own user-agent token.
- Canonical URL: The URL a page declares, with
<link rel="canonical">or a Link header, as the preferred address for its content. - Content freshness: How current a page is and how consistently it says so: visible dates, dateModified, sitemap lastmod and copyright year.
- Crawl budget: The number of URLs a search engine is willing and able to crawl on a site in a given time, set by crawl capacity and crawl demand.
- Googlebot file size limit: The byte limit after which Googlebot stops reading a fetched HTML file or PDF; content past it is not indexed.
- HTTP status code: The three-digit code a server returns with every response; 2xx success, 3xx redirect, 4xx client error, 5xx server error.
- Indexability: Whether a crawled URL is eligible to be indexed: reachable, a 200 response, not noindexed, not blocked and not canonicalised elsewhere.
- noindex: A robots directive, in a meta tag or the X-Robots-Tag header, that asks search engines to keep a page out of their index.
- Pagination: Splitting a long list across numbered URLs; each page should be crawlable and canonicalise to itself.
- Redirect chain: A sequence of two or more redirects between the requested URL and the final response.
- Rich results: Search result features (stars, breadcrumbs, product prices, video thumbnails) that Google can show when a page has eligible structured data.
- Robots meta tag: A
<meta name="robots">tag carrying page-level directives such as noindex, nofollow, nosnippet and max-snippet. - robots.txt: A plain-text file at the site root that tells crawlers which paths they may fetch, per user-agent token (RFC 9309).
- Sitemap lastmod: The optional
<lastmod>date on a sitemap URL entry, meant to say when the page content last changed. - Soft 404: A page that tells the visitor it does not exist (or is empty) while the server answers HTTP 200.
- Structured data: Machine-readable statements about a page’s content, usually schema.org vocabulary in JSON-LD, that search engines and agents can parse.
- X-Robots-Tag: An HTTP response header that carries the same directives as the robots meta tag, and works for non-HTML files such as PDFs.
- XML sitemap: An XML file listing the URLs a site wants crawled, with optional lastmod dates, referenced from robots.txt or submitted to search engines.
Related categories
Core SEO covers the canonical tag and the robots meta tag on each page. Links checks the internal links that crawlers follow, and Agent experience checks what AI crawlers may fetch under your robots.txt.
Learn more
- Fix SEO issues with AI, the coding agent workflow for working through a report
- Screaming Frog alternative, for crawling a site from the command line
- robots.txt generator
- Sample report, with the Crawlability section on a real site
Disable all Crawlability rules
[rules]
disable = ["crawl/*"]