Common Crawl and Wayback: is your site in the archives
A site missing from the Common Crawl index or the Wayback Machine, or archived only with a capture over a year old. squirrel checks both.
What archive indexing is
Two archives sit upstream of most AI training data. Common Crawl runs its own crawler, CCBot, and publishes a monthly corpus that the major labs build training sets from. The Internet Archive’s Wayback Machine captures pages through ia_archiver and also ingests Common Crawl’s crawls.
They are not the same thing and neither implies the other. A site can have Wayback snapshots and no Common Crawl presence, because the ingest goes one way. Common Crawl has no submission form: inclusion follows from being crawlable and linked, and the next crawl is monthly.
What squirrel checks
The rule runs once per crawl and looks the domain up in both archives. The lookup is a cloud service, so the check needs a cloud audit. It emits one or two checks:
archive-indexing, skipped with the messageArchive indexing check skippedwhen the lookup did not run or returned nothing, with the reason naming why.common-crawl-indexed: passes with the messageSite is in the Common Crawl index, adding(<source>)when the record names the crawl it came from. It warns withSite is in the Common Crawl index but the latest capture is over a year old — recent content is missing from CC-derived AI training datawhen the newest capture is more than 365 days before the lookup. With no record it warns withSite is not in the Common Crawl index — Common Crawl feeds the training corpora of the major AI labs, so the site is invisible to models trained on it.wayback-archived: passes with the messageSite is archived in the Wayback Machine. It warns withSite is in the Wayback Machine but the latest snapshot is over a year oldpast the same 365-day threshold, and withSite has no Wayback Machine snapshot — the Internet Archive has never captured it, a strong signal the archive crawlers can't reach or discover itwhen there is no snapshot at all.
The staleness threshold is 365 days, measured from the time the lookup was captured. When that timestamp is missing or unparseable the rule falls back to the current time, and a capture with no date is never called stale.
How to fix it
Check that robots.txt does not block the archive crawlers:
User-agent: CCBot
Allow: /
User-agent: ia_archiver
Allow: /
Then request a Wayback snapshot at web.archive.org/save. Common Crawl has no equivalent: fix crawlability, make sure something links to the site, and wait for the next monthly crawl.
The common false positive is a site that launched recently. Common Crawl runs monthly and the Internet Archive captures on its own schedule, so a new domain can be absent from both while nothing is wrong with it.
| Rule ID | ax/archive-indexing |
| Category | Agent Experience |
| Scope | Site-wide |
| Severity | warning |
| Weight | 1/10 |
Enable / disable
Disable this rule
[rules]
disable = ["ax/archive-indexing"]Disable all Agent Experience rules
[rules]
disable = ["ax/*"]Enable only this rule
[rules]
enable = ["ax/archive-indexing"]enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.
Related rules
- ax/ai-crawlers: whether robots.txt blocks
CCBotoria_archiverin the first place. - ax/agent-blocking: edge rules that block crawlers robots.txt allows.
- crawl/robots-txt: whether robots.txt exists and parses at all.
- crawl/indexability: the directives that keep pages out of any index.
Agent experience findings ship in every audit next to the SEO, performance and security rules. See AI crawlers for how an agent works through a report.
References
Check your site
Run squirrel audit https://example.com and open the Agent Experience section of the report. Each archive shows its latest capture date. Cloud audits run the lookup.