Storage Configuration

The [storage] section controls how much of a project’s audit history squirrelscan keeps locally, and how large the content store that holds crawled pages can grow.

Configuration

[storage]
keep_audits = 3
content_store_max_bytes = "4GB"

Options

keep_audits

Type: number or boolean Default: 3

How many audits of this project stay fully renderable. After a successful audit, anything older than the newest keep_audits is retired: its per-check results are deleted and its report can no longer be opened.

The same pass runs at the end of squirrel analyze, the step that finishes an audit when you run crawl, analyze and report separately, so a project audited that way keeps the same bound as one audited with squirrel audit. It uses the keep_audits of the squirrel.toml in the directory you run it from, and only for the project that file names (with no [project] name, only for the latest crawl’s project), so analyzing another project’s crawl with --id never cuts its history. An analysis of a crawl that was blocked or failed retires nothing, as with squirrel audit.

Crawls left behind by audits that were killed before they finished (a watchdog, an out-of-memory kill, a deploy) are retired too, once a newer audit of the project has finished and the crawl has had no activity (no page fetched) for 30 minutes. stopped and paused crawls are what squirrel audit --resume continues, and Ctrl-C leaves one, so they wait 7 days idle instead. A failed crawl follows the same rule as the rest: retired once a newer audit has finished and it is idle, otherwise left alone, and it never takes a slot of the window. Abandoned crawls never count toward keep_audits and never take a slot from a finished audit. A crawl newer than the newest finished audit, or one still fetching pages, is never touched, because it may be a live audit or a crawl waiting for squirrel analyze.

While a command runs on a crawl it writes a lease on it (the process id, host name and a heartbeat every 15 seconds), cleared when the command ends. Retention reads it: a crawl with a live lease is never retired, even when a later audit finished first, and a crawled or running crawl whose owner was killed is retired without waiting for a later audit (after 5 minutes and 24 hours of silence). See self disk for the exact rules.

Every audit writes a fresh set of rows, so without a window a project grows by about one audit each time you run it. On a 1,000-page site that is roughly 95 MB per audit, forever. Three is the default because it covers squirrel report --diff against the previous run, and one before that as a fallback when the most recent audit is the odd one out.

[storage]
keep_audits = 10   # keep more history
keep_audits = 1    # keep only the audit that just ran
keep_audits = 0    # turn retention off (also: false)
keep_audits = false

Retention runs only after a successful audit, and a failed one does not count towards the window either. A run that ends failed or blocked, such as a site that is down or answers 403 to everything, neither deletes anything nor takes a place among the audits you are keeping. A week of downtime cannot fill the window with runs that learned nothing and then take the history you would use to find out when it broke.

An audit says so whenever it retires something, on stderr. The pointer to this setting appears the first time a project retires anything and not afterwards:

Retired 1 older audit and freed 95.2 MB inside project.db for the next audit; keep more with [storage] keep_audits (now 3)
Retired 1 older audit and freed 95.2 MB inside project.db for the next audit

The notice never goes to stdout, so it cannot get into -f json output.

content_store_max_bytes

Type: number (bytes) or a size string such as "4GB" Default: 1GB Environment variable: SQUIRREL_CONTENT_STORE_MAX_BYTES (overrides the config for one run)

The size cap for the content store, ~/.squirrel/content-store.db. Every page an audit crawls is stored there gzipped, and the rules and reports read it back from there. When the store grows past 90% of the cap, the least recently used pages are evicted until it is back at 80%.

[storage]
content_store_max_bytes = "8GB"        # a size string: KB, MB, GB, TB (binary, 1 GB = 1024³ bytes)
content_store_max_bytes = 8589934592   # or a whole number of bytes

SQUIRREL_CONTENT_STORE_MAX_BYTES=8GB squirrel audit https://example.com

The cap counts compressed bytes. HTML usually compresses five to ten times, so the default holds roughly 5 to 10 GB of pages, a few thousand pages of a heavy storefront.

An audit never evicts its own pages. While an audit or crawl runs, every page it stores or reads is protected from eviction, including by another audit running at the same time. A crawl larger than the cap therefore finishes with all of its pages, and the store can sit above the cap until the next audit, which evicts the least recently used pages back down to it before it stores anything.

The store is shared. It belongs to the machine, not to a project, so the cap in force is the one set by whatever is auditing. To use one cap everywhere, set it in ~/squirrel.toml or export the environment variable rather than setting it per project.

Eviction is reported, not silent. Evicting an audit’s pages after it finished does not change its findings, which are stored separately, but anything rebuilt from page HTML is missing for those pages. When that happens squirrelscan says how many pages are affected:

• squirrel report on an audit whose pages were evicted since it ran.
• A re-audit, which requests evicted pages again in full instead of reusing them as unchanged, and says how many.
• squirrel audit --resume and squirrel crawl --resume, when pages the interrupted crawl stored are gone. Run without --resume to fetch them again.

Content store: 812 of 4,000 pages stored for this audit are no longer in the local content store, most likely evicted by its size cap (now 1.0 GB). Findings are unaffected; anything rebuilt from page HTML is missing for them. Raise [storage] content_store_max_bytes or SQUIRREL_CONTENT_STORE_MAX_BYTES to keep more.

squirrel self doctor shows the store’s size, the cap and the last eviction.

What retiring an audit does and does not take

A retired audit stays listed. squirrel report --list still shows it, marked retired with the date its data went, rather than the history quietly shrinking. What it can no longer do is open:

Audit data was reclaimed on 2026-09-08: 06148d76-6c48-465a-bc16-deb44144350d

report <id>, report --diff, report --regression-since and analyze all refuse a retired audit that way. If you keep an older audit as a reference baseline, raise keep_audits or turn it off before it falls out of the window.

Re-auditing stays incremental. Retiring never takes the newest page record for a url, the sub-resource sizes, or the links and images a cached page is reused with, so the next audit still answers every unchanged page from its conditional GET rather than refetching the site. Measured on a 40-page site over five audits, the audits after retirement fetched 0 of 40 pages.

Smart audits are unaffected either way: carried findings are keyed by site, not by crawl.

Retention and reclaiming are different things

Retention deletes rows. SQLite keeps the freed pages in the file and reuses them for the next audit, so the file stops growing rather than shrinking, and an audit does not pay to rewrite it. That is deliberate: at keep_audits = 3 an audit retires one audit’s worth of rows every time, and rebuilding the whole database that often would cost far more than the space it returns.

To hand the space back to the filesystem, rebuild the file explicitly:

squirrel self disk                       # what every project is using
squirrel self disk --prune --keep 3      # retire, then rebuild

See squirrel self disk for the full command.

See also

• Configuration Overview
• squirrel self disk - report and reclaim disk usage
• Reports - diffing and comparing audits