crawl

Crawl a website without running analysis

The crawl command crawls a website and stores the data without running audit rules. Use this to separate crawling from analysis, or to crawl first and analyze later.

Usage

squirrel crawl <url> [options]

Arguments

Argument	Description
`url`	The URL to crawl (required)

Options

Option	Alias	Description	Default
`--max-pages`	`-m`	Maximum pages to crawl (hard cap 10,000)	the level’s page budget (`quick` = 25)
`--concurrency`		Global crawl worker pool size (overrides `[crawler] concurrency`)	`5`
`--per-host`		Max concurrent requests per host (overrides `[crawler] per_host_concurrency`)	`5`
`--level`		Audit level: `quick`, `surface`, `full`	`quick`
`--coverage`	`-C`	Older name for `--level`, still accepted (`fast` means `quick`)
`--refresh`	`-r`	Ignore cache, fetch all pages fresh	`false`
`--fresh-ua`		Re-roll the project’s pinned random user-agent (the new one is pinned for later runs)	`false`
`--resume`		Resume interrupted crawl	`false`

--concurrency and --per-host also suppress the localhost fast path (see Crawler Configuration): against your own dev server, crawl otherwise raises concurrency and drops the per-host delay automatically.

Audit levels

A crawl takes two settings from its audit level: the page budget and how pages are picked.

Level	Pages	How pages are picked
`quick`	25	The URL and its sitemaps, no link following
`surface`	100	Follows links, one page per URL pattern first
`full`	500	Every page it can reach, up to the budget

Unlike audit, whose default level is auth-aware, crawl always defaults to quick. A plain squirrel crawl stops after 25 pages; 500 is the full level’s page budget, not the default.

The page budget resolves as --max-pages / -m > non-default [crawler] max_pages > the level’s page budget, capped at 10,000 pages either way. Switch levels with --level (or [crawler] coverage in config), or override the budget directly with --max-pages. fast is accepted as quick; any other value, or a --max-pages that is not a positive number, stops the crawl before it starts. See Crawling for how each level picks pages.

Examples

Basic crawl

squirrel crawl https://example.com

Comprehensive crawl

squirrel crawl https://example.com --level full

Crawl more pages

squirrel crawl https://example.com -m 1000

Fresh crawl (ignore cache)

squirrel crawl https://example.com --refresh

Resume interrupted crawl

squirrel crawl https://example.com --resume

Crawl behavior

The crawl command:

Fetches and stores HTML content for each page
Extracts and follows internal links (surface/full; quick sticks to the seed URL + sitemap URLs, only falling back to link discovery when the site has no sitemap)
Discovers sitemaps (including via robots.txt); robots.txt rules aren’t enforced by default (respect_robots = true to opt in)
Deduplicates URLs automatically
Caches page content locally

Output

Crawling: https://example.com
Level: quick · max 25 pages

✓ Crawled 18 pages in 6.3s

Crawl ID: a7b3c2d1

After crawling, use squirrel analyze to run audit rules on the stored data.

Exit codes

Code	Meaning
`0`	Success
`1`	Error (invalid URL, crawl failed, etc.)

Configuration

The crawl command respects settings from squirrel.toml:

[crawler]
coverage = "full"
max_pages = 200
delay_ms = 200
timeout_ms = 30000
include = ["/blog/*"]
exclude = ["/admin/*"]

Note that max_pages = 100 (the literal config default) is treated as unset and the level’s page budget applies instead; any other value overrides it. See Crawler Configuration for all options.

Workflow

# 1. Crawl the site
squirrel crawl https://example.com

# 2. Analyze the crawl
squirrel analyze

# 3. View the report
squirrel report

This workflow is useful when:

You want to crawl once and analyze multiple times
Testing different rule configurations
Crawling is slow and you want to iterate on analysis

analyze - Analyze stored crawl
audit - Crawl + analyze in one command
Configuration - Config file options

Edit this page