Crawler Settings

The [crawler] section controls how squirrelscan discovers and fetches pages.

Configuration

[crawler]
max_pages = 100
delay_ms = 100
timeout_ms = 30000
concurrency = 5
per_host_concurrency = 5
per_host_delay_ms = 50
max_backoff_ms = 300000
include = []
exclude = []
allow_query_params = []
drop_query_prefixes = ["utm_", "gclid", "fbclid"]
respect_robots = false
disable_discovery_probes = false
incremental = true
breadth_first = true
max_prefix_budget = 0.25
user_agent = ""
follow_redirects = true

Crawl limits

max_pages

Type: number Default: 100 Range: 1 to 10000 (capped by CLI)

Maximum number of pages to crawl per audit. The literal config default is 100, but when max_pages isn’t explicitly set the effective budget follows the audit level: quick = 25, surface = 100, full = 500. The default level is auth-aware: any signed-in account (free or Pro) defaults to surface, signed-out runs default to quick (see Audit levels). A value other than 100 changes the level’s page budget, and the audit becomes custom.

Examples:

Small site:

[crawler]
max_pages = 50

Large site:

[crawler]
max_pages = 2000

CLI override:

squirrel audit https://example.com -m 100

The resolution order is --max-pages / -m > non-default [crawler] max_pages > the level’s page budget (quick 25, surface 100, full 500). max_pages = 100 reads as unset because squirrel init writes it into every new config.

Note: The CLI enforces a hard cap (currently 10,000 pages) regardless of config or plan. Signed-out local audits cost nothing, and this cap never changes based on your account. When a crawl stops because it hit the limit, the CLI prints a hint naming --max-pages and the cap; see Hitting the page limit.

Cloud audits are different: an audit dispatched from the dashboard (on-demand or scheduled) reads the pages its website’s level asks for, lowered to your plan’s ceiling when it asks for more - see max pages per cloud audit and Limits.

coverage

Type: string Default: unset (auth-aware for audit; always quick for crawl) Values: quick, surface, full (fast is read as quick)

The default audit level for crawls and audits. The key keeps its older name. When unset, audit picks the level by auth state (surface signed in, quick signed out) while crawl always uses quick. The level sets the page budget (quick 25, surface 100, full 500), how pages are picked, the cloud checks, rendering, external link checks and probing. Any other value fails config validation with the list of levels.

[crawler]
coverage = "full"

CLI override:

squirrel audit https://example.com --level surface

max_depth

Type: number Default: unset (unlimited)

Maximum crawl depth measured in link hops from the start page (seed and sitemap URLs are depth 0; a link found on a depth-N page is depth N+1). When set, the crawler never follows links past this depth: max_depth = 1 audits the start page plus the pages it links to directly. Unset leaves crawling depth-unbounded (the page budget still applies).

[crawler]
max_depth = 2

CLI override:

squirrel audit https://example.com --max-depth 2

The resolution order is --max-depth > [crawler] max_depth > unlimited.

timeout_ms

Type: number Default: 30000 (30 seconds) Range: 1000 to 60000 recommended

Timeout for each page request in milliseconds.

Examples:

Fast timeout for quick sites:

[crawler]
timeout_ms = 10000  # 10 seconds

Slow sites or APIs:

[crawler]
timeout_ms = 45000  # 45 seconds

When request exceeds timeout:

• Page marked as failed
• Crawl continues with next URL
• Logged in error output

Rate limiting

delay_ms

Type: number Default: 100 (100ms)

Base delay between requests in milliseconds.

Examples:

Fast crawl (be careful):

[crawler]
delay_ms = 50

Polite crawl:

[crawler]
delay_ms = 500

No delay (local development only):

[crawler]
delay_ms = 0

Note: Actual delays depend on per_host_delay_ms and concurrency settings.

per_host_delay_ms

Type: number Default: 50 (50ms)

Minimum delay between consecutive request starts to the same host. Request starts stagger independently, so up to per_host_concurrency requests can be in flight at once while still spacing new requests apart for politeness.

A Crawl-delay directive in the target’s robots.txt overrides this value only when respect_robots is true, and is capped at 2 seconds even then.

Examples:

Very polite:

[crawler]
per_host_delay_ms = 1000  # 1 second

How it works:

With per_host_concurrency = 5 and per_host_delay_ms = 50:

• At most 5 concurrent requests to same host
• New requests to the host start at least 50ms apart
• Other hosts can be fetched simultaneously

max_backoff_ms

Type: number (positive integer) Default: 300000 (5 minutes)

How long the crawl will wait out a host that is rate limiting it. The value caps both a single backoff wait and the total waiting spent on one URL or host.

When a host answers 429 Too Many Requests (or Shopify’s 430, or a 503 that carries a Retry-After header), squirrelscan does not simply fail the page:

• Every request to that host pauses for the backoff window, not just the one that was refused. Requests to other hosts are unaffected.
• The host drops to one in-flight request, and its per_host_delay_ms is multiplied while it recovers.
• Retry-After is honoured as the minimum wait, in both its seconds and HTTP-date forms, clamped to max_backoff_ms.
• Without Retry-After the first wait is 5 seconds, doubling with jitter on each further refusal, for up to six attempts.
• After a sustained run of successful responses the host earns its configured concurrency and delay back in steps.

Once a host has been throttled for longer than max_backoff_ms without recovering, its remaining URLs are recorded as rate limited rather than fetched, and the crawl finishes with the other hosts. Rate-limited URLs are never reported as broken links or 4xx pages: their status is unknown, so they are listed in an informational finding and excluded from scoring.

The report also says how much of the site the crawl never reached: the status is partial, and the summary names the pages the host refused and the URLs that were discovered but never fetched. When a site’s known pages are counted, those unfetched URLs are included, so a throttled run never reads as full coverage.

In the JSON report this is the rateLimited object: pages (refused fetches), hosts, and unfetched (URLs abandoned or still queued when the crawl stopped). unfetched is absent unless the run was throttled.

Examples:

Give a slow host longer to recover:

[crawler]
max_backoff_ms = 600000  # 10 minutes

Fail fast in CI, where a long wait is worse than a short report:

[crawler]
max_backoff_ms = 30000  # 30 seconds

See Crawling rate-limited hosts for the full recipe, including Shopify storefronts.

concurrency

Type: number Default: 5 Range: 1 to 20 recommended

Maximum number of concurrent requests globally.

Examples:

Sequential (single request at a time):

[crawler]
concurrency = 1

Moderate parallelism:

[crawler]
concurrency = 10

High parallelism (use cautiously):

[crawler]
concurrency = 20

Impact:

• Higher = faster crawls
• Higher = more server load
• Bounded by per_host_concurrency

per_host_concurrency

Type: number Default: 5 Range: 1 to 8 recommended

Maximum number of concurrent requests per host.

Prevents overwhelming a single server even with high global concurrency.

Examples:

One request per host at a time:

[crawler]
per_host_concurrency = 1

Allow more parallel requests:

[crawler]
per_host_concurrency = 4

How it interacts with concurrency:

[crawler]
concurrency = 10
per_host_concurrency = 2

• Up to 10 total concurrent requests
• At most 2 concurrent requests to any single host
• Can fetch from up to 5 different hosts simultaneously

Command-line overrides:

Both squirrel audit and squirrel crawl accept --concurrency <n> and --per-host <n> (positive integers) to set these per run without a config file. They take precedence over [crawler] concurrency / per_host_concurrency.

squirrel audit https://example.com --concurrency 10 --per-host 4
squirrel crawl https://example.com --concurrency 10 --per-host 4

Localhost fast path:

When the target is a loopback host (localhost, *.localhost, 127.0.0.1, ::1), which is your own dev server, the crawler skips politeness throttling and runs at raised concurrency with no per-host delay.

squirrel crawl never renders, so the fast path applies to every loopback crawl. For squirrel audit it applies to plain-HTTP crawls only: anonymous runs, --offline, or --http / --render-mode off. When cloud browser rendering is active (the default for signed-in users at the surface and full levels), each page is a credit-priced render governed by your plan’s render concurrency, so the loopback boost does not apply there. On either command, it’s also suppressed when you set concurrency, per_host_concurrency, or per_host_delay_ms explicitly (via flags or config). Use --debug to see whether the fast path engaged.

Compressed responses

Pages are requested with Accept-Encoding: gzip, deflate, br, zstd, unless your headers set another value, and squirrel decodes gzip, br, deflate and zstd bodies itself, as it reads them. A read stops at its size limit whether the body arrived compressed or not: 10 MB of decoded HTML for a page, less for robots.txt, llms.txt and the other root files. A response that would inflate past the limit, a compression bomb included, is cut at the limit rather than decoded in full, so it costs about what an uncompressed response over the limit costs. Bytes a server appends after a complete compressed body are ignored, as a browser ignores them, and a gzip body is read to the end of its first member.

Responses that cannot be decoded

If a server declares a content-encoding (br, gzip) that its response body does not honour, or the compressed body is cut short, the body cannot be read. The crawl records that page as failed with a message naming the URL and the declared encoding, for example Could not decode response body from https://example.com/a (content-encoding: gzip): gzip body does not decode (no gzip header). The CLI lists those messages after the crawl, the first five and a count of the rest, so you can see which pages were lost without opening the crawl database.

The page is missing from the analysis, so a rule that reports an absence cannot tell it from a page that really lacks the thing. When any page is lost this way the report status is partial and its reason counts them, for example 2 pages could not be decoded (bad content-encoding). The pages that were read are scored normally. The fix is on the server: serve the body with the encoding it declares, or stop declaring one.

URL filtering

include

Type: string[] Default: [] (empty = include all URLs from seed domain)

URL patterns to include. If set, only matching URLs are crawled, and only within the crawl’s host scope unless a pattern is absolute (see Scope below).

Pattern Syntax:

* is the only special character, and it stands for any run of characters, / included. There is no ? or [abc]: those match themselves.

A pattern is matched against the URL’s path unless it starts with http:// or https://, in which case it is matched against the whole URL. So a pattern that needs to see a query string or a host has to be absolute.

A pattern ending in /* is a plain prefix test, which is both the cheapest form and the one to reach for. ** is not a separate construct: it is two * in a row and matches exactly what one * does, so older examples written as /blog/** still work.

Examples:

Only crawl blog:

[crawler]
include = ["/blog/*"]

Multiple sections:

[crawler]
include = ["/blog/*", "/docs/*", "/products/*"]

Specific file types:

[crawler]
include = ["*.html", "*.htm"]

Scope: A path-only pattern narrows the crawl within its host scope: a URL must match the pattern and also be on the seed’s host (or on a host allowed by domains in [project]). So include = ["/blog/*"] means the blog section of your site, and a third-party URL whose path starts /blog/ is not crawled. Setting domains alongside include is honoured the same way.

An absolute pattern (one that starts with http:// or https://) names its host, so a URL that matches it is crawled even when that host is outside the scope. That is the way to reach another host on purpose:

[crawler]
include = ["/blog/*", "https://docs.example.org/*"]

The exception is an explicit domains list in [project], which is a hard allowlist. If domains is set and does not list the host an absolute pattern names, that URL is not crawled, and the crawl warns once for each include and host (include "https://docs.example.org/*" matches docs.example.org, which allowedDomains does not list). Add the host to domains, or remove domains, to let the include reach it.

exclude

Type: string[] Default: [] (empty = exclude nothing)

URL patterns to exclude from crawling.

Takes precedence over include - if a URL matches both, it’s excluded.

Examples:

Exclude admin areas:

[crawler]
exclude = ["/admin/*", "/wp-admin/*"]

Exclude file types:

[crawler]
exclude = ["*.pdf", "*.zip", "*.tar.gz"]

Exclude API endpoints:

[crawler]
exclude = ["/api/*", "/v1/*"]

Exclude query parameters. These have to be absolute, because a relative pattern is matched against the path, which never contains a query string:

[crawler]
exclude = ["https://example.com/*?preview=*", "https://example.com/*?draft=*"]

Common exclusions:

[crawler]
exclude = [
  "/admin/*",
  "/wp-admin/*",
  "/wp-content/*",
  "/api/*",
  "*.pdf",
  "*.zip",
  "*.jpg",
  "*.png"
]

Pattern matching examples

* does not stop at a /, so a pattern cannot select one level of a path and not the level below it. Reach for exclude when you need to carve a subtree back out.

Query parameters

allow_query_params

Type: string[] Default: [] (empty list keeps every query parameter except known tracking parameters)

Query parameters to preserve during URL deduplication.

Why this matters:

The crawler deduplicates URLs before the crawl. The empty list keeps non-tracking parameters and drops known tracking parameters:

• /page?id=1&utm_source=google → /page?id=1 (id is kept, utm_source is a tracking prefix)
• https://example.com/products?category=shoes stays https://example.com/products?category=shoes
• https://example.com/products?category=shoes&utm_source=google becomes https://example.com/products?category=shoes
• https://example.com/products?fbclid=abc becomes https://example.com/products with no query

Examples:

Keep page in any case:

[crawler]
allow_query_params = ["page"]

This keeps page in any case. It drops other non-tracking names such as category. The crawler still drops tracking names.

• ?page=2&category=shoes becomes ?page=2
• ?Page=2 is kept with its original spelling

Keep these exact names:

[crawler]
allow_query_params = ["category", "sort", "filter", "q"]

This keeps category, sort, filter, and q. It drops every other non-tracking name, including page.

* is a literal name:

[crawler]
allow_query_params = ["*"]

A non-empty list is an exact-name allow list. * matches only a parameter named *. * does not mean all parameters. The empty default keeps non-tracking parameters.

Use case:

E-commerce site with filters:

[crawler]
allow_query_params = ["category", "price", "brand", "page"]

This preserves:

• /products?category=shoes ✓
• /products?category=shoes&page=2 ✓

This drops:

• /products?color=red ✗ (color is outside the list)
• /products?utm_source=google ✗ (becomes /products)
• /products?gclid=abc123 ✗ (becomes /products)
• /products?page=2&utm_source=google becomes /products?page=2, even if utm_source is added to the allow list

A name outside that list, such as color, is dropped. A tracking name is dropped even when the allow list includes it.

drop_query_prefixes

Type: string[] Default: ["utm_", "gclid", "fbclid"]

URL normalization ignores this list and always strips the built-in tracking set, including when the list is empty.

The built-in tracking set is larger than the schema default. It includes:

• utm_ - names such as utm_source and utm_medium
• gclid - Google Ads
• fbclid - Facebook Ads

Example:

An empty list still strips the built-in tracking set:

[crawler]
drop_query_prefixes = []

Crawl strategy

breadth_first

Type: boolean Default: true

Use breadth-first crawling for better site coverage.

Breadth-first (default):

• Crawls level-by-level
• Discovers homepage, then all links from homepage, then all links from those pages
• Better site coverage
• Avoids getting stuck in deep paths

Depth-first (false):

• Crawls as deep as possible before backtracking
• Can get stuck in deep sections
• Less even coverage

Example:

Disable breadth-first:

[crawler]
breadth_first = false

Recommendation: Keep true (default) for most sites.

max_prefix_budget

Type: number Default: 0.25 (25%) Range: 0.1 to 1.0

Maximum percentage of crawl budget for any single path prefix.

Prevents the crawler from spending all pages on one section (e.g., /blog/ with 1000+ posts).

How it works:

With max_pages = 500 and max_prefix_budget = 0.25:

• At most 125 pages (25%) from any single path prefix
• Ensures diverse coverage across site sections

Examples:

More strict (better coverage):

[crawler]
max_prefix_budget = 0.15  # Max 15% per prefix

More lenient (deeper coverage):

[crawler]
max_prefix_budget = 0.5   # Max 50% per prefix

Disable budget (not recommended):

[crawler]
max_prefix_budget = 1.0   # No limit

Use case:

Site with large blog:

/blog/post-1
/blog/post-2
...
/blog/post-5000
/about
/contact

With max_prefix_budget = 0.25 and max_pages = 500:

• At most 125 blog posts crawled
• Remaining budget for other sections

Request configuration

user_agent

Type: string Default: "" (empty = random browser user agent, pinned per project)

Custom user agent string.

Default behavior (empty string):

A random modern browser user agent is drawn on the first crawl and pinned to the project, so every later run reuses the same one:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36

Pinning keeps the served markup stable between runs (sites often vary HTML by user agent, e.g. mobile themes), which keeps incremental crawls and render caches effective. Pass --fresh-ua to squirrel audit or squirrel crawl to draw a new one; the new user agent replaces the pin for later runs. Setting an explicit user_agent always overrides the pin.

Examples:

Custom user agent:

[crawler]
user_agent = "SquirrelScan Bot (https://squirrelscan.com)"

Mobile user agent:

[crawler]
user_agent = "Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15"

Recommendation: Leave empty (default) for best results with bot protection.

headers

Type: table (map of header name → value) Default: {} (no custom headers)

Custom HTTP request headers attached to every crawl request to the audited origin: pages, assets, robots.txt, sitemaps, llms.txt, markdown probes and the soft-404 and cloaking re-fetches. The audited origin is the scheme, host and port of the start URL after its redirects. A URL on another scheme (http: when you audit https:), another port or another host is still crawled and reported, but is requested without them, including a host that the domains list in [project] or an include pattern admits. A subdomain such as cdn.example.com does not get them unless you name it in header_hosts, because a subdomain is not always run by the site’s owner. The status checks reliability/broken-resources sends to scripts, stylesheets and fonts on other hosts never carry them, and neither does the status check of a sitemap URL on another host. Use this to authorize the crawler with schemes that require signed headers, such as Web Bot Auth (used by Shopify and Cloudflare).

[crawler]
headers = { "Signature-Agent" = "\"https://shopify.com\"", "Signature-Input" = "...", "Signature" = "..." }

CLI override:

squirrel audit https://example.com \
  -H 'Signature-Agent: "https://shopify.com"' \
  -H 'Signature-Input: sig1=("@authority");keyid="..."' \
  -H 'Signature: sig1=:...:'

The repeatable --header / -H flag takes a Name: Value string (split on the first colon, so values may contain colons). CLI headers merge over the TOML [crawler] headers map; a flag with the same name wins. Quoting is preserved verbatim, so Signature-Agent: "https://shopify.com" keeps its quotes end-to-end.

When a page redirects to another origin, the headers are dropped on that hop. An https: start URL whose redirect lands on plain http: keeps its https: scope: the crawl follows the redirect, and the headers stay withheld. The soft-404 confirmation and the cloaking probe follow redirects the same way, and refuse a hop to a link-local or cloud metadata address.

header_hosts

Type: array of hosts Default: [] (the audited origin only)

Hosts, besides the audited origin, that also get the custom headers on page and asset requests: a signed CDN, or Cloudflare Access on assets.example.com. Only the hosts you name get them, so a www host of an apex site is named here too. An entry is a host name, optionally with an http:// or https:// prefix and a port; a path or trailing slash is ignored and an internationalised name is matched in its punycode form. A wildcard, userinfo, empty label or trailing dot is refused. A name matches the host on any port, over https: only; write it with an http:// prefix (header_hosts = ["http://assets.example.com"]) to allow plain http too.

[crawler]
header_hosts = ["assets.example.com"]

CLI override: the repeatable --header-host flag adds to the TOML list.

squirrel audit https://example.com -H 'X-Api-Key: ...' --header-host assets.example.com

If an asset on another host answers 401 or 403 to a request sent without the headers, it is reported as not checked rather than as a broken resource, and the message names the host to add here.

Cloud audits: custom headers for dashboard cloud audits are available on every plan, Free included. Set them per-website under Settings → Crawl → Custom request headers. Plain HTTP requests carry them. For pages rendered in the cloud browser, by cloud audits and by signed-in CLI audits, see rendering and custom headers. A signed-in CLI audit’s cloud renders never carry -H headers, so use --http when the site is only reachable with them. A cloud audit’s renders carry the website’s custom headers.

follow_redirects

Type: boolean Default: true

Follow HTTP 3xx redirects.

When true (default):

• Follows redirects automatically
• Crawls final destination URL
• Redirect chains tracked for analysis

When false:

• Stops at redirect
• Does not fetch redirect destination
• Useful for debugging redirect issues

Example:

Disable redirect following:

[crawler]
follow_redirects = false

Recommendation: Keep true (default) for normal audits.

Robots.txt

squirrelscan always fetches and parses /robots.txt, regardless of respect_robots below: it’s used for sitemap discovery (Sitemap: directives) and the crawl/robots-txt audit rule.

respect_robots

Type: boolean Default: false

Enforce robots.txt Disallow: rules and honor its Crawl-delay: directive.

When false (default):

• robots.txt is still fetched and parsed (for sitemaps + the audit rule)
• Disallow: rules are not enforced — all in-scope URLs are crawled
• Crawl-delay: is ignored — per_host_delay_ms applies instead
• squirrelscan audits are something you run against your own site, not an uninvited crawl, so directives aimed at bots don’t apply unless you opt in

When true:

• Respects Disallow: rules for the SquirrelScan and * user agents
• Honors Crawl-delay:, capped at 2 seconds even if the directive asks for longer

Example:

Enforce robots.txt (auditing a site you don’t own):

[crawler]
respect_robots = true

Recommendation: Turn this on when auditing a third-party site, or when you want a fully compliant crawl. Leave it off (default) for your own sites.

Cloud audits: the per-website toggle under Settings → Crawl → Respect robots.txt applies the same enforcement to dashboard cloud audits. It defaults to off, matching the CLI default above.

Discovery probes

Before the first page, squirrelscan asks the site for a set of files that agents look for. These are the discovery probes, and the agent experience rules read them: whether you publish an llms.txt, serve Markdown, expose an MCP server card or an OpenAPI spec, and whether AI crawlers get the same homepage a browser does.

That is 28 requests on most sites. robots.txt, sitemap discovery and the page you asked to audit are not probes: the crawl needs them, so they always run.

disable_discovery_probes

Type: boolean Default: false

Set to true to send no discovery probes at all.

[crawler]
disable_discovery_probes = true

CLI override:

squirrel audit https://example.com --disable-discovery-probes

The flag wins over the config for one run, in both directions: --disable-discovery-probes turns the probes off, and --disable-discovery-probes=false sends them even when the config says true.

Some servers ban a client for asking for these paths. A common fail2ban “sensitive files” jail reads a request for /swagger.json or /openapi.json as an attack and bans the auditing IP, so every request after it gets connection refused and the audit fails right after it starts. Turn the probes off for that site and the crawl goes through.

Or let squirrelscan through

To keep the checks, make an exception for the audit instead. Firewall and WAF exceptions covers the options for cloud audits. A CLI audit comes from your own machine, so for one of those:

• fail2ban or another log-based jail: add your own IP to the jail’s ignoreip.

• A WAF or bot protection: send a secret header with the audit and skip the rule when it matches:

  squirrel audit https://example.com -H "X-Squirrelscan-Auth: <long random value>"

  Copy

  Or set it under headers. The probes, robots.txt, sitemaps and every page carry it. The checks that resolve your start URL before the crawl begins do not.

Probes follow your rules

A probe is only sent when a rule that reads it is going to run. Skip the rules and their requests go with them, with no other setting:

# No discovery probes: every rule that reads one is skipped
squirrel audit https://example.com --rule-exclude ax

# Everything except the OpenAPI, Swagger and OAuth paths
squirrel audit https://example.com --rule-exclude ax/api-discovery

[rules] disable in squirrel.toml works the same way.

What the report says

When a rule’s probe was not sent, or a probe for a file went out and got no answer (a refused connection or a timeout), the rule reports not checked, with the value not-checked: no answer is not evidence the file is missing. Those results never count as a failure and are not scored. Two rules keep their own reading of an unanswered request: ax/agent-blocking calls an AI-crawler fetch with no answer inconclusive, and ax/rsl-license still reports a license document your robots.txt names, but that could not be fetched, as a broken reference.

Pacing

Discovery probes wait their turn behind the same per-host limits as pages: at most per_host_concurrency at once, with starts at least per_host_delay_ms apart. When cloud rendering limits how many pages load at once, the probes still use these two settings, since they are plain requests. A robots.txt Crawl-delay applies to pages only. For a slow or touchy site, per_host_concurrency = 1 sends them one at a time, which is usually gentler than a longer delay alone.

The probes share one time budget with the start page and robots.txt reads: 45 seconds, or three times timeout_ms when that is shorter. They go out in the order of the table above, so with a long per_host_delay_ms the last ones in it may not fit. Those are skipped and reported as not checked, and the crawl carries on.

Crawling politely & re-scanning efficiently

squirrelscan is designed to be a good guest on the sites it audits. Two mechanisms keep load low: politeness controls (how fast it fetches) and incremental re-scanning (avoiding re-downloading pages that haven’t changed).

incremental

Type: boolean Default: true

Re-scan only pages that changed since the last crawl. When enabled, the crawler records each page’s ETag and Last-Modified response headers (plus a content hash) and, on the next audit of the same site, sends conditional requests (If-None-Match / If-Modified-Since). Unchanged pages return a lightweight 304 Not Modified: no body is transferred and the cached content is reused.

This is the answer to “incremental vs full scans so I don’t overwhelm the host”: incremental crawling is on by default, so repeat audits re-download only what actually changed.

• First audit of a site: there’s nothing cached yet, so every page is fetched in full; incremental has no effect on a cold run.
• Repeat audits: unchanged pages come back as 304s, cutting bandwidth and load on the target server.

[crawler]
incremental = true

CLI overrides:

# Force incremental on for this run — overrides a project config with incremental = false
squirrel audit https://example.com --incremental

# Disable conditional requests — fetch every page in full
squirrel audit https://example.com --no-incremental

# Full re-scan, ignoring all cached content (alias for a cold crawl)
squirrel audit https://example.com --refresh

The resolution order is --refresh (always full) > --incremental / --no-incremental > [crawler] incremental (default true). --refresh also ignores the cross-audit freshness cache (see use_cache_control), so use it when you want a guaranteed clean re-fetch.

Politeness checklist

For a gentle crawl of a third-party site, combine incremental re-scanning with the rate-limit and robots controls documented above:

• respect_robots: obey robots.txt and its Crawl-delay (set true; default false).
• per_host_delay_ms: space out requests to a single host.
• per_host_concurrency: cap simultaneous requests per host.
• delay_ms / concurrency: global pacing.
• incremental: skip re-downloading unchanged pages on repeat audits.
• disable_discovery_probes: turn off the pre-crawl requests for agent files when the site’s firewall objects to them (see Discovery probes).

A Crawl-delay directive in the target’s robots.txt overrides per_host_delay_ms only when respect_robots = true, and is capped at 2 seconds even then.

What squirrelscan does not access

squirrelscan crawls public HTTP only: the same pages a browser or search engine bot would see. It does not read server logs, databases, or any authenticated/internal source, and it requires no access beyond what the public site serves. Scanning protected or internal sources is not currently supported; if you need it, please open an issue describing your use case so it can be scoped separately.

Complete examples

Fast local development

[crawler]
max_pages = 50
delay_ms = 0
per_host_delay_ms = 0
concurrency = 10
respect_robots = false

Polite production crawl

[crawler]
max_pages = 500
delay_ms = 200
per_host_delay_ms = 500
concurrency = 5
per_host_concurrency = 2
respect_robots = true

High-volume crawl

[crawler]
max_pages = 2000
delay_ms = 100
per_host_delay_ms = 200
concurrency = 10
per_host_concurrency = 3
breadth_first = true
max_prefix_budget = 0.2

Focused blog crawl

[crawler]
max_pages = 200
include = ["/blog/*"]
exclude = ["*.pdf", "/blog/drafts/*"]
allow_query_params = ["page"]

E-commerce site

[crawler]
max_pages = 1000
include = ["/products/*", "/categories/*"]
exclude = ["/cart/*", "/checkout/*", "/account/*"]
allow_query_params = ["category", "sort", "page", "filter"]
drop_query_prefixes = ["utm_", "gclid", "fbclid", "ref"]

Related

• Project Settings - Domains configuration
• Rules Configuration - Which rules to run
• Examples - More configuration examples