squirrelscan
GitHub

Firewalls, bot protection and custom headers

Get cloud, CLI and MCP audits past a WAF, bot protection, an IP allowlist or an access gateway, and fix sites that reset browser-like clients

A WAF, bot protection, an IP allowlist or an access gateway can turn an audit into a wall of 403s, challenge pages, redirects to a sign-in page or dropped connections. The fix is the same few exceptions whichever way you audit. What changes is where you set them.

Where you set it

How you auditWhere requests come fromWhere to set the exception
Cloud audit from the dashboard, the API or a schedulesquirrelscan’s cloudThe website’s Settings → Crawl → Custom request headers in the dashboard
Hosted MCP auditsquirrelscan’s cloudThe same website settings. A hosted MCP audit runs in the cloud and uses the website’s custom request headers, as dashboard and scheduled audits do
squirrel audit in a terminal, in CI, or driven by a coding agent over the CLIYour machine or CI runnersquirrel.toml under [crawler], or the -H and --header-host flags

The cloud side of this is set out in full on Firewall and WAF exceptions, which stays the cloud reference. This guide puts the cloud setting and the CLI setting next to each other.

Which exception fits

In front of the siteException
A WAF rule, bot protection or rate limitA secret request header and a rule that skips the protection for it
A platform that verifies signed crawlersWeb Bot Auth
A firewall that only matches IP addressesIP allowlisting
Cloudflare AccessA service token
Vercel Authentication or Password ProtectionThe protection bypass header
HTTP basic authAn Authorization header
Okta, Entra ID, Google IAP and other sign-in gatewaysNo header route
The connection is reset before any responseA site that resets browser-like clients

A secret request header and a WAF skip rule

The most precise exception: a header whose value only you know, and a firewall rule that skips the protection when it sees that exact value.

Setting
CloudSettings → Crawl → Custom request headers: add X-Squirrelscan-Auth with a long random value and select Save headers
CLI[crawler] in squirrel.toml, or -H for a CI secret (below)
MCPHosted MCP audits use the website’s cloud setting. A CLI-driven agent uses squirrel.toml or -H
[crawler]
headers = { "X-Squirrelscan-Auth" = "your-long-random-value" }
squirrel audit https://example.com -H "X-Squirrelscan-Auth: $SQUIRREL_AUTH"

The firewall rule is the same for every kind of audit. The Cloudflare, nginx and other-WAF rules are on Firewall and WAF exceptions. Prefer -H with a CI secret over committing the value to a shared squirrel.toml.

Web Bot Auth

If your platform verifies crawlers by signature (Shopify, Cloudflare verified bots), the three signed headers go in the same place as a secret header. See Web Bot Auth for what each header contains.

Setting
CloudAdd the signed headers under Settings → Crawl → Custom request headers
CLIheaders = { "Signature-Agent" = "...", "Signature-Input" = "...", "Signature" = "..." }, or three -H flags
MCPHosted: the website’s cloud setting. CLI-driven: squirrel.toml or -H

IP allowlisting

What to allow
CloudThere is no dedicated squirrelscan address. Cloud audits leave from Cloudflare’s shared egress addresses, so allow Cloudflare’s published ranges and pair the rule with a secret header, as on Firewall and WAF exceptions
CLIPlain fetches come from your machine or CI runner, so allow that public IP. A signed-in audit that renders pages in the cloud browser sends those renders from squirrelscan’s network, so allow those addresses too, as for cloud audits, or run with --http to keep every request on your machine
MCPHosted: the cloud ranges. CLI-driven: as the CLI row

Datacenter and CI addresses are challenged more often than a laptop, so a CI audit may need its runner’s address allowed even when a local run works.

Cloudflare Access

Access redirects a request with no valid session to <team-name>.cloudflareaccess.com. Create a service token and a Service Auth policy in Cloudflare (the steps are on Firewall and WAF exceptions), then send the token as two headers:

Setting
CloudAdd CF-Access-Client-Id and CF-Access-Client-Secret under Settings → Crawl → Custom request headers
CLIsquirrel.toml (below), or two -H flags
MCPHosted: the cloud setting. CLI-driven: squirrel.toml or -H
[crawler]
headers = { "CF-Access-Client-Id" = "<client-id>", "CF-Access-Client-Secret" = "<client-secret>" }

Audit the exact https:// origin Access protects. A redirect to another origin (including http:// to https://, or the apex to www) removes the headers before Access sees them. If Access also protects an assets host, name it in header_hosts.

Vercel protection bypass

Create a secret under Protection Bypass for Automation in the Vercel project (see Firewall and WAF exceptions), then send it as x-vercel-protection-bypass. Leave out x-vercel-set-bypass-cookie.

Setting
CloudAdd x-vercel-protection-bypass under Settings → Crawl → Custom request headers
CLIheaders = { "x-vercel-protection-bypass" = "<secret>" } under [crawler], or -H "x-vercel-protection-bypass: <secret>"
MCPHosted: the cloud setting. CLI-driven: squirrel.toml or -H

HTTP basic auth

The value is Basic followed by the Base64 of username:password. Encode it with printf '%s' 'username:password' | base64, so no newline is included, and use an account that exists only for the audit.

Setting
CloudAdd Authorization with the value Basic <base64-value> under Settings → Crawl → Custom request headers
CLIheaders = { "Authorization" = "Basic <base64-value>" } under [crawler], or -H "Authorization: Basic <base64-value>"
MCPHosted: the cloud setting. CLI-driven: squirrel.toml or -H

Other zero-trust gateways

A gateway that only offers an interactive sign-in (Okta, Auth0, Microsoft Entra ID, Amazon Cognito, OneLogin, Clerk) or short-lived tokens (Google Identity-Aware Proxy) has no header you can save, in the cloud or in squirrel.toml.

What to do
CloudAdd an exception in your own application or proxy that lets a secret header through, then send that header as in the secret header section. Otherwise a cloud audit cannot pass the gateway
CLIRun squirrel audit from a network the gateway already admits (an office network or VPN), or audit a local build of the site: squirrel audit http://localhost:3000. See local audits
MCPHosted: as for cloud. CLI-driven: as for the CLI

A signed-in CLI audit sends page renders to the cloud browser, which comes from squirrelscan’s network. Add --http so every page is fetched from yours.

A site that resets or refuses browser-like clients

Some servers drop any connection whose request claims to be a browser but whose TLS handshake is not a browser’s. There is no HTTP status to read: the audit fails before it crawls anything.

The symptom. A CLI audit stops with an error such as Cannot reach https://example.com: ECONNRESET, a socket closed message, or an HTTP/2 protocol error.

What the CLI does. With no user_agent configured, the CLI treats a reset or protocol error as a possible bot filter only after its browser-agent attempts, with their transient retries, have all failed. It then makes one attempt as SquirrelScan/2.0 (+https://squirrelscan.com). If the site answers that, the whole crawl runs as that agent, the CLI prints one line saying so, and the report says so too (userAgentFallback in the JSON report). The honest agent is not pinned: the next run probes with a browser agent again. If the site resets the honest agent as well, the error reads connection reset by the site: it may block browser-like clients.

A configured agent is never replaced. A user_agent in [crawler], or a User-Agent in [crawler] headers or -H, is used as set and the fallback never runs.

The manual fix. Set the agent yourself to one the site accepts:

[crawler]
user_agent = "SquirrelScan/2.0 (+https://squirrelscan.com)"

If a firewall rule matches on the agent, allow SquirrelScan there, or use a secret header instead. The fallback is a CLI behaviour. Cloud audits have no custom user-agent setting: see what squirrelscan cloud looks like in your logs. The full reference is user_agent.

Header scoping

Custom headers are credentials, so both the cloud and the CLI send them to as few places as they can.

CLI (squirrel audit)

  • Headers go to the audited origin only: the scheme, host and port of the start URL after its redirects. http:// when you audit https://, another port, or another host is requested without them.
  • A subdomain such as cdn.example.com gets them only when you name it in header_hosts (or pass --header-host). A name matches over https:. Prefix http:// to allow plain http.
  • Status checks of scripts, stylesheets and fonts on other hosts never carry them.
  • On a redirect to another origin, every header is dropped except the generic client headers User-Agent, Accept, Accept-Language, Accept-Encoding and Upgrade-Insecure-Requests. Never put a secret in one of those.
  • A signed-in CLI audit renders pages in the cloud browser, and those renders never carry -H or [crawler] headers: the CLI sends the render service the URLs, a timeout and a run id. Use --http when the site is only reachable with a header.
[crawler]
headers = { "X-Api-Key" = "..." }
header_hosts = ["assets.example.com"]

Cloud audits (cloud behaviour, per the existing Firewall and WAF exceptions)

  • A website’s headers are used only when the audit crawls that website’s own host, with or without www.. A subdomain is its own website with its own headers.
  • Plain HTTP requests carry them. The cloud browser carries them to the rendered page’s own origin only. See rendering and custom headers for the request-by-request table.
  • Status checks of resources on other hosts, and external link checks, never carry them.

Check that it worked

Request the audited address with the same headers using curl -sS -o /dev/null -D - -H 'Name: value' <url>. It should answer 200 with no location on a sign-in host. Then run the audit and check that the start page has your page’s title, not a sign-in or challenge page. For a failed cloud audit, the verification checklist and failed audits say what each outcome means.

Type to search…

↑↓ navigate↵ openesc close