Firewalls, bot protection and custom headers
Get cloud, CLI and MCP audits past a WAF, bot protection, an IP allowlist or an access gateway, and fix sites that reset browser-like clients
A WAF, bot protection, an IP allowlist or an access gateway can turn an audit into a wall of 403s, challenge pages, redirects to a sign-in page or dropped connections. The fix is the same few exceptions whichever way you audit. What changes is where you set them.
Where you set it
| How you audit | Where requests come from | Where to set the exception |
|---|---|---|
| Cloud audit from the dashboard, the API or a schedule | squirrelscan’s cloud | The website’s Settings → Crawl → Custom request headers in the dashboard |
| Hosted MCP audit | squirrelscan’s cloud | The same website settings. A hosted MCP audit runs in the cloud and uses the website’s custom request headers, as dashboard and scheduled audits do |
squirrel audit in a terminal, in CI, or driven by a coding agent over the CLI | Your machine or CI runner | squirrel.toml under [crawler], or the -H and --header-host flags |
The cloud side of this is set out in full on Firewall and WAF exceptions, which stays the cloud reference. This guide puts the cloud setting and the CLI setting next to each other.
Which exception fits
| In front of the site | Exception |
|---|---|
| A WAF rule, bot protection or rate limit | A secret request header and a rule that skips the protection for it |
| A platform that verifies signed crawlers | Web Bot Auth |
| A firewall that only matches IP addresses | IP allowlisting |
| Cloudflare Access | A service token |
| Vercel Authentication or Password Protection | The protection bypass header |
| HTTP basic auth | An Authorization header |
| Okta, Entra ID, Google IAP and other sign-in gateways | No header route |
| The connection is reset before any response | A site that resets browser-like clients |
A secret request header and a WAF skip rule
The most precise exception: a header whose value only you know, and a firewall rule that skips the protection when it sees that exact value.
| Setting | |
|---|---|
| Cloud | Settings → Crawl → Custom request headers: add X-Squirrelscan-Auth with a long random value and select Save headers |
| CLI | [crawler] in squirrel.toml, or -H for a CI secret (below) |
| MCP | Hosted MCP audits use the website’s cloud setting. A CLI-driven agent uses squirrel.toml or -H |
[crawler]
headers = { "X-Squirrelscan-Auth" = "your-long-random-value" }squirrel audit https://example.com -H "X-Squirrelscan-Auth: $SQUIRREL_AUTH"The firewall rule is the same for every kind of audit. The Cloudflare, nginx and other-WAF rules are on Firewall and WAF exceptions. Prefer -H with a CI secret over committing the value to a shared squirrel.toml.
Web Bot Auth
If your platform verifies crawlers by signature (Shopify, Cloudflare verified bots), the three signed headers go in the same place as a secret header. See Web Bot Auth for what each header contains.
| Setting | |
|---|---|
| Cloud | Add the signed headers under Settings → Crawl → Custom request headers |
| CLI | headers = { "Signature-Agent" = "...", "Signature-Input" = "...", "Signature" = "..." }, or three -H flags |
| MCP | Hosted: the website’s cloud setting. CLI-driven: squirrel.toml or -H |
IP allowlisting
| What to allow | |
|---|---|
| Cloud | There is no dedicated squirrelscan address. Cloud audits leave from Cloudflare’s shared egress addresses, so allow Cloudflare’s published ranges and pair the rule with a secret header, as on Firewall and WAF exceptions |
| CLI | Plain fetches come from your machine or CI runner, so allow that public IP. A signed-in audit that renders pages in the cloud browser sends those renders from squirrelscan’s network, so allow those addresses too, as for cloud audits, or run with --http to keep every request on your machine |
| MCP | Hosted: the cloud ranges. CLI-driven: as the CLI row |
Datacenter and CI addresses are challenged more often than a laptop, so a CI audit may need its runner’s address allowed even when a local run works.
Cloudflare Access
Access redirects a request with no valid session to <team-name>.cloudflareaccess.com. Create a service token and a Service Auth policy in Cloudflare (the steps are on Firewall and WAF exceptions), then send the token as two headers:
| Setting | |
|---|---|
| Cloud | Add CF-Access-Client-Id and CF-Access-Client-Secret under Settings → Crawl → Custom request headers |
| CLI | squirrel.toml (below), or two -H flags |
| MCP | Hosted: the cloud setting. CLI-driven: squirrel.toml or -H |
[crawler]
headers = { "CF-Access-Client-Id" = "<client-id>", "CF-Access-Client-Secret" = "<client-secret>" }Audit the exact https:// origin Access protects. A redirect to another origin (including http:// to https://, or the apex to www) removes the headers before Access sees them. If Access also protects an assets host, name it in header_hosts.
Vercel protection bypass
Create a secret under Protection Bypass for Automation in the Vercel project (see Firewall and WAF exceptions), then send it as x-vercel-protection-bypass. Leave out x-vercel-set-bypass-cookie.
| Setting | |
|---|---|
| Cloud | Add x-vercel-protection-bypass under Settings → Crawl → Custom request headers |
| CLI | headers = { "x-vercel-protection-bypass" = "<secret>" } under [crawler], or -H "x-vercel-protection-bypass: <secret>" |
| MCP | Hosted: the cloud setting. CLI-driven: squirrel.toml or -H |
HTTP basic auth
The value is Basic followed by the Base64 of username:password. Encode it with printf '%s' 'username:password' | base64, so no newline is included, and use an account that exists only for the audit.
| Setting | |
|---|---|
| Cloud | Add Authorization with the value Basic <base64-value> under Settings → Crawl → Custom request headers |
| CLI | headers = { "Authorization" = "Basic <base64-value>" } under [crawler], or -H "Authorization: Basic <base64-value>" |
| MCP | Hosted: the cloud setting. CLI-driven: squirrel.toml or -H |
Other zero-trust gateways
A gateway that only offers an interactive sign-in (Okta, Auth0, Microsoft Entra ID, Amazon Cognito, OneLogin, Clerk) or short-lived tokens (Google Identity-Aware Proxy) has no header you can save, in the cloud or in squirrel.toml.
| What to do | |
|---|---|
| Cloud | Add an exception in your own application or proxy that lets a secret header through, then send that header as in the secret header section. Otherwise a cloud audit cannot pass the gateway |
| CLI | Run squirrel audit from a network the gateway already admits (an office network or VPN), or audit a local build of the site: squirrel audit http://localhost:3000. See local audits |
| MCP | Hosted: as for cloud. CLI-driven: as for the CLI |
A signed-in CLI audit sends page renders to the cloud browser, which comes from squirrelscan’s network. Add --http so every page is fetched from yours.
A site that resets or refuses browser-like clients
Some servers drop any connection whose request claims to be a browser but whose TLS handshake is not a browser’s. There is no HTTP status to read: the audit fails before it crawls anything.
The symptom. A CLI audit stops with an error such as Cannot reach https://example.com: ECONNRESET, a socket closed message, or an HTTP/2 protocol error.
What the CLI does. With no user_agent configured, the CLI treats a reset or protocol error as a possible bot filter only after its browser-agent attempts, with their transient retries, have all failed. It then makes one attempt as SquirrelScan/2.0 (+https://squirrelscan.com). If the site answers that, the whole crawl runs as that agent, the CLI prints one line saying so, and the report says so too (userAgentFallback in the JSON report). The honest agent is not pinned: the next run probes with a browser agent again. If the site resets the honest agent as well, the error reads connection reset by the site: it may block browser-like clients.
A configured agent is never replaced. A user_agent in [crawler], or a User-Agent in [crawler] headers or -H, is used as set and the fallback never runs.
The manual fix. Set the agent yourself to one the site accepts:
[crawler]
user_agent = "SquirrelScan/2.0 (+https://squirrelscan.com)"If a firewall rule matches on the agent, allow SquirrelScan there, or use a secret header instead. The fallback is a CLI behaviour. Cloud audits have no custom user-agent setting: see what squirrelscan cloud looks like in your logs. The full reference is user_agent.
Header scoping
Custom headers are credentials, so both the cloud and the CLI send them to as few places as they can.
CLI (squirrel audit)
- Headers go to the audited origin only: the scheme, host and port of the start URL after its redirects.
http://when you audithttps://, another port, or another host is requested without them. - A subdomain such as
cdn.example.comgets them only when you name it inheader_hosts(or pass--header-host). A name matches overhttps:. Prefixhttp://to allow plain http. - Status checks of scripts, stylesheets and fonts on other hosts never carry them.
- On a redirect to another origin, every header is dropped except the generic client headers
User-Agent,Accept,Accept-Language,Accept-EncodingandUpgrade-Insecure-Requests. Never put a secret in one of those. - A signed-in CLI audit renders pages in the cloud browser, and those renders never carry
-Hor[crawler] headers: the CLI sends the render service the URLs, a timeout and a run id. Use--httpwhen the site is only reachable with a header.
[crawler]
headers = { "X-Api-Key" = "..." }
header_hosts = ["assets.example.com"]Cloud audits (cloud behaviour, per the existing Firewall and WAF exceptions)
- A website’s headers are used only when the audit crawls that website’s own host, with or without
www.. A subdomain is its own website with its own headers. - Plain HTTP requests carry them. The cloud browser carries them to the rendered page’s own origin only. See rendering and custom headers for the request-by-request table.
- Status checks of resources on other hosts, and external link checks, never carry them.
Check that it worked
Request the audited address with the same headers using curl -sS -o /dev/null -D - -H 'Name: value' <url>. It should answer 200 with no location on a sign-in host. Then run the audit and check that the start page has your page’s title, not a sign-in or challenge page. For a failed cloud audit, the verification checklist and failed audits say what each outcome means.