squirrelscan
GitHub

Firewall and WAF exceptions

Let squirrelscan cloud audits through your firewall, WAF, bot protection, or access gateway

Cloud audits fetch your pages from squirrelscan’s infrastructure, not from your machine. If your site sits behind a WAF, bot protection, an IP allowlist, or an access gateway such as Cloudflare Access, those requests can come back as 403s, challenge pages, rate limits or redirects to a sign-in page, and the audit either fails or audits the block page instead of your site.

This page covers how to recognise squirrelscan cloud traffic, the three ways to let it through a firewall, from most to least reliable, and how to get it past the access gateways that protect staging sites.

Which exception fits

In front of the siteWhat a cloud audit gets without an exceptionException
A WAF rule, bot protection or rate limit403, 429 or a challenge pageA secret request header and a rule that skips the protection for it
A platform that verifies signed crawlers403 or a challengeWeb Bot Auth
A firewall that only matches IP addressesRefused connections or 403IP allowlisting, paired with the header
Cloudflare AccessA redirect to <team-name>.cloudflareaccess.comA service token
Vercel Authentication or Password ProtectionA redirect to a Vercel login, or a password pageThe protection bypass header
HTTP basic auth401 with WWW-Authenticate: BasicAn Authorization header
Single sign-on through Okta, Auth0, Microsoft Entra ID, Google IAP and similarA redirect to the identity providerNo header route. See other zero-trust gateways

Every exception except IP allowlisting is a custom request header. Read how custom request headers are sent before you rely on one: they are not sent on every kind of request.

How custom request headers are sent

Every website in the dashboard can send custom HTTP request headers with its cloud audits. You set them under Settings → Crawl → Custom request headers (see crawl settings). Whether a request carries them depends on the kind of request and, for pages, on the website’s Rendering setting.

The headers belong to the website they are set on, and an audit uses them only when it crawls that website’s own host: the host it is registered under, with or without www.. Audits started from the dashboard, over MCP or by a schedule always crawl that host.

An audit started through the API that names the website beside a URL on another host runs without the headers, and without the authorised pace they bring. That includes a subdomain: staging.example.com named under the website example.com gets neither. To audit a subdomain with headers, add it as its own website and set its headers there.

Rendering and custom headers

This table is the reference for which requests of a cloud audit carry your custom request headers. Rendering is the setting in the audit level card under Settings → Crawl.

RequestRendering: OffRendering: AutoRendering: All
Page requestSent. Every page is a plain HTTP requestSent on the plain HTTP request every page starts with, and on the cloud browser render of a page that needs JavaScript (see which renders carry them)Sent on the cloud browser render (see which renders carry them), and on the plain HTTP request that replaces a render the site refuses, or that starts when the render has no result after 18 s
Subresources the cloud browser loads while it renders a page: scripts, stylesheets, images, and the requests its scripts makeNo browser is usedWhen the page’s render carries them: sent to the rendered page’s own origin only. Never to another origin, such as a CDN, an analytics script or a sign-in pageWhen the page’s render carries them: sent to the rendered page’s own origin only. Never to another origin, such as a CDN, an analytics script or a sign-in page
Asset checks: stylesheets, scripts, images and fonts on the audited host and its subdomainsSentSentSent
Status checks of scripts, stylesheets and fonts on other hosts (reliability/broken-resources)Not sentNot sentNot sent
Seed redirect probe, which checks where the audited address redirects before the crawl startsNot sentNot sentNot sent
robots.txt, sitemaps, llms.txt and the other root filesSentSentSent

External link checks and the GitHub check never send them, whatever the setting.

Every Rendering setting therefore sends the headers on the requests for your pages, so a site that is only reachable with a header, such as one behind Cloudflare Access or Vercel deployment protection, can keep rendering on.

Which renders carry them

A cloud browser render carries the headers when the page is on the audited address’s host or its www twin, with the audited address’s scheme and port. For an audit of https://staging.example.com/, that is every page on https://staging.example.com and https://www.staging.example.com. A page on another subdomain, or on http:// for a site audited over https://, is rendered without them, so a secret never travels over plain http:// for a site you audit over https://. A site you audit over http:// gets them over http://, unencrypted, as its plain requests do, so audit a gated site over https://.

Inside the browser, the headers go only to the origin of the page being rendered: the page itself, and the scripts, stylesheets, images and API calls it loads from that origin. A redirect to another origin drops them, and a redirect back to the page’s origin carries them again, because that is the origin they are for. If the browser cannot attach them to one of those requests, the render is dropped and the page is fetched over plain HTTP instead, never rendered without them.

Three behaviours follow:

  • A refusal is retried without the browser. When the site answers a cloud browser render with 401, 403, 429, 503 or a challenge page, the page is fetched again over plain HTTP, which carries the headers. A WAF exception keyed on a header therefore still lets the page through, unrendered.
  • A gateway’s sign-in page is never kept as your page. When custom request headers are set and a render still lands on the sign-in page of a known access gateway (Cloudflare Access, Vercel, Okta, Auth0, Microsoft Entra ID and the others in access gateways), the page is fetched again over plain HTTP with the headers. If that request lands on the sign-in page too, the gateway did not accept the headers, and the audit fails as a redirect to another site.
  • The CLI’s renders never carry -H headers. A signed-in squirrel audit renders in the cloud browser too, but without the headers passed with -H: add --http when the audit depends on them.

Redirects and scope

On a plain HTTP request that is sent with them, the headers are removed at the first redirect to a different origin (scheme, host and port), and stay removed for the rest of that redirect chain. Only User-Agent, Accept, Accept-Language and Upgrade-Insecure-Requests are kept across it. http:// to https:// is a different origin, and so is example.com to www.example.com. For the cloud browser, see which renders carry them.

Headers that are sent go to every host the audit fetches from on your site: the audited host, and any of its subdomains whose stylesheets, scripts, images or fonts the pages load. Hosts on other domains, such as a CDN or an analytics provider, never get them. A credential in a custom request header therefore reaches your subdomains too, but nothing off your domain. Give the audit a credential of its own, scoped as narrowly as the gateway allows.

Accepted names and limits

The header card accepts any valid HTTP header name, Authorization included. Nothing is reserved. A website can have up to 50 headers, each name up to 255 characters and each value up to 8,192 characters. A value may contain tabs but no other control characters. Values are stored as secrets and never shown in reports or logs. Setting any header also makes the crawl authorised, which raises its pace and stops robots.txt being applied.

A firewall rule that matches a header value only you know is the most precise exception you can make, and it works with any firewall that can match a request header.

  1. Open the website in the dashboard and go to Settings → Crawl.
  2. Scroll to Custom request headers at the bottom of the page.
  3. Add a header, for example X-Squirrelscan-Auth with a long random value, and click Save headers.
  4. Add a rule to your firewall that skips bot protection when that header carries that exact value.

Custom request headers are available on every plan.

Create a custom rule under Security → WAF → Custom rules with the action Skip, and select what to skip: Super Bot Fight Mode rules, rate limiting rules and managed rules, as the block requires.

(http.request.headers["x-squirrelscan-auth"][0] eq "your-long-random-value")

Header names are matched in lowercase in Cloudflare expressions.

Bot Fight Mode, the bot protection on Cloudflare’s Free plan, cannot be skipped by a WAF custom rule or a Page Rule (Cloudflare’s Bot Fight Mode documentation). If Bot Fight Mode is what challenges the crawler, turn it off, or use Super Bot Fight Mode, which a Skip rule can skip. A Skip rule cannot skip Cloudflare Access either: see Cloudflare Access.

map $http_x_squirrelscan_auth $squirrelscan_ok {
    default 0;
    "your-long-random-value" 1;
}

# Inside the server or location block that enforces your rule:
if ($squirrelscan_ok) {
    # e.g. skip the rate-limit zone or the geo/IP deny list
}

Any WAF that can match a request header will work: Akamai, AWS WAF, Fastly, Vercel firewall, and Netlify all support a header equals condition with an allow or bypass action. Match the header name case-insensitively and compare the full value.

Rotate the value from the same settings page at any time. The next audit picks up the new value automatically.

Option 2: Web Bot Auth

If your platform verifies crawlers cryptographically (Cloudflare verified bots, Shopify crawler access keys), add the three signed headers from your provider as custom request headers instead of a shared secret. The mechanics are the same as option 1; the values come from your platform. See Web Bot Auth for what each header contains.

Option 3: IP allowlisting

squirrelscan cloud runs on Cloudflare. Audit requests, including browser renders, leave from Cloudflare’s shared egress addresses, so there is no dedicated squirrelscan IP range to allowlist.

If your firewall can only work with IP addresses, allow Cloudflare’s published ranges:

Cloudflare Access

Cloudflare Access answers a request that has no valid session with a 302 redirect to its login page on your team domain, <team-name>.cloudflareaccess.com. A cloud audit cannot sign in, so it fails as a redirect to another site. A WAF custom rule does not help: Cloudflare Access is not one of the phases or products a Skip rule can skip.

Access lets a machine through with a service token: a Client ID and a Client Secret sent as two request headers, matched by a policy with the Service Auth action. The menu paths below follow Cloudflare’s documentation as of October 2026; if they have moved, Cloudflare’s service tokens and Access policies pages have the current ones.

  1. Create the token. In Cloudflare, go to Zero Trust → Access controls → Service credentials → Service Tokens and select Create Service Token. Name it (for example squirrelscan), choose a Service Token Duration, and select Generate token. Copy the Client ID and the Client Secret. Cloudflare shows the Client Secret only once.

  2. Add a Service Auth policy. In Zero Trust → Access controls → Applications, open the application that protects the hostname you audit and add a policy to it with the action Service Auth and an Include rule with the selector Service Token set to the token you created. Service Auth policies are evaluated before Allow and Block policies, so your existing login policy stays as it is for people. The selector Any Access Service Token also works, but it admits every service token in your account.

  3. Add the headers to squirrelscan. Under Settings → Crawl → Custom request headers, add two headers and select Save headers:

    NameValue
    CF-Access-Client-Id<client-id>
    CF-Access-Client-Secret<client-secret>
  4. Audit the exact origin Access protects, with https:// and the exact hostname, so no redirect to another origin removes the headers before Access sees them. Rendering can stay on: pages rendered in the cloud browser carry the headers too (see which renders carry them).

  5. Check it with the verification checklist.

From your own machine, the difference looks like this:

curl -sS -o /dev/null -D - https://staging.example.com/ | grep -i -E '^(HTTP/|location)'
# HTTP/2 302
# location: https://<team-name>.cloudflareaccess.com/...

curl -sS -o /dev/null -D - \
  -H 'CF-Access-Client-Id: <client-id>' \
  -H 'CF-Access-Client-Secret: <client-secret>' \
  https://staging.example.com/ | grep -i -E '^(HTTP/|location)'
# HTTP/2 200

The location path and query vary. What matters is that the second request answers 200 and has no location on cloudflareaccess.com. A secret typed on the command line stays in your shell history, so clear it afterwards or test with a token you revoke.

Things that change the result:

  • Strict service token authentication. With it on, a request whose token does not match gets a 401 or 403 instead of the redirect, so a wrong or expired token makes the audit blocked rather than redirected.
  • Expiry. The token stops working at the end of its duration, and audits fail with the redirect again. Refresh extends a token by one year, and Cloudflare’s Expiring Access Service Token notification warns a week before expiry.
  • A single header. An Access application can be set to read the token from one header, such as Authorization, holding both values as JSON. Two headers need no extra setup, so use them unless your application already works that way.

Vercel deployment protection

Vercel Deployment Protection puts Vercel Authentication (a redirect to a Vercel login), Password Protection (a password page) or Trusted IPs in front of a deployment. Standard Protection, the recommended scope, covers every deployment except production domains, so a production custom domain needs no exception. Preview deployments and generated *.vercel.app URLs do.

Vercel lets automation through with Protection Bypass for Automation, available on every Vercel plan. The steps follow Vercel’s Protection Bypass for Automation documentation as of October 2026:

  1. Create the secret. In the Vercel project, open Settings → Deployment Protection and create a secret under Protection Bypass for Automation.
  2. Add the header to squirrelscan. Under Settings → Crawl → Custom request headers, add x-vercel-protection-bypass with the secret as its value, and select Save headers.
  3. Leave out x-vercel-set-bypass-cookie. It makes Vercel answer with a redirect that sets a cookie, which a crawl does not need: the header goes with every request.
  4. Check it with the verification checklist.
curl -sS -o /dev/null -D - https://my-project-git-staging-team.vercel.app/ | grep -i -E '^(HTTP/|location)'
curl -sS -o /dev/null -D - -H 'x-vercel-protection-bypass: <secret>' \
  https://my-project-git-staging-team.vercel.app/ | grep -i -E '^(HTTP/|location)'

The first request, without the header, is redirected to a Vercel login or gets the password page. The second answers 200.

The bypass also skips Vercel’s bot protection challenges and its system mitigations, but not an active DDoS mitigation. A secret works until it is revoked: when you regenerate or delete it in Vercel, update the header in squirrelscan.

HTTP basic auth

A site with HTTP basic auth, such as nginx auth_basic or Apache AuthType Basic, answers 401 with a WWW-Authenticate: Basic header, and the audit is blocked. A browser sends the username and password in an Authorization header, and a cloud audit can send the same header.

  1. Encode the credentials. The value is Basic followed by a space and the Base64 encoding of username:password:

    printf '%s' 'username:password' | base64
    # dXNlcm5hbWU6cGFzc3dvcmQ=

    Use printf '%s', not echo, so no newline is encoded with it. The output shown is the encoding of the placeholder username:password; your value will differ, and it decodes back to your real password, so treat it as a secret.

  2. Add the header. Under Settings → Crawl → Custom request headers, add Authorization with the value Basic <base64-value>, for example Basic dXNlcm5hbWU6cGFzc3dvcmQ=, and select Save headers. The header card accepts Authorization like any other name.

  3. Check it from your machine:

    curl -sS -o /dev/null -w '%{http_code}\n' -H 'Authorization: Basic <base64-value>' https://staging.example.com/
    # 200

Rendering can stay on: pages rendered in the cloud browser send the header too (see which renders carry them), and a render that still gets the 401 is fetched again over plain HTTP with it.

Use an account that exists only for the audit, because the credentials go with every request to the site and to the subdomains it loads assets from. HTTP Digest authentication cannot be passed this way: its Authorization value depends on a one-time value the server sends with each challenge, so a fixed header never matches.

Other zero-trust gateways

A gateway can be passed with a custom request header only when it offers a long-lived machine credential, as Cloudflare Access service tokens, Vercel’s bypass secret and HTTP basic auth do. These cannot:

GatewayWhy a saved header does not work
Okta, Auth0, Microsoft Entra ID, Amazon Cognito, OneLogin, Clerk, or another identity provider in front of your applicationThe application, or the proxy in front of it, sends an anonymous visitor to an interactive sign-in. The identity provider has no request header that skips it
Google Identity-Aware ProxyIt accepts OIDC ID tokens, which expire after about an hour, so a saved header stops working
Shopify storefront passwordShopify does not document a header that opens a password-protected storefront. Its crawler access signatures let a crawler through bot protection on a public store

For these, audit from somewhere the gateway already lets through, or take the gateway out of the path:

  • The CLI on a trusted network. squirrel audit fetches from your machine, so a gateway that admits requests from your office network or VPN admits the audit too. Access that depends on a browser session or a device check may not extend to the CLI’s requests. Signed in, the CLI renders pages in the cloud browser, which comes from squirrelscan’s network: add --http so every page is fetched from yours.
  • The CLI against a local build. Run the same site on your machine without the gateway and audit it there: squirrel audit http://localhost:3000. See local audits.
  • An exception in your own application or proxy. If you control the code or proxy that starts the sign-in, let requests through that carry a secret header with an exact value, as in option 1, and add that header to squirrelscan.
  • For a Shopify store, remove the storefront password for the time of the audit, or audit a development copy locally.

What squirrelscan cloud looks like in your logs

Cloud audits deliberately look like a browser so that the audit sees what your visitors see. Do not rely on a user agent match to identify them:

Request typeUser agent
Page fetchesA current desktop or mobile browser user agent, chosen at random per audit
Rendered pages (headless browser)The headless Chrome user agent from the rendering service
Asset checks and external link checksSquirrelScan/2.0 (+https://squirrelscan.com)

The cloud crawl does not have a custom user-agent setting. If you need a fixed identifier on every request, use a custom request header (option 1) rather than the user agent.

Checking it worked

Run an audit after saving the rule and open the report. A block that is still in place usually shows up as:

  • a low page count with many 403 or 429 responses in the crawl summary,
  • rules complaining that every page has the same title, such as “Attention Required” or “Just a moment”,
  • a rate-limited host warning.

If the audit still fails, see Failed audits or contact support from the dashboard with the audit id.

Verification checklist

Before the audit:

  1. Without credentials, curl -sS -o /dev/null -D - <audited-url> shows what the protection does: a 302 with a location on the sign-in host, a 401, a 403, or a challenge.
  2. With the same headers as the website’s custom request headers, added with -H, the same request answers 200 and has no location on a sign-in host.
  3. The audited address is the exact URL that second request used, scheme and host included.
  4. Settings → Crawl shows the headers saved, and the header card shows Authorised crawl: on (custom header).
  5. The pages you expect to audit are on the audited address’s host or its www twin, with the same scheme, so their cloud browser renders carry the headers too (see which renders carry them).

After the audit:

  1. The report audited at least one page. The start page’s status is 200 and its title is your page’s, not a sign-in or challenge page.

  2. A failed audit’s run page lists what happened under Technical details:

    Recorded attemptWhat it means
    A crawler or direct attempt redirected to the gateway’s originThe request reached the sign-in page: either the headers did not reach the protected origin (a redirect to another origin removed them) or the gateway did not accept them. Check the audited address, the token or secret, the policy (for Cloudflare Access, the Service Auth action and the selector), that the policy covers this hostname, and that the token has not expired
    A browser attempt that landed on the gateway’s sign-in, then a direct attemptThe render did not get through, so the page was fetched over plain HTTP with the headers instead. If the direct attempt landed on the sign-in too, read the first row
    A direct attempt with status 401 or 403The request was refused: by the gateway (for example Cloudflare Access with strict service token authentication and a token that does not match), or by another firewall or access rule in front of the site
    A cf-ray idSearch for it in Security → Events on Cloudflare to see which product acted on the request

See Failed audits for every cause and its fix.

Type to search…

↑↓ navigate↵ openesc close