Web Bot Auth
Audit bot-protected sites by sending signed Web Bot Auth headers
Some platforms block unknown crawlers by default. Web Bot Auth is an emerging web standard that fixes this with cryptography instead of user-agent strings: the crawler signs its requests using HTTP Message Signatures (RFC 9421) and the site verifies the signature against a published key. Cloudflare uses it for verified bots, and Shopify uses it to let merchants authorize custom crawlers on their stores.
squirrelscan can attach the signed headers to every crawl request (pages,
assets on the audited origin, robots.txt, sitemaps) via the
[crawler] headers map or the repeatable
-H / --header CLI flag, so the same recipe works anywhere the scheme is
accepted.
The three headers
Whichever platform issued your crawler credentials, the shape is the same:
| Header | Value |
|---|---|
Signature-Input | which request components are signed, plus the key id and creation time |
Signature | the signature value |
Signature-Agent | the quoted signing-agent URI, e.g. "https://shopify.com" for Shopify-issued keys |
The signature typically covers only the host (@authority) and the
signature-agent header, so one set of values works for a whole crawl of that
host until the signature expires.
Where the values come from depends on who verifies them:
- Shopify issues the header values directly: generate crawler access keys in your store admin (see Crawling your store) and paste the three values.
- Sites behind Cloudflare or self-hosted verifiers expect signatures from a key you own: sign with your Ed25519 key per RFC 9421 and publish your key directory as the architecture draft describes, then pass the resulting headers the same way.
CLI
Pass each header with a repeatable -H flag:
squirrel audit https://store.example.com \
-H 'Signature-Input: sig1=("@authority" "signature-agent");keyid="...";created=...' \
-H 'Signature: sig1=:BASE64SIG:' \
-H 'Signature-Agent: "https://signer.example.com"'Headers are attached to every request the crawl makes. The audit preamble lists the header names only; values are treated as secrets and never printed:
Headers Signature-Input: <redacted>, Signature: <redacted>, Signature-Agent: <redacted>Project config
To make the headers persistent for a project, add them to the [crawler]
section of your squirrel.toml:
[crawler]
headers = { "Signature-Input" = "sig1=(\"@authority\");keyid=\"...\"", "Signature" = "sig1=:BASE64SIG:", "Signature-Agent" = "\"https://signer.example.com\"" }CLI -H flags merge over the TOML map, so you can keep a base set in config
and override a single header per run.
Cloud audits
On the dashboard, set custom headers per-website under Settings → Crawl → Custom request headers. Custom headers for cloud audits are available on every plan, Free included. Plain HTTP requests carry them. For pages rendered in the cloud browser, see rendering and custom headers. When the site refuses a render, the page is fetched again over plain HTTP with the signed headers.
Other hosts
Only the audited origin gets the headers (scheme, host and port), so a page or
asset on a subdomain (cdn.example.com), another port, or plain http: is
requested without them. If that host needs the signature too, name it with header_hosts or
--header-host cdn.example.com; it gets them over https only, unless you write
it with an http:// prefix. A 401 or 403 from a host that was not sent the
headers is reported as not checked, with the host named.