Web Bot Auth (Shopify)

Some platforms block unknown crawlers by default. Shopify’s Web Bot Auth lets a merchant authorize their own crawler so it isn’t blocked — but it requires three signed headers on every request. squirrelscan can attach those headers to every crawl request (pages, assets, robots.txt, sitemaps) via the [crawler] headers map or the repeatable -H / --header CLI flag.

Web Bot Auth is the IETF HTTP Message Signatures standard (also used by Cloudflare Verified Bots), so the same recipe works anywhere that scheme is accepted.

The three headers

Shopify gives you the values from your store admin. They are always the same shape:

CLI

Pass each header with a repeatable -H flag:

squirrel audit https://your-store.myshopify.com \
  -H 'Signature-Input: sig1=("@authority" "signature-agent");keyid="...";created=...' \
  -H 'Signature: sig1=:BASE64SIG:' \
  -H 'Signature-Agent: "https://shopify.com"'

Headers are attached to every request the crawl makes. The audit preamble lists the header names only — values are treated as secrets and never printed:

Headers   Signature-Input: <redacted>, Signature: <redacted>, Signature-Agent: <redacted>

Project config

To make the headers persistent for a project, add them to the [crawler] section of your squirrel.toml:

[crawler]
headers = { "Signature-Input" = "sig1=(\"@authority\");keyid=\"...\"", "Signature" = "sig1=:BASE64SIG:", "Signature-Agent" = "\"https://shopify.com\"" }

CLI -H flags merge over the TOML map, so you can keep a base set in config and override a single header per run.

Cloud audits (Pro)

On the dashboard, set custom headers per-website under Settings → Crawl → Custom request headers. Applying custom headers to cloud audits is a Pro feature — free plans receive a 403 upgrade_required. The cloud render worker applies the headers to the headless browser before navigation (setExtraHTTPHeaders), so they ride the initial load and every subresource.

Related

• Crawler settings → headers
• Browser rendering
• Shopify — Crawling your store