URL: /guides/web-bot-auth

---
title: "Web Bot Auth (Shopify)"
description: "Audit a protected Shopify store by sending signed Web Bot Auth headers"
---

Some platforms block unknown crawlers by default. [Shopify's Web Bot
Auth](https://changelog.shopify.com/posts/authorize-custom-crawlers-and-tools-with-new-crawler-access-keys)
lets a merchant authorize their own crawler so it isn't blocked — but it requires
**three signed headers on every request**. squirrelscan can attach those headers
to every crawl request (pages, assets, `robots.txt`, sitemaps) via the
[`[crawler] headers`](/configuration/crawler#headers) map or the repeatable
`-H` / `--header` CLI flag.

Web Bot Auth is the IETF [HTTP Message
Signatures](https://www.rfc-editor.org/rfc/rfc9421.html) standard (also used by
[Cloudflare Verified Bots](https://blog.cloudflare.com/verified-bots-with-cryptography/)),
so the same recipe works anywhere that scheme is accepted.

## The three headers

Shopify gives you the values from your store admin. They are always the same
shape:

| Header | Value |
| --- | --- |
| `Signature-Input` | the signature input (from Shopify admin) |
| `Signature` | the signature value (from Shopify admin) |
| `Signature-Agent` | the quoted signing-agent URI — `"https://shopify.com"` |

<Note>
  `Signature-Agent` is a quoted string. Keep the inner double quotes —
  `Signature-Agent: "https://shopify.com"` — squirrelscan preserves the quoting
  verbatim end-to-end.
</Note>

## CLI

Pass each header with a repeatable `-H` flag:

```bash
squirrel audit https://your-store.myshopify.com \
  -H 'Signature-Input: sig1=("@authority" "signature-agent");keyid="...";created=...' \
  -H 'Signature: sig1=:BASE64SIG:' \
  -H 'Signature-Agent: "https://shopify.com"'
```

Headers are attached to every request the crawl makes. The audit preamble lists
the header **names** only — values are treated as secrets and never printed:

```
Headers   Signature-Input: <redacted>, Signature: <redacted>, Signature-Agent: <redacted>
```

<Warning>
  Header values are secrets. Prefer the `-H` flag in CI with values sourced from
  your secret store rather than committing real signatures to a shared
  `squirrel.toml`.
</Warning>

## Project config

To make the headers persistent for a project, add them to the `[crawler]`
section of your `squirrel.toml`:

```toml
[crawler]
headers = { "Signature-Input" = "sig1=(\"@authority\");keyid=\"...\"", "Signature" = "sig1=:BASE64SIG:", "Signature-Agent" = "\"https://shopify.com\"" }
```

CLI `-H` flags **merge over** the TOML map, so you can keep a base set in config
and override a single header per run.

## Cloud audits (Pro)

On the dashboard, set custom headers per-website under **Settings → Crawl →
Custom request headers**. Applying custom headers to **cloud** audits is a
**Pro** feature — free plans receive a `403 upgrade_required`. The cloud render
worker applies the headers to the headless browser before navigation
(`setExtraHTTPHeaders`), so they ride the initial load and every subresource.

## Related

- [Crawler settings → `headers`](/configuration/crawler#headers)
- [Browser rendering](/guides/browser-rendering)
- [Shopify — Crawling your store](https://help.shopify.com/en/manual/promoting-marketing/seo/crawling-your-store)