Crawl settings

Cloud audits don’t read your local squirrel.toml, so the Website → Settings → Crawl page in the dashboard is where you control what the cloud crawls for a given website.

Open it at Dashboard → your website → Settings → Crawl.

The audit level applies to the audits you start by hand, scheduled audits and the first audit after you add the site. An API or MCP request that names its own level runs at that level instead. Crawl scope and the include and exclude patterns apply to every cloud audit of that website: those three, plus the GitHub check and an audit our infrastructure retries after a failure. None of it affects squirrel audit on your own machine, which reads your config file instead.

The GitHub check is the one trigger that does not take the rest. It is a fixed 25-page quick crawl whatever the site’s audit level says, and it sends no custom request headers, which means it is never an authorised crawl and does not read your robots.txt preference. Scope and patterns reach it because they can only ever narrow a crawl.

The page is three cards, each with its own save button, so saving one leaves the others alone.

Card	What it sets
Audit level	The site’s level, any setting you change to make it custom, and the max link depth.
Crawl rules	Crawl scope, robots.txt, incremental crawl, internal link checks, and the include and exclude patterns.
Custom request headers	Headers sent on every request, on every plan. Also what makes a crawl authorised.

Audit level

The top card is the site’s audit level: the same Quick, Surface and Full slider as the Add website form, with every setting of the level listed underneath it. The form only offers whole levels; this card is where a site gets its own settings.

• Pick a level and every setting fills in with that level’s values. A site on a level follows it, so a later change to the level’s definition reaches the site too.
• Change any setting and the site becomes Custom. The line above the settings says what it started from and what changed, for example Custom · Based on Surface, 2 changes: Pages, Rendering., and each changed setting carries a Changed tag.
• Reset to Surface (or whichever level it started from) drops every change. Picking a different level on the slider does too, and fills in that level’s values instead.

Setting	Choices
Pages	The most pages one audit reads, 1 to 10,000. An audit is held to your plan’s ceiling, and the line under the field says what that is, such as “Pro plan allows up to 2,000 pages per audit.”
Crawl strategy	Sitemap: the start page and your sitemaps, no link following. Sampled: follows links, one page per URL pattern first. All pages: every page it can reach.
Cloud checks	On or off: AI and authority signals, the editor’s summary, tech detection and domain stats.
Rendering	Off (plain HTTP), Auto (only pages that need JavaScript) or All (every page in a browser).
External link checks	On or off. Each distinct external destination the cloud verifies costs 1 credit.

Probing is not on the list: cloud audits probe passively whatever the level says, so a choice there would do nothing.

Max link depth sits in the same card but is not part of a level: how many link hops from the start page the crawler follows (the start page is 0). It applies at every level; leave it empty for the engine default. Save level saves both.

The box under the slider prices whatever the settings currently are: the level’s name, or Custom, and Up to N credits for one audit, after your plan’s page ceiling and your balance. It changes as you edit, so you see the cost of a change before you save it.

Full on the Free plan

On the Free plan the slider’s Full stop and the All pages crawl strategy are greyed out and tagged Pro. Free cannot run Full’s every-page crawl. A site already set to crawl every page, because it was set up before a downgrade or through the API, keeps its settings: nothing stored is rewritten, and this limit does not touch its schedule. Its audits sample the way Surface does instead, one page per URL pattern first, up to the plan’s 500 pages, and the card says so at the top. See Audit levels.

Sites added before audit levels

A site added before audit levels existed was moved onto them as Custom, with the settings it already had, so what it crawls and when it runs did not change (apart from the Free plan’s Full limit above). External link checks are now on for these sites too, as they are at every level, unless you turn them off, and each distinct external link a cloud audit verifies costs 1 credit. The slider sits on the level its old depth maps to: Fast on Quick, Surface and Deep on Surface, Full on Full. A site on the old Surface depth, for instance, reads Custom · Based on Surface, 1 change: Pages.: the old depth read 50 pages.

It stays that way until you change it. Reset it, or pick another level, and the site moves onto that level’s settings, and its price.

What the Run audit dialog quotes

Run New Audit on a website opens a dialog that prices the run on that site’s own level and settings, after your plan’s ceiling and your balance. It says what the audit will run at, such as “This audit runs at Surface, 100 pages.” or “This audit runs at Custom, based on Surface, 200 pages.”, with a link back to this page, and its button reads Run audit · up to N credits. The run uses the site’s settings too, so it runs at the level the dialog quoted. On Free, a site set to crawl every page also gets a sentence saying it will sample one page per URL pattern first instead.

Crawl scope

Two choices for where the crawl is allowed to go.

Sitemap only is for sites whose sitemap is the list you actually care about. It is the same idea as pointing a desktop crawler at a sitemap instead of a homepage.

It matters most on large sites. One store we audit publishes 5,322 URLs in its sitemap; following links grew the crawl to more than 20,000 candidate URLs, and the audit spent its budget on pages the owner had never asked about. Narrowing the scope puts that budget back on the pages that matter.

What sitemap scope does not change:

• robots.txt is still fetched and parsed, and still enforced if you have asked for that below
• your sitemap is still fetched and parsed, which is how the URLs are found in the first place
• the start URL is queued even when your sitemap does not list it
• include and exclude patterns still apply, so an excluded sitemap URL is still skipped
• assets on the pages being audited are still fetched, so rules about images, scripts and stylesheets still work

Queued is not the same as crawled: the start URL goes through your patterns and your robots.txt preference like any other URL. Exclude everything, start URL included, and the audit finishes with no pages rather than quietly widening.

If your sitemap turns out to have no crawlable URLs, because it is empty, unreachable, or everything in it is excluded, the crawl follows links rather than auditing nothing. Either way the run’s event log says which happened:

Crawl scope: sitemap only. 5322 sitemap URLs seeded, 2000 pages audited. Cached pages were not reused, because reusing one re-discovers its links.
Crawl scope: sitemap only, but the sitemap yielded no crawlable URLs, so links were followed instead. 412 pages audited.

The last sentence appears only when the audit would otherwise have reused cached pages, so a first audit of a site does not carry it.

What sitemap scope gives up

Two things a normal audit does are turned off, because each is a way a URL reaches the crawl behind the setting that is supposed to be keeping it out.

Pages carrying findings from your last audit are not re-queued. A normal audit puts them back in the queue so a fix can be confirmed, ahead of everything else. Under sitemap scope that would spend your page limit on pages the sitemap does not list, before reaching the ones it does. A page that is in your sitemap is still audited, so a fix on it is still confirmed.

Cached pages are not reused. Reusing a page from your last audit means re-reading the links on it, and there is no way to take the page without following them. So a sitemap-scoped audit fetches every page fresh. It is slower and costs more than the same audit with links followed, and the run’s event log says so rather than leaving you to find it on the bill.

Include and exclude patterns

Two lists of URL patterns, using the same syntax as the include and exclude options in the config file. * is the only special character, and it stands for any run of characters, / included:

Pattern	How it matches
/blog/*	prefix: any path starting /blog/, however deep
/blog	prefix: any path starting /blog, so /blogroll too
*.pdf	the whole path, anchored and case-insensitive
https://example.com/*?preview=*	the whole URL, anchored and case-insensitive

A pattern is matched against the URL’s path unless it starts with http:// or https://, in which case it is matched against the whole URL. A pattern that has to see a query string is therefore an absolute one. Include patterns are the exception, and the next section says why.

Leave include empty to crawl the whole site. Exclude wins: a URL matching both lists is excluded.

A non-empty include list replaces host scoping rather than narrowing within it, so every pattern is pinned to your own site’s addresses before the crawl runs. /blog/* means the blog on your site, not every address anywhere whose path starts with /blog/.

Pinning is also what stops an include list reaching off your site. An absolute include on somebody else’s origin is dropped, since honouring it would send the crawl somewhere nobody authorised. If every pattern in the list is one of those, the save is refused outright, because the result would be an audit that crawls nothing.

One consequence to know about: because an include pattern is pinned, it is matched against the whole URL rather than the path, and a query string is part of that. An include of *.pdf does not match /report.pdf?download=1. The same pattern as an exclude is not pinned, matches the path only, and does.

Limits

Each list is bounded. Save a list that breaks one of these and the page refuses it, rather than accepting it and quietly dropping the tail when the crawl runs. That matters most for excludes, where a dropped pattern means a crawl visits pages you believe you excluded.

Limit	Value
Patterns per list	50
Characters per pattern	500
Encoded size per list	4 KB
* wildcards per pattern	2, unless the pattern is exactly * or ends in /*

The wildcard cap only applies to patterns our matcher has to compile into a regular expression. /blog/* ends in /* and is a prefix test, so it is not counted; /a*b*c*d is, and is refused. Control characters are refused too, which is usually a pattern pasted in from a document.

Trimming whitespace and collapsing a repeated pattern are not refusals. They lose nothing you meant, so they just happen, and what the page shows back afterwards is what will be sent to the crawl.

What a retry uses

An audit records the scope it started with. If our infrastructure retries a failed run, the retry reproduces the crawl that failed, not a new one shaped by settings you changed in the meantime. Change a pattern and it applies to your next audit.

The exception is an audit old enough to predate the setting, which has no recorded scope to replay. Those fall back to the site’s current settings.

Respect robots.txt

Off by default: these are audits of your own site, started by you. robots.txt is always fetched and parsed regardless, because that is where Sitemap: directives are found and what the crawl/robots-txt rule reports on.

Turn the setting on and the crawler also skips URLs your robots.txt disallows, and waits between requests if your robots.txt asks it to with Crawl-delay.

Internal link checks

On by default. For each internal link target the crawl did not fetch, a cloud audit sends one status-only HEAD request, so links/broken-links finds a linked 404 even when the crawl did not follow links (the Quick level, Sitemap scope) or ran out of page budget. It is free: no page charge, no render, no rules. It sends a one-byte GET only when your server answers HEAD with 405 or 501.

The check never requests links that look like actions (log out, unsubscribe, delete, confirm, add to cart) or carry a token. It sends no cookies, credentials or custom request headers, and it re-checks every redirect hop against the crawl scope. See What the internal link check never requests.

Turn it off if requesting your links can change something on your site that those rules would not catch. With it off, an audit that did not follow links lists the internal link targets it could not see as Not checked, never as passing. Like the crawl scope, the setting applies to every cloud audit of the website, the GitHub check and retries included, and a retry never checks internal links that the run or the website has switched off.

An API or MCP request can set it for one run: internalLinks on POST /v1/agent-runs/cloud, or internal_links on the hosted MCP server’s run_audit.

Custom request headers

Custom request headers are available on every plan, Free included. The bottom card sets them for this website, and its cloud audits send them with the requests they make to your site: pages, whether fetched over plain HTTP or rendered in the cloud browser, assets on your audited origin (a subdomain such as cdn.example.com does not get them), robots.txt and sitemaps. The probe that checks where the audited address redirects before the crawl starts never sends them. For which requests carry them under each Rendering setting, including pages rendered in the cloud browser, see rendering and custom headers. On a plain HTTP request, the headers are removed at a redirect to a different origin, except User-Agent, Accept, Accept-Language and Upgrade-Insecure-Requests. In the cloud browser they go only to the rendered page’s own origin (see which renders carry them).

The card accepts any valid header name, Authorization included, up to 50 headers per website, with names up to 255 characters and values up to 8,192 characters (tabs allowed, no other control characters). Header values are stored as secrets and never shown in reports or logs.

The usual reason to add one is a firewall exception: Firewall and WAF exceptions has the rules. A header is also how a cloud audit gets past an access gateway in front of a staging site, such as Cloudflare Access, Vercel deployment protection or HTTP basic auth. Every Rendering setting sends them, the cloud browser included (to the rendered page’s own origin only). When an audit fails anyway, failed audits says what the crawler saw. A header also makes the crawl authorised.

Authorised crawls

Configure a custom request header for this website and its cloud audits run as authorised. The header card shows the status, Authorised crawl: on (custom header).

Setting a header that only your origin knows about is you telling us this crawler is yours, and it is usually the same header that exempts us from your rate limit in the first place. An authorised crawl:

• runs at higher concurrency, with no fixed wait between requests
• does not apply robots.txt at all

While a crawl is authorised, the Respect robots.txt toggle is shown disabled. Your saved preference is kept and applies again if authorisation goes away.

Authorisation is decided from the headers the audit will actually send. Every plan permits them today; if a plan ever stopped permitting them, the headers would not be sent, so the crawl would not be authorised and would return to polite pacing on its own.

An authorised crawl never reuses pages from your last audit either, for a different reason: the requests carry your header, and an authenticated response must not be stored where another audit could read it.

An authorised crawl is not unlimited. It still backs off per host when your server answers 429, or answers 503 with a Retry-After, and it waits as long as that header asks for. A bare 503 with no Retry-After reads as an outage rather than a request to slow down. A site that pushes back is still crawled at the rate it will accept.

See also