URL: /dashboard/crawl-settings
---
title: "Crawl settings"
description: "Set a website's audit level, which URLs its cloud audits crawl, and how fast"
icon: "route"
---
Cloud audits don't read your local `squirrel.toml`, so the **Website → Settings → Crawl** page in the dashboard is where you control what the cloud crawls for a given website.
Open it at **Dashboard → your website → Settings → Crawl**.
The [audit level](#audit-level) applies to the audits you start by hand, scheduled audits and the first audit after you add the site. An API or MCP request that names its own level runs at that level instead. Crawl scope and the include and exclude patterns apply to **every cloud audit of that website**: those three, plus the GitHub check and an audit our infrastructure retries after a failure. None of it affects `squirrel audit` on your own machine, which reads your [config file](/configuration/crawler) instead.
The GitHub check is the one trigger that does not take the rest. It is a fixed 25-page quick crawl whatever the site's audit level says, and it sends no custom request headers, which means it is never an [authorised crawl](#authorised-crawls) and does not read your robots.txt preference. Scope and patterns reach it because they can only ever narrow a crawl.
The page is three cards, each with its own save button, so saving one leaves the others alone.
| Card | What it sets |
| --- | --- |
| **Audit level** | The site's level, any setting you change to make it custom, and the max link depth. |
| **Crawl rules** | Crawl scope, robots.txt, incremental crawl, internal link checks, and the include and exclude patterns. |
| **Custom request headers** | Headers sent on every request, on [every plan](#custom-request-headers). Also what makes a crawl [authorised](#authorised-crawls). |
## Audit level
The top card is the site's [audit level](/guides/audit-levels): the same **Quick**, **Surface** and **Full** slider as the [Add website](/dashboard/add-website) form, with every setting of the level listed underneath it. The form only offers whole levels; this card is where a site gets its own settings.
- **Pick a level** and every setting fills in with that level's values. A site on a level follows it, so a later change to the level's definition reaches the site too.
- **Change any setting** and the site becomes **Custom**. The line above the settings says what it started from and what changed, for example **Custom · Based on Surface, 2 changes: Pages, Rendering.**, and each changed setting carries a **Changed** tag.
- **Reset to Surface** (or whichever level it started from) drops every change. Picking a different level on the slider does too, and fills in that level's values instead.
| Setting | Choices |
| --- | --- |
| **Pages** | The most pages one audit reads, 1 to 10,000. An audit is held to your plan's ceiling, and the line under the field says what that is, such as "Pro plan allows up to 2,000 pages per audit." |
| **Crawl strategy** | **Sitemap**: the start page and your sitemaps, no link following. **Sampled**: follows links, one page per URL pattern first. **All pages**: every page it can reach. |
| **Cloud checks** | On or off: AI and authority signals, the editor's summary, tech detection and domain stats. |
| **Rendering** | **Off** (plain HTTP), **Auto** (only pages that need JavaScript) or **All** (every page in a browser). |
| **External link checks** | On or off. Each distinct external destination the cloud verifies costs 1 credit. |
Probing is not on the list: cloud audits probe passively whatever the level says, so a choice there would do nothing.
**Max link depth** sits in the same card but is not part of a level: how many link hops from the start page the crawler follows (the start page is 0). It applies at every level; leave it empty for the engine default. **Save level** saves both.
The box under the slider prices whatever the settings currently are: the level's name, or **Custom**, and **Up to N credits** for one audit, after your plan's page ceiling and your balance. It changes as you edit, so you see the cost of a change before you save it.
### Full on the Free plan
On the **Free** plan the slider's Full stop and the **All pages** crawl strategy are greyed out and tagged **Pro**. Free cannot run Full's every-page crawl. A site already set to crawl every page, because it was set up before a downgrade or through the API, keeps its settings: nothing stored is rewritten, and this limit does not touch its schedule. Its audits sample the way Surface does instead, one page per URL pattern first, up to the plan's 500 pages, and the card says so at the top. See [Audit levels](/guides/audit-levels#full-on-the-free-plan).
### Sites added before audit levels
A site added before audit levels existed was moved onto them as **Custom**, with the settings it already had, so what it crawls and when it runs did not change (apart from the Free plan's Full limit above). External link checks are now on for these sites too, as they are at every level, unless you turn them off, and each distinct external link a cloud audit verifies costs 1 credit. The slider sits on the level its old depth maps to: Fast on Quick, Surface and Deep on Surface, Full on Full. A site on the old Surface depth, for instance, reads **Custom · Based on Surface, 1 change: Pages.**: the old depth read 50 pages.
It stays that way until you change it. Reset it, or pick another level, and the site moves onto that level's settings, and its price.
### What the Run audit dialog quotes
**Run New Audit** on a website opens a dialog that prices the run on that site's own level and settings, after your plan's ceiling and your balance. It says what the audit will run at, such as "This audit runs at Surface, 100 pages." or "This audit runs at Custom, based on Surface, 200 pages.", with a link back to this page, and its button reads **Run audit · up to N credits**. The run uses the site's settings too, so it runs at the level the dialog quoted. On Free, a site set to crawl every page also gets a sentence saying it will sample one page per URL pattern first instead.
## Crawl scope
Two choices for where the crawl is allowed to go.
Start from your sitemap and follow the links found on those pages. This is the default and what every existing website uses.
Audit the URLs your sitemap publishes, and stop there. Links found on those pages are not followed.
The **Quick** level, and the **Sitemap** crawl strategy on a custom level, do not follow links while the sitemap has pages to give, whichever scope you pick: only a site with no usable sitemap has its links followed. On those settings a **Follow links** crawl behaves like a sitemap-only one. Sites added on the old **Fast** depth use the Sitemap strategy too. Pick Surface or Full if following links is the point.
**Sitemap only** is for sites whose sitemap is the list you actually care about. It is the same idea as pointing a desktop crawler at a sitemap instead of a homepage.
It matters most on large sites. One store we audit publishes 5,322 URLs in its sitemap; following links grew the crawl to more than 20,000 candidate URLs, and the audit spent its budget on pages the owner had never asked about. Narrowing the scope puts that budget back on the pages that matter.
The page limit still applies. Sitemap scope changes **which** URLs are eligible, not how many are crawled. If your sitemap lists more URLs than your page limit allows, the crawl stops at the limit.
What sitemap scope does **not** change:
- robots.txt is still fetched and parsed, and still enforced if you have asked for that below
- your sitemap is still fetched and parsed, which is how the URLs are found in the first place
- the start URL is queued even when your sitemap does not list it
- include and exclude patterns still apply, so an excluded sitemap URL is still skipped
- assets on the pages being audited are still fetched, so rules about images, scripts and stylesheets still work
Queued is not the same as crawled: the start URL goes through your patterns and your robots.txt preference like any other URL. Exclude everything, start URL included, and the audit finishes with no pages rather than quietly widening.
If your sitemap turns out to have no crawlable URLs, because it is empty, unreachable, or everything in it is excluded, the crawl follows links rather than auditing nothing. Either way the run's event log says which happened:
```
Crawl scope: sitemap only. 5322 sitemap URLs seeded, 2000 pages audited. Cached pages were not reused, because reusing one re-discovers its links.
Crawl scope: sitemap only, but the sitemap yielded no crawlable URLs, so links were followed instead. 412 pages audited.
```
The last sentence appears only when the audit would otherwise have reused cached pages, so a first audit of a site does not carry it.
### What sitemap scope gives up
Two things a normal audit does are turned off, because each is a way a URL reaches the crawl behind the setting that is supposed to be keeping it out.
**Pages carrying findings from your last audit are not re-queued.** A normal audit puts them back in the queue so a fix can be confirmed, ahead of everything else. Under sitemap scope that would spend your page limit on pages the sitemap does not list, before reaching the ones it does. A page that is in your sitemap is still audited, so a fix on it is still confirmed.
**Cached pages are not reused.** Reusing a page from your last audit means re-reading the links on it, and there is no way to take the page without following them. So a sitemap-scoped audit fetches every page fresh. It is slower and costs more than the same audit with links followed, and the run's event log says so rather than leaving you to find it on the bill.
## Include and exclude patterns
Two lists of URL patterns, using the same syntax as the [`include` and `exclude` options](/configuration/crawler#include) in the config file. `*` is the only special character, and it stands for any run of characters, `/` included:
| Pattern | How it matches |
| --- | --- |
| `/blog/*` | prefix: any path starting `/blog/`, however deep |
| `/blog` | prefix: any path starting `/blog`, so `/blogroll` too |
| `*.pdf` | the whole path, anchored and case-insensitive |
| `https://example.com/*?preview=*` | the whole URL, anchored and case-insensitive |
A pattern is matched against the URL's path unless it starts with `http://` or `https://`, in which case it is matched against the whole URL. A pattern that has to see a query string is therefore an absolute one. Include patterns are the exception, and the next section says why.
Leave **include** empty to crawl the whole site. **Exclude wins**: a URL matching both lists is excluded.
**A non-empty include list replaces host scoping rather than narrowing within it**, so every pattern is pinned to your own site's addresses before the crawl runs. `/blog/*` means the blog on your site, not every address anywhere whose path starts with `/blog/`.
Pinning is also what stops an include list reaching off your site. An absolute include on somebody else's origin is dropped, since honouring it would send the crawl somewhere nobody authorised. If **every** pattern in the list is one of those, the save is refused outright, because the result would be an audit that crawls nothing.
One consequence to know about: because an include pattern is pinned, it is matched against the whole URL rather than the path, and a query string is part of that. An include of `*.pdf` does not match `/report.pdf?download=1`. The same pattern as an **exclude** is not pinned, matches the path only, and does.
### Limits
Each list is bounded. Save a list that breaks one of these and the page **refuses** it, rather than accepting it and quietly dropping the tail when the crawl runs. That matters most for excludes, where a dropped pattern means a crawl visits pages you believe you excluded.
| Limit | Value |
| --- | --- |
| Patterns per list | 50 |
| Characters per pattern | 500 |
| Encoded size per list | 4 KB |
| `*` wildcards per pattern | 2, unless the pattern is exactly `*` or ends in `/*` |
The wildcard cap only applies to patterns our matcher has to compile into a regular expression. `/blog/*` ends in `/*` and is a prefix test, so it is not counted; `/a*b*c*d` is, and is refused. Control characters are refused too, which is usually a pattern pasted in from a document.
Trimming whitespace and collapsing a repeated pattern are not refusals. They lose nothing you meant, so they just happen, and what the page shows back afterwards is what will be sent to the crawl.
### What a retry uses
An audit records the scope it started with. If our infrastructure retries a failed run, the retry reproduces **the crawl that failed**, not a new one shaped by settings you changed in the meantime. Change a pattern and it applies to your next audit.
The exception is an audit old enough to predate the setting, which has no recorded scope to replay. Those fall back to the site's current settings.
## Respect robots.txt
Off by default: these are audits of your own site, started by you. robots.txt is always fetched and parsed regardless, because that is where `Sitemap:` directives are found and what the [`crawl/robots-txt`](/rules/crawl/robots-txt) rule reports on.
Turn the setting on and the crawler also skips URLs your robots.txt disallows, and waits between requests if your robots.txt asks it to with `Crawl-delay`.
A `Crawl-delay` is a hard limit on how fast a host can be crawled, because the wait applies per host no matter how many workers are running. The directive is capped at 2 seconds even when it asks for longer, and 2 seconds a page is 1,800 pages an hour from one host. Most sites are one host, so for most sites that is the whole crawl. If your page count cannot finish within the audit's time budget at that pace, the audit says so at the start and crawls fewer pages rather than running out of time with nothing to show. An [authorised crawl](#authorised-crawls) is the way to tell us that limit is not meant for us.
## Internal link checks
On by default. For each internal link target the crawl did not fetch, a cloud audit sends one status-only `HEAD` request, so [`links/broken-links`](/rules/links/broken-links) finds a linked 404 even when the crawl did not follow links (the Quick level, Sitemap scope) or ran out of page budget. It is free: no page charge, no render, no rules. It sends a one-byte `GET` only when your server answers `HEAD` with 405 or 501.
The check never requests links that look like actions (log out, unsubscribe, delete, confirm, add to cart) or carry a token. It sends no cookies, credentials or custom request headers, and it re-checks every redirect hop against the crawl scope. See [What the internal link check never requests](/configuration/external-links#what-the-internal-link-check-never-requests).
Turn it off if requesting your links can change something on your site that those rules would not catch. With it off, an audit that did not follow links lists the internal link targets it could not see as **Not checked**, never as passing. Like the crawl scope, the setting applies to every cloud audit of the website, the GitHub check and retries included, and a retry never checks internal links that the run or the website has switched off.
An API or MCP request can set it for one run: `internalLinks` on `POST /v1/agent-runs/cloud`, or `internal_links` on the hosted MCP server's `run_audit`.
## Custom request headers
Custom request headers are available on every plan, Free included. The bottom card sets them for this website, and its cloud audits send them with the requests they make to your site: pages, whether fetched over plain HTTP or rendered in the cloud browser, assets on your audited origin (a subdomain such as `cdn.example.com` does not get them), `robots.txt` and sitemaps. The probe that checks where the audited address redirects before the crawl starts never sends them. For which requests carry them under each **Rendering** setting, including pages rendered in the cloud browser, see [rendering and custom headers](/cloud/firewall#rendering-and-custom-headers). On a plain HTTP request, the headers are removed at a redirect to a different origin, except `User-Agent`, `Accept`, `Accept-Language` and `Upgrade-Insecure-Requests`. In the cloud browser they go only to the rendered page's own origin (see [which renders carry them](/cloud/firewall#which-renders-carry-them)).
The card accepts any valid header name, `Authorization` included, up to 50 headers per website, with names up to 255 characters and values up to 8,192 characters (tabs allowed, no other control characters). Header values are stored as secrets and never shown in reports or logs.
The usual reason to add one is a firewall exception: [Firewall and WAF exceptions](/cloud/firewall) has the rules. A header is also how a cloud audit gets past an access gateway in front of a staging site, such as [Cloudflare Access](/cloud/firewall#cloudflare-access), [Vercel deployment protection](/cloud/firewall#vercel-deployment-protection) or [HTTP basic auth](/cloud/firewall#http-basic-auth). Every **Rendering** setting sends them, the cloud browser included (to the rendered page's own origin only). When an audit fails anyway, [failed audits](/guides/failed-audits#access-gateway) says what the crawler saw. A header also makes the crawl [authorised](#authorised-crawls).
## Authorised crawls
Configure a **custom request header** for this website and its cloud audits run as **authorised**. The header card shows the status, *Authorised crawl: on (custom header)*.
Setting a header that only your origin knows about is you telling us this crawler is yours, and it is usually the same header that exempts us from your rate limit in the first place. An authorised crawl:
- runs at higher concurrency, with no fixed wait between requests
- **does not apply robots.txt at all**
That second point is wider than it sounds. `Disallow` and `Crawl-delay` are governed by one setting, so an authorised crawl ignores both. **Anything you were keeping out of the crawl with `Disallow` needs an exclude pattern instead.**
While a crawl is authorised, the **Respect robots.txt** toggle is shown disabled. Your saved preference is kept and applies again if authorisation goes away.
Authorisation is decided from the headers the audit will actually send. Every plan permits them today; if a plan ever stopped permitting them, the headers would not be sent, so the crawl would not be authorised and would return to polite pacing on its own.
An authorised crawl never reuses pages from your last audit either, for a different reason: the requests carry your header, and an authenticated response must not be stored where another audit could read it.
An authorised crawl is not unlimited. It still backs off per host when your server answers `429`, or answers `503` with a `Retry-After`, and it waits as long as that header asks for. A bare `503` with no `Retry-After` reads as an outage rather than a request to slow down. A site that pushes back is still crawled at the rate it will accept.
## See also
Which rules run for this website's cloud audits.
The same ideas for local audits, in `squirrel.toml`.