Link checking
What squirrelscan checks about your links, how it tells a broken link from a blocked one, what Not checked means, and what is coming
Broken links are easy to make and hard to notice: a page gets renamed, a section is restructured, a site you cite shuts down. squirrelscan checks the links on the pages it audits, so you hear about it before your visitors do.
This page covers what is checked today, how to read the results, and what is coming. Sections marked Coming soon describe work in progress, not something you can use yet.
What gets checked
| Internal links | External links | |
|---|---|---|
| What | Links to pages on your own site | Links to other sites |
| Rules | links/broken-links | links/broken-external-links, links/dead-links |
| Quick level | On | On |
| Surface and Full levels | On | On |
Each audit checks up to a per-audit limit of links, the most-linked first. The limits and every switch are in Link checking defaults.
Internal links
A link to your own site is checked two ways:
- Pages the crawl fetched are checked from the crawl itself: a target that answered with an error (400 or above, other than throttling) is a broken link.
- Pages the crawl did not fetch get a status-only check each, the most-linked first. This is what lets a Quick audit, which does not follow links, find a 404 that only a link points at. It also finds a broken page past a Surface or Full audit’s page budget.
The status-only check is built not to change anything on your site:
- It never requests links that look like actions (log out, unsubscribe, delete, confirm, add to cart) or that carry a token.
- It sends no cookies, credentials or custom request headers.
- It stays inside the crawl scope and, when the crawl respects it, robots.txt.
The full rules are in What the internal link check never requests. Switch it off with --no-internal-links or [internal_links] enabled = false.
External links
External links are checked at every level, Quick included. The links on the most pages go first: the nav and footer links every page shares. Links past the audit’s limit are listed as Not checked.
Where the check runs depends on how you audit:
- A cloud audit checks external links with squirrelscan’s link checker, Perch, which shares what it learns across audits. External link checks are a website setting in the dashboard.
- A signed-in CLI audit at Surface or Full sends its external links to the same checker. Whatever Perch has not answered in time, the CLI checks itself from your machine.
- Signed out, or at Quick, the CLI checks external links from your machine.
--offlineand--no-external-linksskip them.
What it costs
External link checks cost credits, at most 1 per page of the audit’s page limit:
- A cloud audit pays 1 credit for each distinct external link Perch can verify: one that works, redirects or is broken. A link it cannot verify costs nothing, and the report lists it as unverified: one behind a bot wall, a throttled one, one the run’s limit on deeper checks for its site left unverified, or a 404 waiting for its confirming check. A link Perch does not answer in time costs nothing either. The audit reserves its link credits up front and is settled down to the links Perch verified. A link the audit has already paid for is not charged again.
- A signed-in CLI audit pays the same way, for each distinct external link Perch verifies. A link Perch leaves unanswered is not charged either: the CLI checks it from your machine for free.
A link to another of your own domains, such as example.co.uk linking to example.com, is an external link. It is checked, reported and charged like any other. Only the audited site’s own host and its www twin are internal.
Everything else is free: internal links, every link a signed-out audit checks, and a Quick CLI audit’s external links, checked from your machine. The audit’s estimate includes the link checks before it starts. See Credits and pricing.
Like the internal check, the external check never requests a link that looks like an action or carries a token: a newsletter’s one-click unsubscribe, a magic sign-in link, a signed download. It never follows a redirect into one either. See What the external link check never requests.
How Perch tells broken from blocked
Many sites refuse automated requests. A refusal says nothing about whether a person would reach the page. So Perch does not take the first answer at face value:
- A link is reported broken only after the failure is confirmed. Until then it is listed as status unverifiable, not broken, and an item the link checker marks as waiting reads
(unverified: awaiting confirmation), likely broken rather than blocked. A link checked again by a later audit counts toward this too. - A domain that no longer exists is reported as dead.
- A page that answers 200 but whose title is just “404” or “Page not found” (a soft 404) counts as broken.
- A link that still cannot be verified is listed as status unverifiable, never as broken. This covers firewall challenges, bot walls and CDN error pages. A rate limit is listed as rate-limited, never as broken.
The first check of a new link
Perch’s first answer for a link it has never seen can take a while. A cloud audit hands its external links to Perch early, so most are answered by the time the audit needs them. Any link Perch has not answered in time gets a quicker single check instead, and that check can still report a blocked link as broken. The next audit gets Perch’s answer.
Checking links from the CLI without signing in
Signed out, there is no Perch. When squirrel audit checks external links, every request goes straight from your machine: your laptop, a CI runner, a server. A lot of sites bot-block exactly that kind of traffic, so expect plenty of false positives: links reported broken that load fine in your browser.
Sign in with squirrel auth login, and a Surface or Full audit (the signed-in default) checks external links with Perch instead, with the same confirmed-or-unverified answers as a cloud audit. Those checks cost credits, as above.
When the CLI checks a link itself (signed out, at Quick, or when Perch has not answered in time), it keeps walls apart from broken links:
- A sign-in or bot wall (a 401, 403, 405, 406 or LinkedIn’s 999), a challenge page served as a 503, or a site that never answers in time: listed as status unverifiable under
waf-blocked-external-links, never broken. An audit that checked such links only from your machine says so after it finishes, with what gets them checked by Perch and what that costs: signed out, sign in and run a Surface or Full audit; signed in at Quick, run--level surfaceor--level full. - Throttled (429 or 430, or a 503 with
Retry-After): listed as rate-limited, status unverifiable, never broken. - An internal link that answers 401 or 403 to the status-only check, or that answered the crawl with a bot-wall challenge: listed as refused, never broken.
For an external link, a real 404 or 410, a plain 503, a domain that does not exist and a refused connection are still broken, and so are a storage bucket’s 403 for a missing file and an HTTP sign-in prompt. A site can still answer a check from your machine with an error it would never show a person, so a link in the broken list may load fine in your browser. Signed in, Perch’s confirmed answers avoid most of these.
Not checked
When a check did not run, squirrelscan says so rather than reporting the rule as passing. Every report format lists these under Not checked, with the reason and what to run instead. The score leaves them out.
| Reason | What happened |
|---|---|
external-links-not-checked | External link checking was off: --offline, --no-external-links, [external_links] enabled = false, or a website or run that switched it off. |
external-links-over-cap | The audit found more external links than its limit. The most-linked were checked. |
external-links-out-of-time | A cloud audit’s link check ran out of time, or waited too long on one request, before these links were answered. |
link-targets-not-fetched | The audit did not follow links, and some internal link targets were fetched neither by the crawl nor by the status-only check: it was off, over its limit, skipped the targets, got no answer, or could not be trusted because the site answers status-only requests with errors. |
In the cloud’s issue tracker, an open broken-link issue stays open while an audit could not check links for one of the reasons above.
Coming soon: getting past the last walls
Some sites still refuse every check we can make today. We are working on those too, so fewer links end up unverifiable.
Coming soon: link monitoring
For paid plans, a website on a recurring audit schedule will have its links re-checked between audits. You will be told when a link breaks, when it recovers, or when a domain it points to disappears, in the dashboard and by email if you have alerts on.
See also
- Link checking defaults: every switch and limit, per level and per surface.
- External and internal link configuration: limits, timeouts and caching.
- Audit levels: what each level checks.
- Crawling rate-limited hosts: why a throttled link is never reported broken.