External Links Configuration

The [external_links] section controls how squirrelscan validates outbound links during crawls.

Configuration

[external_links]
enabled = true
cache_ttl_days = 7
timeout_ms = 10000
concurrency = 5
# max_links = 500   # unset: the page budget or 200, whichever is more

Internal links have a section of their own, [internal_links].

Options

enabled

Type: boolean Default: true, which leaves the audit level in charge: checked at every level

Enable external link checking during crawl.

When enabled, squirrelscan validates all external links found during crawling to detect broken outbound links (404s, timeouts, DNS failures).

External link checks are one of the audit level’s settings. true is the default squirrel init writes into every config, so it reads as unset: every level checks external links. false turns them off at every level, and makes the audit custom. For one run, --external-links checks them at any level and --no-external-links skips them; either beats this setting. With them off, links/broken-external-links reports Not checked, never a pass.

Examples:

Follow the level (default):

[external_links]
enabled = true

Disable for faster crawls, at every level:

[external_links]
enabled = false

When to disable:

• Local development (localhost URLs)
• Network restrictions (firewalls, VPNs)
• Speed priority over external link validation
• Large sites with many external links

Impact when disabled:

• Faster crawls (no external HTTP requests)
• links/broken-external-links reports Not checked instead of findings
• Outbound link quality not validated

max_links

Type: number Default: unset: the audit’s page budget or 200, whichever is more

The most distinct external links one audit requests. The links on the most pages go first (the nav and footer links every page shares), then the rest by URL, so the same crawl always checks the same set. Links over the limit are reported as Not checked with reason external-links-over-cap. It bounds requests, not credits: a cloud audit is charged for at most one external link per page of its budget either way.

[external_links]
max_links = 500

cache_ttl_days

Type: number Default: 7 (days) Range: 1 to 365 recommended

How long to cache external link check results in days.

External link checks are cached globally per URL to avoid repeatedly checking the same external resources across multiple crawls.

How caching works:

1. First crawl checks https://example.com/article
2. Result cached for 7 days (default)
3. Next crawl within 7 days reuses cached result
4. After 7 days, link is re-checked

Only audits of public sites read and write the cache. An audit of a site on a private network (localhost, a private address, a .local name) checks every link afresh and caches nothing. Whatever it reached from inside your network is never handed to another audit.

Examples:

Short cache (1 day):

[external_links]
cache_ttl_days = 1

Long cache (30 days):

[external_links]
cache_ttl_days = 30

No caching (always fresh):

[external_links]
cache_ttl_days = 0  # Not recommended

Recommendations:

Cache location:

~/.squirrel/cache/external-links/

Clear cache:

rm -rf ~/.squirrel/cache/external-links/

timeout_ms

Type: number Default: 10000 (10 seconds) Range: 1000 to 60000 (1-60 seconds)

Timeout for external link checks in milliseconds.

External link checks use HEAD requests by default (faster, no body download). If a site doesn’t respond within this timeout, it’s marked as failed.

Examples:

Fast timeout (5 seconds):

[external_links]
timeout_ms = 5000

Slow sites tolerance (30 seconds):

[external_links]
timeout_ms = 30000

Very aggressive (3 seconds):

[external_links]
timeout_ms = 3000

When request exceeds timeout:

• Link marked as “timeout”
• Reported as broken external link
• Counted in links/broken-external-links rule

Recommendations:

concurrency

Type: number Default: 5 Range: 1 to 20 recommended

Maximum number of concurrent external link checks.

Controls how many external URLs are validated simultaneously during crawling.

Examples:

Sequential (one at a time):

[external_links]
concurrency = 1

Moderate parallelism (default):

[external_links]
concurrency = 5

High parallelism:

[external_links]
concurrency = 15

Impact:

Higher concurrency:

• ✓ Faster external link validation
• ✓ Better for sites with many external links
• ✗ More network connections
• ✗ May trigger rate limits

Lower concurrency:

• ✓ Polite to external sites
• ✓ Less network overhead
• ✗ Slower external link validation

Recommendations:

Internal links

The [internal_links] section controls the internal link check: a status-only HEAD request for each internal link target the crawl did not fetch, so links/broken-links finds a linked 404 even when the crawl did not follow links (the Quick level, the Sitemap crawl strategy) or ran out of page budget. No render, no rules and no credits. It requests the audited site only, so it runs with --offline too, and it stays inside the crawl scope and, when the crawl respects it, robots.txt. It never requests a link that looks like an action or carries a token, and never sends credentials; see What the internal link check never requests.

[internal_links]
enabled = true      # false, or --no-internal-links for one run, switches it off
max_targets = 200   # most targets one audit checks, most-linked first; 0 checks none

With it off, or past max_targets, an audit that did not follow links reports the targets it could not see as Not checked. max_targets = 0 checks none, and reports them the same way; enabled = false is the way to switch it off.

What the internal link check never requests

The internal link check requests links the crawl did not choose to follow, and a request to the wrong link can change something on your site: log a session out, unsubscribe an address, add to a cart, use up a confirmation token. So it is safe by construction.

It skips links that look like actions. Each path segment, query key and query value is split into words on anything but letters and digits, and at camelCase humps. So /users/sign_out, ?route=account/logout, ?task=user.logout, /logout;jsessionid=…, /account%2Flogout and ?action=deleteItem all count. How far a word reaches depends on how clearly it means an action, counted in words with a server-script extension (.aspx, .php, .jsp, .do) left out (/MyAccountLogOut.aspx is four words):

• logout, logoff, signout, signoff and unsubscribe count anywhere, even run together with other words (/UserLogout.aspx, /dologout), and so do unsub and optout.
• The pairs log out, log off, sign out, sign off, opt out and add to (/add-to-cart) count in a path segment of up to four words, and in a query. A version number stuck to the second word (/LogOut2, /SignOut1.aspx) does not stop a pair from counting; oauth2 stays one word, so /oauth2/docs is still a page. Log out, log off and sign out also count in a segment of five words (/my-account-log-out-now).
• delete, destroy, trash, revoke, deactivate, confirm, verify, activate, approve, unlock, purchase and callback count in a path segment of up to three words (/delete-your-account), and in a query.
• remove, cancel, disable, reset, checkout, cart, basket, buy, vote, like, follow, subscribe, oauth, invite, accept and decline count in a path segment of up to two words (/cart, /remove-item, /password/reset), and in a query.

A longer slug that only mentions one of them is still a page and is checked: /where-to-buy, /how-to-remove-stains, /blog/how-to-verify-your-domain, /blog/how-to-log-out-of-every-device, /blog/a-perfect-sign-off-email.

It skips links that carry a secret or a one-time value.

• A query key that has one of these words in it, or a path segment of up to two words that does: token, nonce, wpnonce, csrf, xsrf, sig, signature, otp, apikey, secret, password or passwd.
• A query key that is exactly code, key, hash, auth, session, sid, ticket or state.
• A query key that names a credential, matched in any case, ignoring -, _ and ., and inside brackets (x_api_key, config[api_key], pwd[], pwd[0]): access_token, access-token, accesstoken, token, id_token, refresh_token, auth, auth_token, auth-token, authorization, jwt, sig, signature, x-amz-signature, x-amz-credential, x-amz-security-token, x-goog-signature, x-goog-credential, api_key, api-key, x-api-key, apikey, password, passwd, pwd, secret, client_secret, sessionid, session_id, access_key, auth_key or private_key.
• A session id anywhere (jsessionid, phpsessid, sessionid).
• A credential in a matrix parameter, the ;key=value pieces of a path segment (/file;sig=…, /a.zip;X-Amz-Credential=…, /x;token=…): a key from the lists above, or a value that looks like a token when the URL also says one of the path-token words below. Ordinary matrix parameters (/shoes;color=red;size=10) are pages.
• A query key or value that looks like a token: a long hex string, a long run of letters and digits with no separator, a long mixed-case base64 string, or a JWT.
• A user name or password in the URL itself (https://user:{password}@example.com/).
• A fragment written as key=value pairs with a secret in it: a key from the lists above or a value that looks like a token, such as an OAuth sign-in redirect’s #access_token=…&token_type=bearer. The key lists are the query’s, so #state=… or #code=… is skipped as ?state=… is. A fragment is never sent with a request, but the URL still carries the credential. A plain anchor (#pricing), a hash route (#/settings) and other state (#q=shoes, #page=3) are pages.
• A token in the path: a JWT always, and any other token-like segment when the URL also says login, signin, magic, token, auth, session, password, email, invite, reset, verify, confirm, activate, unsubscribe or unsub (/magic-login/…, /login/token/…). A plain id such as /posts/<uuid> is still checked.

It skips links whose text is an action. Short text that is clearly one (“Log out”, “Unsubscribe”, “Delete account”) is skipped whatever its URL says. Any action text (“Remove”, “Buy now”) is skipped on a rel="nofollow" link.

It skips Cloudflare email-protection links. A link to /cdn-cgi/l/email-protection is an email address Cloudflare’s email obfuscation rewrote, not a page, so it is never requested. See links/broken-links.

It sends no credentials: no cookies, no Authorization, and none of the audit’s custom request headers ([crawler] headers, -H), which can carry auth. It identifies itself with the crawl’s User-Agent.

It uses HEAD only. A GET asking for one byte (the body is discarded) is sent only when the server answers the HEAD with 405 or 501, which say it does not support HEAD. Some servers answer HEAD with an error on pages they serve. So the check first sends HEAD to the audited URL, which the crawl already fetched. If that answer is an error, the targets’ errors are not trusted. They are never reported as broken, and in an audit that did not follow links they are listed as Not checked.

It re-checks every redirect hop. It does not follow a redirect to another host, out of the crawl scope, to a path robots.txt disallows, or to an action link; that redirect is the target’s status. It follows at most 5 hops.

It stays polite. It stays inside the crawl scope and, when the crawl respects it, robots.txt. It never has more of its own requests in flight than the crawl’s per-host concurrency, and checks at most max_targets targets an audit, each with one HEAD and the redirect hops that follow it.

A skipped target is counted in the Not checked line of links/broken-links when the audit did not follow links, and is never reported as passing.

These rules are a careful default, not a guarantee. The word lists are English, and no list can know every route a site wires to a GET. If your site changes state on GET or HEAD requests to links that read like pages, switch the check off for that site with --no-internal-links or [internal_links] enabled = false, or, in the cloud, with the website’s Internal link checks crawl setting.

How external link checking works

Request strategy

1. HEAD request first

   • Faster (no body download)
   • Checks if URL responds
   • Most efficient

2. GET fallback

   • If HEAD fails/not supported
   • Downloads full response
   • Slower but more reliable

3. User agent

   • Uses browser-like headers and user agent
   • Detects WAF/bot protection (Cloudflare, Akamai, etc.)
   • Sign-in and bot walls (401, 403, 405, 406, 999), challenge pages and timeouts listed as unverifiable, apart from broken links

What the external link check never requests

An external link can carry its own credential: a newsletter’s one-click unsubscribe link, a magic sign-in link, a signed download URL. Requesting it, even with HEAD, can use it up or act on it. So the external link check never requests a link that the internal link check would skip as an action or token link, using the same word lists and token shapes as What the internal link check never requests, and it skips a link whose text is plainly an action (“Log out”, “Unsubscribe”, “Delete account”). rel="nofollow" does not widen that here: on an external link it marks a paid or untrusted link such as “Buy on Amazon”, not an action. A signed-in audit does not send these links to the cloud’s dead-link check either. Nor does it check a Cloudflare email-protection link (/cdn-cgi/l/email-protection) on another site: it is an email address, not a page.

When squirrel checks a link itself, it follows redirects by hand, at most 20. A redirect into an action or token link, or to anything but http or https, is not followed, and that redirect is the link’s status. Every request it sends, the first and each redirect hop, goes only to a public address (or a private one when the audited site is itself on a private network), never to a link-local or cloud metadata address, and connects to the address it checked, so a DNS answer cannot change between the check and the request. A link whose own address is refused is not requested, never reported as working or broken, and listed in links/broken-external-links as refused-external-links. One refused on the URL alone (a private or metadata IP literal, a metadata hostname, localhost on a public site’s audit) also takes no slot under max_links, and is never answered from the link cache or sent to the cloud’s dead-link check. A redirect toward a refused address is not followed. Whatever site it audits, a signed-in audit never sends the cloud’s dead-link check a link whose URL names a non-public host (a private or loopback address, localhost, a .local, .test or single-label name, or another name public DNS does not resolve). An audit of a site on a private network checks those itself, and a public site’s audit refuses them. The cloud check’s refusal of a link, because the link or a redirect it leads to is on a non-public address, is never reported as a broken link. squirrel checks that link itself instead, as it checks a link the cloud left unanswered: an audit of a site on a private network gets its real status, and a public site’s audit refuses it on your machine too. Each request carries only squirrel’s User-Agent and Accept, plus the Accept-Encoding it decodes, never your custom request headers.

These links are listed in links/broken-external-links as not requested, never as working or broken, with their query values, token-like parts and email addresses masked and any fragment left off (a best effort; an asset name such as logo@2x.png is kept), and they do not count against max_links.

These rules are a careful default, not a guarantee: the word lists are English, and no list can recognize every link that acts on a GET. If your pages link to third-party URLs that change state when requested and read like pages, switch external link checks off with --no-external-links or [external_links] enabled = false.

Status detection

This is how squirrel reads a link it checks itself. A storage bucket’s 403 for a missing file, an HTTP sign-in prompt and a 405 that does not allow GET are the site’s own answers and stay broken; links/broken-external-links has the details.

Caching behavior

Cached for TTL period:

• 200-299 (success)
• 404 (not found)
• Redirects (with final destination)

Not cached:

• Timeouts (may be transient)
• Server errors (5xx - may be temporary)
• DNS failures (may recover)

Configuration examples

Fast crawl (disable external Links)

For local development or quick audits:

[external_links]
enabled = false

Skips all external link validation.

Aggressive external link checking

For comprehensive link validation:

[external_links]
enabled = true
cache_ttl_days = 1        # Fresh daily
timeout_ms = 30000        # 30s tolerance
concurrency = 15          # High parallelism

Use cases:

• Link quality audits
• Outbound link monitoring
• Content freshness validation

Polite external link checking

Conservative settings for respectful crawling:

[external_links]
enabled = true
cache_ttl_days = 30       # Cache longer
timeout_ms = 15000        # 15s tolerance
concurrency = 2           # Low parallelism

Use cases:

• Many external links
• Avoid rate limits
• Network restrictions

CI/CD pipeline

Fast feedback with fresh data:

[external_links]
enabled = true
cache_ttl_days = 1        # Fresh each run
timeout_ms = 5000         # Fail fast
concurrency = 10          # Speed up validation

Use cases:

• Automated testing
• PR checks
• Daily builds

Performance impact

External link checking adds overhead to crawls. Typical impact:

With external links enabled (default):

100 internal pages
50 unique external links
= 100 page fetches + 50 external checks (if not cached)
= ~150 total HTTP requests

With external links disabled:

100 internal pages
= 100 page fetches
= ~100 total HTTP requests

Cache impact:

First crawl:

50 external links × 10s timeout = up to 10s total (5 concurrent)

Second crawl (within cache TTL):

50 external links cached = 0s overhead

External link rules

External link configuration affects these rules:

links/broken-external-links

What it checks:

• External links returning 4xx/5xx errors
• Timeout failures
• DNS resolution failures

Requires:

[external_links]
enabled = true  # Must be enabled

Configuration:

[external_links]
enabled = true
timeout_ms = 10000

[rules]
enable = ["links/broken-external-links"]

links/https-downgrade

What it checks:

• HTTPS page linking to HTTP external URL
• Security downgrade warnings

Requires:

[external_links]
enabled = true  # To validate external URLs

Troubleshooting

External links all timeout

Symptoms:

• Many “timeout” failures
• External link checks slow

Solutions:

Increase timeout:

[external_links]
timeout_ms = 20000  # 20 seconds

Reduce concurrency:

[external_links]
concurrency = 2  # Fewer parallel requests

Check network:

curl -I https://example.com  # Test external access

False positives (working links marked broken)

Cause: Some sites block HEAD requests or use WAF/bot protection

Solution: External link checker automatically:

• Falls back to GET requests when HEAD fails
• Detects WAF/bot protection (Cloudflare, Akamai, etc.)
• Lists walls, challenge pages and timeouts as unverifiable (not broken)

If a site truly blocks bot traffic, the link is listed as unverifiable under waf-blocked-external-links, with the vendor in its label when one was named.

Too slow with many external links

Symptoms:

• Crawl takes long time
• Hundreds of external links

Solutions:

Disable external links:

[external_links]
enabled = false

Or increase concurrency:

[external_links]
concurrency = 15
timeout_ms = 5000

Cache not working

Verify cache location:

ls -la ~/.squirrel/cache/external-links/

Clear and rebuild:

rm -rf ~/.squirrel/cache/external-links/
squirrel audit https://example.com  # Rebuild cache

Complete example

[project]
name = "mysite"

[crawler]
max_pages = 500

[external_links]
# Enable external link validation
enabled = true

# Cache results for 7 days
cache_ttl_days = 7

# 15 second timeout for slow sites
timeout_ms = 15000

# Check 10 links concurrently
concurrency = 10

[rules]
# Enable broken external link detection
enable = ["*"]
disable = ["ai/*"]

Running this config:

• Crawls up to 500 pages
• Validates all external links
• Caches results for 7 days
• Allows 15s per external link
• Checks 10 external links in parallel
• Reports broken external links in audit

Related

• Crawler Settings - Request method and user agent
• Rules Configuration - Enable link rules
• Examples - Common configurations