Debug and tracing

SquirrelScan writes debug logs to ~/.squirrel/logs/ for diagnosing issues. By default, only errors are logged. Increase the log level for more detail.

Log levels

Configuration

Settings file

squirrel config set log_level debug

Valid values: error, warn, info, debug

Environment variable

Override the setting for a single run:

SQUIRREL_LOG_LEVEL=debug squirrel audit https://example.com

The environment variable takes precedence over the settings file.

Console debug flag

The --debug flag shows debug messages in the console, independent of file logging:

squirrel audit https://example.com --debug

This is useful for real-time debugging without changing your log level setting.

Log files

Logs are written to ~/.squirrel/logs/:

Log format

2025-01-10T14:30:45.123Z [debug] [request] url="https://example.com" status=200 loadTimeMs=120
2025-01-10T14:30:45.200Z [info] ========== COMMAND END: audit (success, 5432ms) ==========

Log rotation

Logs rotate automatically:

Size-based rotation also triggers at 10MB to prevent runaway growth.

Configure via settings:

squirrel config set log_compress_after_days 7
squirrel config set log_delete_after_days 30

Performance tracing

For performance analysis, enable tracing:

squirrel audit https://example.com --trace

This writes timing data to ~/.squirrel/logs/trace.log:

2025-01-10T14:30:45.123Z [trace] [fetch:https://example.com] duration=145.32ms
2025-01-10T14:30:45.300Z [trace] [rule:meta/title] duration=2.15ms {"checks":1}

Traces include:

• HTTP request timing
• Rule execution duration
• Crawl phase boundaries

Memory on large audits

Everything after the crawl reads the site in batches and drops each batch before reading the next, so a 2,500-page audit holds roughly what a 400-page one does rather than several gigabytes. The batch is sized from the site’s own average page against a byte budget, because a docs site of 30 KB pages and a storefront of 1 MB ones need very different page counts to hold the same bytes.

Two environment variables override that, and they are the same names and the same default the hosted runtime uses:

Both take a plain integer. A value with a unit suffix is refused with a warning rather than partly read, so SQUIRREL_STREAM_BATCH_BYTES=48mb does not quietly become 48 bytes.

Turning the budget down is usually the wrong move. Peak memory is a fixed floor for the rule set and caches plus a few megabytes per page of batch, so the floor is not reachable from this dial, and below roughly a dozen pages per batch the same crawl becomes several times as many read-and-parse cycles whose churn peaks higher than a larger batch does.

To see where memory goes, set SQUIRREL_STREAM_PHASE_MEM=1. Each sub-phase of the rules pass then prints its heap after a forced collection:

[stream] universe end heapUsed=285MB external=51MB rss=915MB
[stream] page-rules end heapUsed=287MB external=56MB rss=798MB

Read heapUsed, not rss. Under memory pressure macOS compresses pages and the resident figure stops tracking what the process actually holds.

Rules that run once per template

Most of a large site is a handful of templates repeated: a real storefront of 247 pages has 13 distinct page chromes. A rule whose answer is a property of the template rather than of the page therefore gives the same verdict hundreds of times, so the audit runs those rules once per template and reports the result on every page of it. The findings are the same either way, and the audit is faster on exactly the sites where the rules phase hurts.

Which rules qualify is a property of the rule, not a setting: a rule reading anything that varies page to page, including any response header and the page’s own URL, keeps running everywhere.

If you suspect a finding that looks copied from a sibling page, turn it off and compare:

SQUIRREL_TEMPLATE_FANOUT=0 squirrel audit https://example.com

0, false, off and no all disable it; anything else, including leaving it unset, leaves it on. A report produced with it off should be identical to one produced with it on, so a difference is a bug worth reporting.

Sensitive data redaction

Logs automatically redact sensitive data:

Headers like Authorization, Cookie, and X-Api-Key are also redacted.

Sharing logs for support

When reporting issues, include relevant log excerpts:

# View recent logs
tail -100 ~/.squirrel/logs/debug.log

# Copy to clipboard (macOS)
tail -100 ~/.squirrel/logs/debug.log | pbcopy

For full context, attach the log file to your GitHub issue.

Examples

Diagnose a failed crawl

# Run with debug logging
SQUIRREL_LOG_LEVEL=debug squirrel audit https://example.com

# Check what happened
cat ~/.squirrel/logs/debug.log | grep -i error

Profile slow audits

# Enable tracing
squirrel audit https://example.com --trace

# Find slow requests
grep "duration=" ~/.squirrel/logs/trace.log | sort -t= -k2 -rn | head

Temporary verbose mode

# One-off debug session
SQUIRREL_LOG_LEVEL=debug squirrel audit https://example.com --debug

Related

• Configuration - All settings
• squirrel config - Manage settings