squirrelscan
GitHub

Crawlability

Robots.txt, sitemaps, and crawl directives

Robots.txt, sitemaps, and crawl directives

Rules

Disable All Crawlability Rules

[rules]
disable = ["crawl/*"]

Type to search…

↑↓ navigate↵ openesc close