Skip to main content
Detects conflicts between robots meta tags and robots.txt
Rule IDcrawl/robots-meta-conflict
CategoryCrawlability
ScopeSite-wide
Severitywarning
Weight5/10

​
Solution

Robots.txt and robots meta tags should work together, not conflict. If robots.txt blocks a URL, search engines won’t see the meta robots tag at all. Common conflicts: blocking a page in robots.txt while trying to noindex it (unnecessary), or allowing in robots.txt but noindexing (works but confusing). For noindex, let the page be crawled so the directive is seen. For blocked pages, robots.txt alone is sufficient.

​
Enable / Disable

​
Disable this rule

squirrel.toml
[rules]
disable = ["crawl/robots-meta-conflict"]

​
Disable all Crawlability rules

squirrel.toml
[rules]
disable = ["crawl/*"]

​
Enable only this rule

squirrel.toml
[rules]
enable = ["crawl/robots-meta-conflict"]
disable = ["*"]