Sitemap validation: XML errors and the 50,000 entry limit

What sitemap validity means

A valid sitemap is well-formed XML in UTF-8, using the sitemap protocol namespace, with each URL in a <loc> element. A sitemap index uses a different root element and lists child sitemap files instead of URLs.

The protocol caps a single file at 50,000 entries and 50 MB uncompressed. Those two limits apply separately to the two file types: a URL set is capped at 50,000 URLs, and an index is capped at 50,000 child sitemaps. Sites past either limit shard into several files behind an index.

What squirrel checks

The rule is site-wide. It emits these checks:

• sitemap-fetch-failed fails when a sitemap named in robots.txt could not be fetched, excluding ones the walk never requested. Message: 1 sitemap(s) from robots.txt failed to fetch, listing each URL with its error.
• sitemap-valid reports info when discovery was truncated with nothing found. Message: Sitemap discovery did not complete, so no sitemap was validated.
• sitemap-valid is skipped when no sitemap was found. Message: No sitemap to validate.

Both of those branches only run when there was no reportable robots.txt fetch failure. When a sitemap named in robots.txt failed to fetch and discovery found nothing, the fetch failure is the only check returned.

• sitemap-syntax fails when any discovered sitemap recorded parse errors. Message: N error(s) in M sitemap(s), listing each sitemap with its first error. It passes otherwise with Sitemap XML syntax is valid. The error is one of:
  • Unknown sitemap format: the XML has neither a <urlset> nor a <sitemapindex> root. A root written with a prefix (<ns0:urlset>, as Python’s ElementTree writes one) counts when that prefix is bound to the sitemap namespace; a prefix bound to any other namespace does not.
  • Parse error: <reason>: the XML did not parse.
  • XML is not well formed: <reason> (line <L>, column <C>): the file parses, but a strict XML reader would reject it: a closing tag that does not match the element it closes (</lox> for </loc>), an element left open, or a raw & in a <loc> or other element text that is not an escape such as &. Only the first such problem is named, with its position in the file as served. The URLs squirrel can still read are kept. Whitespace before the <?xml declaration is not reported, and a file that declares its own entities in a <!DOCTYPE ... [ ]> is not checked.
  • N <url> entries skipped: <loc> is not a single text value (1 <url> entry skipped: ... for one; <url> and <loc> are the literal element names): an entry’s <loc> holds markup or a raw <, or the entry has more than one <loc>. The other entries are still read, and the text around the markup is never used as the URL. A raw < reads as an unclosed tag, so the entries after it in the file can be lost too. In a sitemap index the error names <sitemap> entries instead.
  • N <lastmod> values ignored: not a single text value (1 <lastmod> value ignored: ... for one), or the same for <changefreq> or <priority>: the field holds markup or a raw <, or is repeated within one entry. The field is dropped and its URL is still read.
  • Not a sitemap: the URL returned an HTML page: a sitemap named in robots.txt or in a sitemap index answered with a web page.
  • Corrupt gzip body: <reason>: a gzip file, or a response sent with Content-Encoding: gzip, does not decompress. A response sent as br or deflate reads Corrupt br body: <reason> or Corrupt deflate body: <reason>.
  • Unsupported Content-Encoding: <coding>: the response used a coding squirrel does not decode, such as zstd, or more than one coding.
  • Sitemap exceeds the 50 MB uncompressed size limit: the response, or the body at any stage of decompressing it, is over 52,428,800 bytes. squirrel stops at the limit, so a gzip bomb cannot exhaust memory.
• sitemap-size warns when a URL set holds more than 50,000 URLs, or an index holds more than 50,000 child sitemaps. Message: 1 sitemap(s) exceed the 50,000-entry protocol limit. It passes otherwise with All sitemaps within URL limits.
• sitemap-lastmod-format warns when a <lastmod> is not in the W3C Datetime format the protocol requires (2024, 2024-09, 2024-09-10, or a datetime with a T and a zone such as 2024-09-10T12:00:00Z or 2024-09-10T12:00:00+02:00). Message: N sitemap lastmod value(s) are not in W3C Datetime format, listing each URL with lastmod "<value>", up to 50, with the exact count in total. Values such as 2024/09/10, 2024-9, 2024-09-31 and Tue, 10 Sep 2024 12:00:00 GMT count. It is left out when every lastmod is in the W3C format, or when no URL has one.
• sitemap-lastmod-zone warns for a datetime with the zone left off (2024-09-10T12:00:00). It reads as a date, just not one in a known zone. Message: N sitemap lastmod datetime(s) have no time zone, listing URLs and values the same way.
• sitemap-orphans reports info for sitemap URLs the crawl did not reach. Message: N URL(s) in sitemap not found during crawl, with N exact even past the 10,000-URL list the audit engine hands the rule. It lists the first 50 URLs, and past 50, additional and pagesTruncated record the rest.
• sitemap-missing reports info for crawled pages the sitemap does not list. Message: 7 crawled page(s) not in sitemap.

The URL count stored on a top-level index is the aggregated total across its children, which is why an index is measured against its child count rather than that total. Severity error, weight 8.

Field values are read as the text the sitemap wrote, so a bare-year <lastmod>2024</lastmod>, which the W3C Datetime format allows, is kept as 2024. A field that is not one text value is dropped and reported as above.

Gzip-compressed sitemap files, such as a sitemap.xml.gz served as application/gzip, are decompressed before parsing. squirrel asks for gzip-compressed responses and decodes them itself, within the same 50 MB limit. A common location that squirrel only guessed, such as /sitemap.xml when neither robots.txt nor a sitemap index names it, counts as no sitemap when it answers with an HTML page or with a body that is not XML. A response labelled text/html that does not parse counts as a page unless it opens with a <urlset> or <sitemapindex> root. A server that answers every path with a page is not reported for sitemaps it never published.

How to fix it

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-posts-1.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-posts-2.xml</loc></sitemap>
</sitemapindex>

Shard past 50,000 entries behind an index. For parse errors, well-formedness errors and skipped entries, look for unescaped ampersands and raw angle brackets in URLs, which is what hand-built or string-concatenated sitemaps usually get wrong. Escape them as & and <. Write <lastmod> as YYYY-MM-DD, or as a full datetime with its zone (2024-09-10T12:00:00+00:00), which is what a date library’s ISO 8601 output gives once a zone is attached.

Enable / disable

Disable this rule

[rules]
disable = ["crawl/sitemap-valid"]

Disable all Crawlability rules

[rules]
disable = ["crawl/*"]

Enable only this rule

[rules]
enable = ["crawl/sitemap-valid"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• crawl/sitemap-exists: whether a sitemap was found at all.
• crawl/sitemap-domain: sitemap URLs pointing at a host that is not yours.
• crawl/sitemap-coverage: the gap between what you crawl and what the sitemap lists.
• crawl/sitemap-lastmod-churn: whether the lastmod values in it mean anything.

Crawlability findings ship in every audit next to the SEO, performance and agent experience rules. See The Screaming Frog alternative for agents for how an agent works through a report.

References

• Build and submit a sitemap, Google Search Central
• Sitemaps overview, Google Search Central

Check your site

Run squirrel audit https://example.com and open the Crawlability section of the report. Each sitemap is listed with its parse errors and entry count. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.