Accept: text/markdown: serving Markdown to AI agents

What a Markdown response is

Content negotiation lets one URL serve more than one representation. A client sends Accept: text/markdown, the server returns Markdown instead of HTML, and a Vary: Accept header tells caches that the response depends on the request header. text/markdown has been a registered media type since RFC 7763.

Two simpler routes exist. A .md twin of the page, such as /index.md, is a second URL with the same content. A Link: <...>; rel="alternate"; type="text/markdown" header on the HTML page advertises where that twin lives, which is how a site whose homepage is an application shell can still point an agent at readable content.

What squirrel checks

The rule runs once per crawl against the homepage and emits one or two checks:

• markdown-response: reports at info status with the message Markdown response not checked: the discovery probe was not sent or got no answer and the value not-checked when the probe did not run, or a request got no answer and the other found no Markdown. When the site refused the request (a 401, 403 or 429, or a bot challenge) the message is Markdown response not checked: the site refused the request (HTTP 403 (Cloudflare)) and the value is refused. The rule is skipped when the crawl fetched no page and a root request was refused or failed. Otherwise it reports at info status with Serves Markdown for agents via <signals>, joining whichever it found with and. The three are content negotiation (<content type>), .md variant (<url>) and link rel=alternate (<url>). With none of them it warns with No Markdown response — consider honoring Accept: text/markdown or publishing a .md variant so agents get clean content.
• markdown-response-headers: emitted only when a supplementary signal is present, at info status, with the message Additional Markdown-negotiation signals: followed by the signals joined with ; . Two are recognised. A Vary header on the negotiated response that mentions Accept is reported as Vary: <header value>. Cloudflare’s markdown-transform fingerprint needs both token headers and is reported as Cloudflare markdown-transform fingerprint (x-markdown-tokens: <n>, x-original-tokens: <n>), gaining a , ~N% of original suffix only when both parse as finite numbers and the original count is above zero.

The rule’s severity is info, so the warning surfaces as a recommendation and does not move the score. The link rel=alternate signal is counted in the first check and deliberately not repeated in the second.

A probe goes unsent when discovery probes are turned off, when no rule that reads it was enabled for the crawl, or when the crawl’s preamble time budget ran out first. A probe that went out and got no answer, a refused connection or a timeout, counts the same. None of those says anything about the site, so the result is not-checked, never absent.

How to fix it

Honour the header and advertise the twin:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
Link: <https://example.com/index.md>; rel="alternate"; type="text/markdown"

Serving the .md twin alone is enough to satisfy the check, and it is the least invasive of the three. Set Vary: Accept whenever the same URL returns different bodies for different Accept headers, or a cache will hand the HTML to an agent that asked for Markdown.

There is no common false positive. Absence is a recommendation and never a defect.

Enable / disable

Disable this rule

[rules]
disable = ["ax/markdown-response"]

Disable all Agent Experience rules

[rules]
disable = ["ax/*"]

Enable only this rule

[rules]
enable = ["ax/markdown-response"]

enable replaces the default ["*"], so this is the only rule selected. Leave disable out: a rule that matches any disable pattern is skipped even when enable names it. Only an enabled setting under a rule’s own [rule_options] table overrides both lists.

Related rules

• ax/token-weight: the markup cost a Markdown response removes.
• ax/llms-txt: the curated index that points an agent at these pages.
• ax/content-without-js: whether the HTML carries the content in the first place.
• content/mime-type: the content type the server declares for its pages.

Agent experience findings ship in every audit next to the SEO, performance and security rules. See llms.txt for how an agent works through a report.

References

• text/markdown media type, RFC 7763
• Content negotiation, MDN
• Vary, MDN

Check your site

Run squirrel audit https://example.com and open the Agent Experience section of the report. Each Markdown signal is listed with the URL or content type it came from. Signed-out local audits need no account and cost nothing. Signed-in audits spend credits.