Serving Markdown to AI agents: .md exports, llms.txt and the caching traps in between

When an AI assistant answers a question using your website, it usually got there by fetching your HTML and trying to dig the content out of it. On a marketing site, that content is buried under navigation, cookie banners, modals, pop-ups and calls to action. The model sees all of it.

So we built a second, cleaner copy of the important pages — plain Markdown and structured JSON — and told agents where to find it. It sounds like a small feature. Most of the interesting work was in caching.

What we built

Every enabled page gets two siblings at predictable URLs:

/guides/what-is-cdc/   →  /guides/what-is-cdc.md
                          /guides/what-is-cdc.json
/pricing/              →  /pricing.md
/guides/               →  /guides/index.md

That last rule exists because /guides.md would read as a sibling of the section rather than the section itself.

The Markdown starts with front matter — title, description, canonical URL, dates, author, tags, word count — then a single # Title and the body. The JSON carries the same metadata plus headings, sections, FAQ pairs, links and a Schema.org type, for agents that prefer structure.

There's also an llms.txt at the root: a curated, Markdown-formatted index of the site in the llms.txt format, so an agent can see what exists before fetching anything.

This exists on two platforms. A colleague built the first version as a WordPress plugin for the blog; I took it to production behind the CDN, then designed and built the Django version for the marketing site.

Turning pages into clean Markdown

Convert from structure, not from HTML, where you can. Many pages on the Django site are built from typed content blocks in a CMS. For those, the Markdown is generated straight from the block data rather than by rendering HTML and converting it back — so a comparison table stays a table and an FAQ stays question-and-answer pairs. Each block type has its own small converter; promotional blocks like calls to action are skipped entirely.

When you have to convert HTML, strip the furniture first. About 300 pages are hand-built templates. Those are rendered in-process, then navigation, headers, footers, modals, cookie banners and exit-intent pop-ups are removed by tag, class prefix and ID before a small, dependency-free HTML-to-Markdown converter runs.

Exactly one H1. Every heading in the body is demoted one level, so the export's title is the only H1. A second H1 reads as the start of a new document to the chunkers that split these files up.

The page list can't drift. Which template pages are exportable comes from the same source as the sitemap.

Telling agents where to look

A clean copy is useless if nothing finds it. Three signals point to it:

• A link in the page head. Each exported page carries <link rel="alternate" type="text/markdown"> pointing at its .md twin. On WordPress it's only emitted when the file exists, so the advertised URL always returns 200.
• A Link response header on HTML pages, pointing agents at the API docs, the product docs and /llms.txt. It's only added to successful HTML responses (Django gives redirects an HTML content type, so without that check every 301 would carry it), and it appends to any existing Link header instead of overwriting the canonical one.
• llms.txt itself, as the map of the site.

Design decisions that mattered

Exports live in the database, not on disk

The Django site runs as several containers behind a load balancer. A file written to disk while serving one request is invisible to every other replica, and disappears on the next deploy. So each export is a database row. That also makes hand-editing an export a plain update, and lets a SHA-256 hash skip writes when nothing actually changed — useful because a CMS save fires on every metadata tweak, not just content edits.

Opt-in, per page

The first version exported every candidate page — around 380 of them. That meant rendering every hand-built view to produce files nothing would ever fetch, and handing crawlers the whole site whether or not anyone intended it. Exports are now enabled page by page. Turning one on generates it immediately (and rolls back if the result is empty or errored); turning it off means a 404.

Critically, a missing export is never generated on request. Otherwise any crawler could make the server render exports for every page just by guessing URLs.

Never publish a truncated llms.txt

The Django llms.txt is a curated preamble of thousands of entries, followed by generated sections. If the curated part ever failed to load, the naive build would happily publish a drastically smaller file — on a URL AI crawlers poll — with nothing looking broken. Now the build refuses: if the preamble can't be read, it logs an error and keeps serving the last good file.

Rebuilding it costs a full scan plus a couple of hundred kilobytes of string building, so a save only marks it stale; the next request rebuilds it.

The caching traps

This is where most of the real bugs were.

Content negotiation can poison your CDN

The elegant version of this feature is content negotiation: same URL, and if the client sends Accept: text/markdown, return Markdown instead of HTML. The WordPress side supports it. It's also off by default, for a good reason.

A CDN caches by URL plus whatever is in its cache key. CloudFront, by default, doesn't put Accept in the key. So the first request for a blog post with Accept: text/markdown, arriving at a cold edge, would store Markdown under the post's URL — and every browser after it would get Markdown instead of the page. One crafted request is enough.

When negotiation is on, three things make it safe:

• A small function at the CDN edge collapses Accept into a simple 0/1 header, so the cache keys on two variants per URL instead of one per raw Accept string.
• Negotiated responses send Vary: Accept.
• Negotiated Markdown is non-storable (no-cache, must-revalidate). It still carries an ETag, so revalidation stays cheap — but a shared cache can never pin it onto the HTML URL.

The explicit .md URLs don't have this problem — they have their own cache key — so they're cached normally for an hour.

Conditional requests that never returned 304

Agents and the CDN revalidate with If-None-Match. On both platforms, unchanged files were being re-downloaded in full on every poll — for two different reasons.

On Django, the gzip middleware downgrades a strong ETag to its weak form (W/"abc") whenever it compresses a response. The client sends the weak form back, and a plain string comparison never matches. The fix is to compare the way the spec says to — handle lists, *, and strip the weak prefix:

def _etag_matches(header, etag):
    """RFC 7232 If-None-Match comparison (gzip weakens strong ETags)."""
    if not header:
        return False
    for candidate in header.split(","):
        candidate = candidate.strip()
        if candidate == "*":
            return True
        if candidate[:2].lower() == "w/":
            candidate = candidate[2:]
        if candidate == etag:
            return True
    return False

The test client never sends Accept-Encoding: gzip, which is why this only showed up on a running server.

On WordPress, the framework backslash-escapes server variables on every request, so the client's quoted ETag arrived as \"...\" and never matched. One wp_unslash() fixed it.

The redirect agents won't follow

WordPress's canonical-URL logic 301-redirected /llms.txt to /llms.txt/ and every .md URL to a trailing-slash version. Plenty of agent fetchers don't follow redirects, so to them llms.txt simply didn't exist — and every .md fetch paid an extra, uncacheable hop. The fix was to exempt the plugin's own routes from canonical redirects.

Testing the part humans can't see

The Django export service has 101 tests — the most of anything on that site. That's deliberate: bugs here are invisible to people browsing the site and very visible to machines. They cover the HTML converter (tables, code, unclosed tags), block conversion, front matter that looks like YAML but isn't, hash-based skipping, opt-in and opt-out, manual edits surviving regeneration, 304s including weak ETags and wildcards, real routes not being shadowed by the catch-all, and the production-safety rules like the truncated-llms.txt guard.

What I'd tell someone building this

• Treat agents as a client with its own constraints. They may not follow redirects, may not send cookies, and will poll. Design the URLs and caching for that.
• Never let a shared cache store a negotiated response unless the negotiation input is in the cache key. When in doubt, use separate URLs.
• Test HTTP behaviour on a real server. Two of the worst bugs here were invisible to the test client.
• Make exports opt-in and never generate on demand from unknown URLs. A feature for crawlers shouldn't hand crawlers a way to make you do work.