Serving markdown to AI agents with content negotiation

I decided to make my site more AI-friendly. I was inspired by the Cloudflare Blog post “Markdown for Agents”. Their solution requires a Pro, Business or Enterprise plan, so I decided to build it myself (for free).

The idea is simple: if a client sends Accept: text/markdown in the request header, serve a .md version of the page instead of HTML. When an agent visits a page, it doesn’t need the CSS, SVGs, or navigation bar—it just needs the content in a format it can parse easily.

My current setup is an Astro static site with:

Blog posts as MDX files
CV as a JSON file

Generating markdown at build time

For blog posts, I created an endpoint that mirrors the existing [post].astro pattern but outputs markdown instead of HTML. You can do this in Astro, and it’s available in other static site generators as well.

The tricky part is that the source files are MDX, which can contain JSX like <Image /> components and import statements. These need to be stripped out to produce clean markdown. I created a function for that:

function cleanMdxBody(body: string): string {
  return (
    body
      // Strip import lines
      .replace(/^import\s+.*$/gm, "")
      // Replace <Image alt="..." ... /> with [Image: alt text]
      .replace(/<Image\s+[^>]*alt="([^"]*)"[^>]*\/>/g, "[Image: $1]")
      // Remove any remaining JSX self-closing tags
      .replace(/<[A-Z][a-zA-Z]*\s[^>]*\/>/g, "")
      .trim()
  );
}

For the CV page, I import the same resume.json file used by the HTML version and convert it to structured markdown with sections for work experience, education, and languages.

After building, I get clean .md files in the dist/ folder:

dist/
├── index.md
├── cv.md
└── posts/
    ├── hosting-my-website-on-a-rpi.md
    ├── serving-content-for-agents.md
    └── ...

Content negotiation in the Rust server

The server is a simple Rust/Axum app that serves static files. I added a middleware that checks the Accept header before the file is served.

async fn content_negotiation_middleware(
    mut request: Request, next: Next
) -> Response {
    let wants_markdown = request
        .headers()
        .get(header::ACCEPT)
        .and_then(|v| v.to_str().ok())
        .map(|v| v.contains("text/md") || v.contains("text/markdown"))
        .unwrap_or(false);

    if wants_markdown {
        let path = request.uri().path();
        let new_path = if path == "/" {
            "/index.md".to_string()
        } else if !path.contains('.') {
            format!("{}.md", path.trim_end_matches('/'))
        } else {
            path.to_string()
        };

        if let Ok(uri) = new_path.parse::<Uri>() {
            *request.uri_mut() = uri;
        }
    }

    let mut response = next.run(request).await;

    if wants_markdown && response.status() == StatusCode::OK {
        response.headers_mut().insert(
            header::CONTENT_TYPE,
            HeaderValue::from_static("text/markdown; charset=utf-8"),
        );
    }

    response
}

The logic is straightforward:

If the Accept header contains text/md or text/markdown, rewrite the request path: / becomes /index.md, /posts/foo becomes /posts/foo.md, etc.
Let ServeDir handle the file serving as usual
If the .md file was found, set the Content-Type to text/markdown

If a path doesn’t have a markdown version, ServeDir returns a 404 and the middleware leaves it alone.

Cloudflare caching gotcha

If you’re behind Cloudflare it caches responses by URL and ignores the Vary: Accept header — so once the HTML version is cached, all requests get HTML regardless of the Accept header.

I tried several approaches:

Vary: Accept header — Cloudflare ignores Vary for anything other than Accept-Encoding
Cache-Control: no-store for markdown responses — doesn’t help because the cached HTML response is served before the request reaches the origin
Cache Rules to bypass cache when Accept contains text/markdown — didn’t work either, the rule never matched

What ultimately worked was a Cloudflare Worker that modifies the cache key based on the Accept header:

export default {
  async fetch(request) {
    const accept = request.headers.get("Accept") || "";
    const isMarkdown = accept.includes("text/markdown")
      || accept.includes("text/md");
    const cacheUrl = new URL(request.url);

    if (isMarkdown) {
      cacheUrl.searchParams.set("format", "md");
    }

    return fetch(request, {
      cf: {
        cacheKey: cacheUrl.toString(),
      },
    });
  },
};

The worker appends ?format=md to the cache key (not to the actual request URL) when the Accept header asks for markdown. This makes Cloudflare store and serve separate cached versions for HTML and markdown requests.

Testing it

You can test this with curl:

# Normal HTML (default behavior, unchanged)
curl https://mtt.engineer/

# Blog post as markdown
curl -H "Accept: text/markdown" https://mtt.engineer/posts/hosting-my-website-on-a-rpi

The .md files are also directly accessible:

curl https://mtt.engineer/cv.md

That’s it — a few lines of build-time code and a middleware layer, and my site now serves clean markdown to any agent that asks for it.