How to make your site AI-agent readable: the checklist we used to get from 29 to an A.

We built an agent-readiness scanner, then pointed it at our own site. It scored 29/100 — an F. Here is exactly what we changed to reach an A.

It's a humbling thing to build a scanner that grades how readable a site is to AI agents, run it on your own domain, and get back 29/100 — an F. So we fixed it, in public, and got to 83/100 (an A). None of it was hard; most of it took an afternoon. Here's the checklist, in priority order, with the actual changes.

1. Declare AI crawlers in robots.txt

Tell the agents you know they exist, and declare how your content may be used. Add explicit rules plus a Content-Signal directive:

User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /

Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap: https://you.com/sitemap.xml

2. Publish an llms.txt

The single highest-leverage file. It's a plain markdown index of what matters, pointing at markdown URLs (not HTML) so agents spend fewer tokens:

# Your Company
> One-line description.

## Products
- [CT8000 3D Hall sensor](https://you.com/ct8000.md): ±40 mT, I²C / SPI

## Docs
- [Selection guide](https://you.com/guide.md)

For large catalogs, add an llms-full.txt that concatenates everything into one fetch.

3. Serve markdown on content negotiation (the big one)

This was our largest single jump. When an agent sends Accept: text/markdown, return markdown instead of a 65,000-token HTML page. In a Cloudflare Worker it's a few lines:

if ((req.headers.get('accept') || '').includes('text/markdown')) {
  return new Response(markdown, {
    headers: { 'Content-Type': 'text/markdown; charset=utf-8' },
  })
}

Our homepage went from ~65,000 tokens to ~1,000 — a 98% reduction in what an agent has to read.

4. Add JSON-LD structured data

Give agents entities they can parse deterministically instead of guessing from prose. Minimum: Organization + WebSite; on product pages, Product. It's a single <script type="application/ld+json"> block.

5. Sitemap + a Link header

Publish /sitemap.xml and add an RFC 8288 Link response header that points agents at your structured resources (service-desc, api-catalog) without parsing HTML.

6. Expose capabilities: OpenAPI, API catalog, agent skills

If you have any API — even a single endpoint — describe it. We published a real /openapi.json for our scanner, a /.well-known/api-catalog (RFC 9727) pointing to it, and a /.well-known/agent-skills/index.json. The next step is a full MCP server so agents can call your tools directly — that's the one box we haven't ticked yet, and it's where the real product value lives.

7. OpenGraph metadata

The easy one: og:title, og:description, og:image. Agents and link previews both resolve them.

The payoff

Those seven changes took us from 29 to 83 — Discoverability, Content, and Access Control all to 100. The remaining gap is Capabilities: a real MCP server and, for a product catalog, structured specs an agent can actually query. The score is the easy part; being correctly citable is the work.

Want your own number first? Run the free scan → or see how your peers stack up on the leaderboard.