Making My Blog Easier for AI Agents to Read

I made this blog return its content as Markdown when an AI agent comes to read it. I also added Link headers and Content Signals. When I measured the blog with scanners from Cloudflare and Vercel, the scores went from 20 to 40 and from 75 to 97. This post records what I did and what I thought after trying it.

Why I started

It started when I read a post titled "Make your website agent-ready (opens in a new tab)." It is a record of making a site Agent Ready: what the author did to make the site easier for AI agents to read, and how the scanner scores changed.

Agent Ready means a state where AI agents can easily read and use a site. According to the post, tools like Claude Code and Cursor tell the server "I'd like Markdown if you have it" when they fetch a page (they send the header Accept: text/markdown). If the site responds with Markdown, the agent gets only the content, instead of HTML that includes the header and the sidebar.

After reading the post, I wanted to make my blog Agent Ready too. I thought it might work as an AIO (AI search optimization) measure, and that it might bring more visitors to this blog. AIO is the work of preparing a site so that AI-generated search results are more likely to cite and rate it (according to a Japanese marketing column (opens in a new tab)). This blog runs on Cloudflare Workers, just like the site in the post, so it looked like I could do the same things.

I checked how far the post went, and decided to try the same scope on my own blog.

Measuring first

The post measured the scores with two scanners.

The two look at different things and give different results. The post also says it is good to use both when checking, so I measured with both. Since I planned to write about it later, I decided to keep the results from before the change, along with screenshots.

Before the change, isitagentready.com gave 20 points (Level 1: Basic Web Presence), and is-agentic.com gave 75 points (Ready with a few material gaps).

The blog already passed the robots.txt and sitemap checks. It failed because it didn't return Markdown, had no Link headers, and had no Content Signals in robots.txt. When I actually fetched the top page with Accept: text/markdown, I got plain HTML back.

What I did

Returning Markdown

After each build, the blog now takes just the content part from the HTML of each page, converts it to Markdown, and puts it next to the page as index.md. The top of the file has the title, URL, publish date, and tags. These files are made on every build, so I don't have to write them by hand when I add posts.

Next, the Worker that receives requests now looks at the Accept header, and returns this index.md if the request wants Markdown. Browsers don't ask for Markdown, so when people open the blog in a browser, they get HTML as before. The URL is the same as the HTML version.

Until now, page requests on this blog didn't go through the Worker, and the static files were returned as they were. Only the APIs, such as the access counter, went through the Worker. To look at Accept, pages also need to go through the Worker, so I changed the settings. JavaScript, CSS, and images are left out.

Link headers and Vary

When returning HTML, the blog now tells where the Markdown version, the RSS feed, and the sitemap are, using Link headers. It works like a signpost for agents.

I also added Vary: Accept. This header tells caches along the way that the content of the same URL changes depending on Accept. The post says that the framework's cache handling had overwritten this header, so the author added it back.

Pages that don't exist

When a request asks for Markdown for a page that doesn't exist, the blog keeps the 404 status and returns Markdown that says the page doesn't exist, with links to the top page and the sitemap. is-agentic.com checks this, and the post handled it too.

Content Signals

I added this line to robots.txt.

Content-Signal: search=yes, ai-input=yes, ai-train=no

It states that showing the content in search results and quoting it in AI answers are allowed, but using it to train AI is not. It is only a statement, so whether it is followed is up to whoever reads it. The post added this automatically with a Cloudflare feature called Bot Preference Sync, but I wrote it directly in robots.txt.

I made this blog to share what I think, what I feel, and what I do. I didn't want that content to be used for AI training, so I opted out of training only.

Structured data

Following a suggestion from is-agentic.com, I added a short bio to the author information in the structured data (JSON-LD). It is the same text as the profile in the sidebar.

Measuring again

After deploying, I measured again with the same two scanners.

On isitagentready.com, the blog now passes three checks: Markdown responses, Link headers, and Content Signals. On is-agentic.com, it passed all six essential checks.

What remains on isitagentready.com is DNS-AID and eight items about APIs, authentication, MCP, and similar things. DNS-AID needs DNS settings, and this blog doesn't have its own domain, so I can't support it. The eight API-related items don't apply to a blog with no API or login, so I left them out, as the post did. The only thing left on is-agentic.com is that there is no contact page (/contact).

What I thought after trying it

Nothing changed in how the blog looks in a browser. The only change is what it returns when an agent comes to read the same URL.

The scores were quite different depending on the scanner. With the same changes, isitagentready.com gives 40 points, and is-agentic.com gives 97. The two scanners check different things, so I thought that getting high scores on both would make a site truly Agent Ready.