A new set of job titles is appearing in documentation teams: knowledge engineer, AI content lead, AI readiness lead. The names vary, the problem does not. A company switches on an AI assistant, connects its knowledge base, or lets coding agents loose on its developer docs, and discovers something uncomfortable: the information is all there, and the AI still cannot use it reliably.
The documentation was written for people who browse, scroll, click and fill in the missing context themselves. An AI does none of that. Somewhere between your page and the answer, the information loses its meaning.
That is what AI readiness is about. Not rewriting every sentence for a machine, but making sure the knowledge stays accurate, understandable and complete wherever it ends up. This post is what we have learned from reading customer sites, building assistants on top of them, and doing the same work on our own documentation.
What an AI actually sees
There is no single way an AI reads documentation, and each way loses something different.
A chat assistant on your site does not read your pages at question time. It reads a handful of short passages that were cut out of your pages earlier, chosen because they looked relevant. The sidebar, the breadcrumb, the tab the passage sat in and the paragraph above it are gone. A coding agent fetches a page live while a developer works, and often gets only the beginning of it. A browsing agent can open pages and follow links, and still never sees what is hidden behind a click.
Across all three, the same few failures come up on almost every site we read:
- The page is a shell. Many documentation platforms send a short loading page and draw the content with a script. Anything that does not behave like a full browser gets the shell. We wrote about this in April; it is still the most common reason an assistant says “not in the documentation” about a page you can see.
- The page is mostly frame. A short answer sits inside a header, a sidebar of hundreds of links, a cookie notice, a feedback widget and a banner about the next release. Cut that page into passages and most of them are navigation. We have watched an assistant quote a release banner as the answer to a product question, because the banner sat inside the article on every page.
- The answer depends on something outside the passage. A section begins with “To enable this, select the option above.” Once the section is read on its own, neither “this” nor “above” means anything.
- The right answer has twenty competing versions. Documentation published for every release creates near-identical passages. Without a clear version on the page, the one chosen is often for a version the visitor does not have.
We saw all of this at once in July on a public help centre of about 960 pages that we read for a trial. Its category pages, the ones that list every question in a topic, returned a successful status and 47 bytes of content to anything that was not a full browser. Each individual answer page held one paragraph of answer, about 190 words, wrapped in 65 question titles from the navigation. Nothing at the root told a machine where to start. A visitor with a mouse would never notice. The AI got a menu with a sentence attached.
No amount of clever instructions to the AI can recover information that never reached it. That is why most of the rules below are about getting the content through intact, and only some are about the writing.
Ten rules for content that survives the trip
Each of these comes from something that went wrong on a real site this year.
- Make the content reachable before you touch a word. Fetch a page without a browser and look for the article. If you get a loading message, nothing else on this list matters yet. Check logins, bot protection and robots rules the same way: a perfectly written page behind a challenge screen does not exist as far as a machine is concerned. Private documentation belongs behind a proper connection, not on the public web and not blocked.
- One clear purpose per page, not necessarily one question. A page that mixes installation, billing, configuration and release history is hard to pick the right passage from. The opposite extreme fails too: splitting every paragraph into its own page forces the reader to rebuild the topic from a dozen fragments. One coherent task or topic per page, with the answer in the first paragraph.
- Headings that say what the section answers. When a passage is pulled out of its page, the heading is the only context it carries. Three sections all headed “Step 1” for three different setups become indistinguishable. “Step 1: Create the storage bucket” survives.
- Every section stands on its own. No “as described above”, no “this option”, no “the table below” unless the table is in the same section. If a passage needs the paragraph before it to make sense, write the missing sentence in.
- Tables with header rows, and content that is actually on the page. A table without a header row turns into a list of unlabelled numbers once it is cut out. We spent weeks on catalogue tables that lost their meaning this way. Content behind a tab or a “show more” can survive if it is in the page from the start; content that only loads when someone clicks does not. Keep the important table in the first tab, or out of tabs altogether.
- Dates on anything that goes stale. Forecasts, prices, staff, release notes, version-specific instructions. We watched an assistant report a forecast as the final figure and name a contact who had left the role years earlier, because nothing on those pages said when they were true. “Recently” and “the current version” mean nothing to a reader that does not know what day it is. Write the date and the version on the page.
- One version per section. If you publish docs for several versions, make it obvious which one a page is about and keep the old ones out of the way. On one customer site, collapsing the versions raised the share of questions that found the right passage from three in four to nineteen in twenty.
- Say what you do not have. When the documentation is silent, cheaper models fill the silence with something plausible. In our tests, asked how to enable a feature the product does not have, one model recommended extensions by names that appear nowhere in the docs. A page that says “there is no such feature, here is what to use instead” ends that. The same goes for scope: state what a page does not cover and where that lives.
- Keep boilerplate out of the article. Cookie text, release banners, “Was this helpful?”, the sidebar. If it is inside the article, it will be quoted as an answer one day. Semantic structure, a main element around the content and nav and footer elements around the rest, is the cheapest fix there is.
- Tell machines where to start. A sitemap with real modification dates. A llms.txt that lists the pages worth reading, with a line on each. A plain-text twin of every page, served to anyone who asks for text instead of a web page, and a pointer to it at the top, because machines do not guess.
Less is a feature
The instinct, when a company decides to become AI-ready, is to produce more: export everything, generate a summary of every page, write an instruction file that covers every case. Our experience runs the other way.
The assistants that answer well sit on content somebody has pruned: duplicates removed, old versions parked, every page about one thing. The ones that struggle sit on everything the company ever published, in full, with the navigation included. We saw the same effect inside the assistant itself: when we tested giving the smaller models fewer, tighter passages instead of more, their answers got better, not worse. More context is not more knowledge. Past a point it is noise the model has to read past.
Checked by a person
There is a second half to the job, and it is the half that will not be automated soon.
Every assistant we run collects three things: answers a visitor rated down, questions it declined, and answers it gave with low confidence. All three end the same way, with a person opening the page and reading it. We also grade answers by hand, against a rubric written before we read them, because that is where we keep finding the errors that look fine at a glance: a confident number from the wrong year, a feature described one version too early, a correct answer to a slightly different question.
AI-ready content is content a person has read, checked and chosen to keep. That is the knowledge engineer’s job, whatever the title says.
Our own homework
In July we did this for our own documentation. Every page now has a plain-text twin, served to anyone who asks for text instead of a web page, with a link back to the original. The site has a llms.txt and an agent instruction file at its root, and a sitemap with real dates. We ran an open agent-readability scorecard before and after: from failing most checks to 85 out of 100, with the remaining points lost to bugs in the scorecard rather than the site. It took one afternoon. None of it changed a single sentence of the content, which is the point: most of AI-readiness is plumbing, and the plumbing is cheap.
A ten-minute check for your own site
- Ask an AI assistant to read one of your URLs and tell you the first sentence of the article. If you get a loading message, a menu or a cookie notice, that is what every machine gets.
- Fetch a page without a browser. Any command-line tool will do. If the content is not in what comes back, it is drawn by a script, and most readers never see it.
- Read a passage out of context. Pick a section at random, cover the rest of the page, and ask whether the heading and the first sentence tell you what it is about. If you cannot tell, neither can the AI.
- Ask your assistant about something you do not have. A feature that does not exist, a comparison you have never published. The answer tells you whether your content says “no” anywhere.
- Look for a starting point. Does the site have a llms.txt, a sitemap with dates, and a plain-text version of each page? If not, that is the afternoon’s work.
If you do only one thing, do the first. Everything else follows from seeing your site the way the machine does.