The most common complaint we hear about documentation assistants is not that they make things up. It is quieter than that: “I asked about a page I can see in my browser, and it told me the documentation does not cover it.”
The page exists. The assistant was pointed at the site. It still answered from nothing. Often the reason is the same, and it has little to do with AI: the assistant never read the page, because the page it downloaded was empty.
Your documentation site is an application
Open any documentation site built in the last few years, press Ctrl+U to view the source, and search for a sentence you can see on the screen. On a surprising number of sites you will not find it.
What you find instead is a short shell: a title, a few style links, and a script. The text, the sidebar and the search box appear only after a browser runs that script, fetches the content, and builds the page. This is how most documentation platforms work today, whether the site is a developer portal, a hosted knowledge base, a wiki or a help centre exported from a content tool. It is a good design for visitors. It is invisible to anything that does not behave like a browser.
A simple crawler does not behave like a browser. It downloads the shell, keeps whatever text is in it, and moves on. Many chatbot products are built on exactly that kind of crawler, because it is fast and cheap. The result is an assistant that has “read” your site and holds a few hundred copies of the words “Loading…” and your cookie notice.
The four ways it goes wrong
Empty pages are the obvious failure. The others are harder to spot, because the assistant still answers.
- Empty pages. The shell contains no article text. Every question about that page gets a polite “not in the documentation”. This is the complaint from the top of the post.
- Half pages. The frame loads first and the content arrives later: a table filled in from a separate request, a list of parameters that appears a moment after the heading, a code block that renders last. A reader that stops at the first paint keeps the introduction and loses the part people actually ask about.
- Menu pages. The page loads fully, including the sidebar with 400 links, the header, the footer, the “Was this helpful?” widget and the banner about the next release. On a short page that boilerplate is most of the text. The assistant then answers questions about your product by quoting your navigation, and cites the sidebar as its source.
- Missing pages. The crawler finds pages by following links. If the links are drawn by the script too, it may find the home page and stop. The assistant knows your landing page and nothing under it, and reports a successful read of one page.
None of these show up as an error. The read “succeeds”, the page count looks plausible, and the first question you try probably lands on a page that happened to work. You find out from a visitor, or from a support ticket that asks a question the docs answer on page three.
A five-minute check for your own site
You do not need us to tell you whether this affects you. Four quick tests, on the site you have now:
- View the source. Open a typical documentation page, press Ctrl+U, and search for a full sentence from the middle of the page. If it is not there, a simple crawler will not see it either.
- Ask about something that loads late. Pick a page where you can see the content fill in after the frame, such as a reference table or a long parameter list, and ask your current assistant about a row near the bottom. If it knows the introduction but not the table, it is reading half pages.
- Ask about a page three levels deep. Choose a page you reach only through the sidebar, never from the home page, and ask about it by topic. “Not covered” means the page was never found.
- Read the sources it cites. If the quoted passage is a list of menu items, a cookie notice or a release banner, the assistant is reading your layout, not your content.
If any of the four fails, the problem is the reading, not the model. Changing the AI model will not fix it, and neither will rewriting your docs.
What reading a site properly involves
The fix is simple to state and costly to do: read each page the way a visitor’s browser does.
- Open the page in a real browser, not a downloader. Run the scripts and wait until the article itself is ready, not just until the first elements appear on screen; keep waiting until the page stops changing. Some sites finish in a second. Others fetch sections one at a time and need several seconds before the last table appears. Stopping early produces half pages.
- Find pages from the site map, not only from links. Most documentation platforms publish a list of every page. Using that list alongside the links means deep pages are found even when no link to them survives in the shell.
- Keep the article, drop the frame. Strip the sidebar, header, footer and widgets, and keep the headings, paragraphs, tables and code of the article itself. Then the assistant’s sources are passages from your docs, not your navigation.
- Be honest about time. A real browser is slower than a downloader. Setting up takes minutes; reading a large site takes longer, and anyone who promises thousands of pages in a minute is downloading shells.
This is how ChatbotIQ reads documentation, and it is the reason we built our own reader rather than using an off-the-shelf crawler. Every page is opened in a browser, given time to settle, cleaned down to its article, and only then handed to the assistant. It makes the first read of a large site take a while. It also means that when a visitor asks about the table that loaded last on a page three levels deep, the assistant has seen it.
If you want to see the difference on your own site, start a trial, paste your documentation URL, and ask the assistant about the page that failed test two or three above. That is the page worth testing. The ones that work with every crawler will make every assistant look good.