Everyone who has owned a documentation set has the same private suspicion: that some meaningful fraction of it is quietly wrong, and that they do not know which fraction. We wanted to replace the suspicion with a number, so we built a corpus and measured it.
The corpus is 1,180 changes drawn from 42 open-source repositories that keep their documentation in the same repository as their code. For each change we know what the code did, and — because the docs are versioned alongside it — we know whether a human eventually updated a page in response, and how long that took. That last part is the useful bit. The gap between the commit that made a page wrong and the commit that fixed it is a directly observable quantity, and nobody seems to have written it down.
The headline number
Across the corpus, the median gap between a change that invalidated a passage and the edit that corrected it was 47 days. The mean was 121 days, dragged out by a long tail of pages that took more than a year. Eleven per cent were never corrected at all within the window we observed; someone eventually deleted the page instead.
That is the number people quote back at us, and on its own it is not very actionable. The interesting result is in the spread.
What actually predicts decay
We expected page age to matter. It barely does. A page written three years ago is not meaningfully more likely to be wrong today than one written last spring, once you control for how often the code beneath it changes. We expected page length to matter. It does not, except insofar as long pages describe more things.
What predicts decay, overwhelmingly, is how many concrete values a page states. Not how much code it contains — how many specific, checkable facts. A default of 5. A header named X-Acme-Signature. A flag called --watch. A rate limit of 100 requests per minute. Each one of those is a claim that the code can falsify.
| Concrete values stated | Median days to first wrong claim | Share of pages |
|---|---|---|
| 0–2 | 412 | 31% |
| 3–7 | 148 | 38% |
| 8–15 | 61 | 22% |
| 16+ | 29 | 9% |
A conceptual overview page that explains why retries exist can sit untouched for years and stay true. The page next to it that lists the retry defaults has a half-life measured in weeks. They look like the same kind of artefact in a file tree. They are not.
The second finding: silence is not agreement
We assumed that a wrong page would generate a complaint reasonably quickly — an issue, a discussion thread, a pull request from an annoyed reader. Mostly it does not. Of the passages we identified as wrong, only 18% had any corresponding issue or discussion filed before they were fixed.
The other 82% were found by someone on the team, usually while doing something else. Which means that the volume of complaints about a documentation set tells you almost nothing about its accuracy. It tells you about its traffic.
The pages nobody complains about are not the correct ones. They are the ones nobody reads carefully enough to notice.
The third finding: fixes cluster
Documentation corrections do not arrive at a steady rate. They arrive in bursts, and the bursts correlate with two events: a major version release, and a new person joining the team. In the corpus, 44% of all documentation corrections landed within two weeks of one of those two things.
That is an efficient way to work if you are optimising for effort. It is a poor one if you are optimising for the reader who arrived in month seven of an eleven-month release cycle. They read a page that four people already know is wrong and nobody has got to yet.
What we did with this
Three things, in the product.
- We index pages by the concrete values they state, not by their prose. A page's watchlist is the set of symbols, literals, flags and paths it commits to. That is the thing a change can break.
- We treat the absence of complaints as no signal at all. Staleness is derived from the code, never from reader behaviour.
- We flag at the moment of the change, not at release. If a default moved on a Tuesday, the page that documents it should be in a review queue on Tuesday, while the person who moved it still remembers why.
What we would do differently
The corpus over-represents open source, which over-represents projects that care about documentation enough to version it in the repository. Private docs sets in a wiki are almost certainly worse, and we cannot measure them. If your documentation lives somewhere the code cannot see, assume the numbers above are optimistic.
We are also aware that "a human eventually edited this page" is a proxy for "the page was wrong", not a proof of it. Some of those edits were improvements to pages that were already correct. We hand-checked a random sample of 200 and found the proxy held 87% of the time, which is good enough to reason with and not good enough to be smug about.