Skip to content

How it works

Six stages, and most changes never get past the second.

The interesting engineering in Docsmith is in what it refuses to do. Below is the whole path a commit takes, including the two places work is thrown away on purpose.
01

Index

Once, then incrementally

On connection, Docsmith walks the paths you nominated and parses every page. It is not building an embedding of your prose — it is extracting the checkable claims: symbols named, literals stated, flags used, headers referenced, error strings quoted, code blocks in full.

In parallel it parses the source tree with a language-aware pass — TypeScript, Go, Python, Rust, Java and Ruby today — to build the reverse map: for each exported symbol, which pages mention it.

A 340-page docs set with a 200,000-line repository takes about four minutes. After that, only what changed is re-indexed.

Index entry

page
docs/guides/retries.md
symbols
RetryOptions, maxAttempts, delayMs
literals
5, 200, 1, 409, 800
code blocks
2
owner
platform-team
02

Triage

Every push

A webhook delivers the change. The first question is whether it can affect documentation at all, and for four pushes in five the answer is no: a lockfile bump, a formatting pass, a test-only change, a rename confined to a private helper.

This runs on a fast model with a tight prompt and costs a fraction of a cent. It exists because the alternative — sending every push to a frontier model — is both expensive and noisy, and because a tool that cries wolf on a Prettier run gets muted in a week.

Commits dropped at triage do not count against your plan.

Dropped at triage

dependency bumps
34%
formatting only
19%
tests only
16%
private-scope renames
12%
reached detection
19%
03

Reach

Per surviving change

The change is resolved to the set of symbols and values it touched, and the reverse map turns that into a candidate set of pages. This is ordinary static analysis, not a model, and it is the reason Docsmith does not have to read your whole docs set on every commit.

Reach is deliberately generous at this stage and narrowed later. A page that merely mentions a changed symbol is a candidate; whether it is actually wrong is the next question, and a more expensive one.

Reach for a4f19c2

symbols changed
4
pages mentioning them
3
sent to detection
3
found stale
1
04

Detect

Per candidate page

Each candidate page goes to a model with the change, and comes back with the passages it believes are now wrong. Every finding must quote its passage verbatim.

Then the guard runs: each quote is looked up in the page that was submitted, after normalising whitespace and smart quotes. A finding whose quote cannot be located is dropped — not shown with a caveat, not softened. Roughly one in seven findings dies here, and the reasoning panel reports how many, so the filter is visible rather than silent.

This single check took our false-positive rate from 19.4% to 6.1% with no change to the model.

Verification

findings proposed
7
quote located
6
quote not found
1 — dropped
below confidence floor
1 — hidden
05

Draft

Only when something is stale

The page is rewritten whole, in its own voice, changing only what the evidence requires. No reorganising, no added changelog note, no tonal drift toward the model's house style. If a default moved from 5 to 3, the number changes and the sentence around it does not.

The result is diffed against the original. If the diff is empty, the verdict flips back to current and nothing is queued — Docsmith does not open a pull request that only reflows prose.

Draft output

verdict
stale
lines added
3
lines removed
3
sections touched
2 of 5
model
frontier tier
06

Review

You

The draft arrives with the evidence attached: the passage, the reason, the commit and the confidence. Approve it, edit it first, or reject it with a reason.

Rejections are the most useful signal we get. They are aggregated (never the page content, only the reason and the shape of the finding) and used to tune the confidence floor and the detection prompt against the frozen evaluation set.

What Docsmith will not do

merge a pull request
never
edit outside your paths
never
push directly to a branch
never
train a model on your code
never

Honest limits

What it is bad at

Screenshots

A UI change that invalidates an image is invisible to Docsmith today. Detecting it is on the roadmap for this quarter; until it ships, images are your problem.

Behaviour with no textual trace

If a change alters what a function does without touching any name, literal or signature, reach will not find the page. Detection is anchored in things a page can state.

Docs the code cannot see

A wiki with no API, or a Google Doc, cannot be indexed. The sync API will still draft an update if you can hand it the page.

Deciding whether the change was right

Docsmith assumes your code is the truth and the page is behind. It will happily document a mistake you just shipped.

The model side of all this is described on How the AI works.

Get started

See it on a change of your own.

The demo on the home page runs the real engine. Paste one of your diffs and one of your pages — nothing you type is stored.