How the AI works

The model is the engine. That is exactly why we don't trust it.

Docsmith is impossible without language models and dangerous if it defers to them. This page is the whole arrangement: what routes where, what is checked mechanically, what the numbers are, and where your code goes.

Providers wired: OpenAI, Anthropic
Providers declared: Google, self-hosted open models
Training on your code: Excluded by contract
Evaluation corpus: 1,180 changes, frozen

Premise

Why this needs a model

Consider the sentence: “Because the delay is fixed, a request that fails five times takes roughly 800 ms of waiting in total.”

Now a commit replaces a fixed delay with exponential backoff and lowers the attempt count. Nothing in that sentence is a symbol. Nothing in it is a flag. The number 800 does not appear anywhere in the diff. Knowing that the sentence is now wrong requires understanding what “fixed” meant, what backoff does, and that 800 was a derived figure.

Every team that has tried to solve this with a linter has ended up with a linter nobody runs. It is a reasoning problem, and it became tractable about three years ago.

Routing

Each step of a sync is routed on its own

Sending every step to the best available model would be simpler and about eleven times more expensive, and triage does not need a frontier model to notice a lockfile. Routing is preference-ordered: the first candidate whose provider has a key configured takes the step, and if it cannot answer, the next one does.

Triage

Decide whether a change can plausibly affect documentation at all. A lockfile bump and a rename of a private helper are dropped here, before anything expensive runs. Roughly four in five commits stop at this step.

Preference order

gpt-4.1-nano
gemini-flash
gpt-4.1-mini

Detect

Read the change against the pages that reference the symbols it touches, and decide which passages the change has made wrong. Every claim must carry a quote from the page it is about.

Preference order

gpt-4.1-mini
gpt-4.1
claude-sonnet

Rank

Order the affected pages by how badly they mislead a reader now. A wrong default in a quickstart outranks a stale sentence in a changelog.

Preference order

gpt-4.1-mini
gpt-4.1

Draft

Rewrite the affected passages in the voice of the page they belong to, changing only what the evidence requires. The heaviest step, and the one sent to a frontier model.

Preference order

gpt-4.1
claude-sonnet
gpt-4.1-mini

The candidate set

Model	Provider	Tier	Routed for	USD / Mtok
gpt-4.1	openai	Frontier	long-context reading across a diff and a full docs page; holding an API contract in mind while rewriting prose	$8.00
gpt-4.1-mini	openai	Balanced	structured extraction with a strict schema; ranking candidate pages by impact	$1.60
gpt-4.1-nano	openai	Fast	deciding whether a commit can touch documentation at all; classifying a change as behavioural or cosmetic	$0.40
claude-sonnet	anthropic	Frontier	matching an existing document's voice when rewriting; declining to answer on thin evidence	$9.00
gemini-flash	google	Fast	very large docs sets; cheap first-pass triage	$0.50
llama-3.1-70b	meta	Open	self-hosting for customers who cannot send code to a vendor; data residency and a hard cost floor	$0.60

Cost is a blended figure used for routing decisions, not for billing — your plan price does not change with which model answered. A model is only callable when its provider has a key configured in the deployment; the rest are declared so the routing table stays honest about what would happen if one were added.

The guard

Verify, do not instruct

Our prompt has said “quote exactly, never paraphrase” for over a year. It helps. It cannot be relied upon, because an instruction is a hope.

So every quote is checked against the page it claims to come from, after normalising whitespace and smart quotes. Models re-wrap lines constantly, so exact equality would reject correct citations; normalising fixes that without letting an invented sentence through.

A finding that fails the check is discarded. The count of discarded findings is shown to the reviewer.

The write-up, with the before and after numbers →

lib/sync.ts — the whole check

function normalise(text: string): string {
  return text
    .replace(/\r\n/g, "\n")
    .replace(/['']/g, "'")
    .replace(/[""]/g, '"')
    .replace(/\s+/g, " ")
    .trim()
    .toLowerCase();
}

export function verifyQuote(quote: string, doc: string) {
  const q = normalise(quote);
  if (q.length < 8) return false;
  return normalise(doc).includes(q);
}

Twelve lines. It moved the false-positive rate from 19.4% to 6.1% with no change to the model or the prompt, and cost 0.6 points of recall.

Evaluation

The numbers, including the ones that are not flattering

Corpus size

1,180

Real changes from 42 repositories that version docs with code.

False positives

6.1%

A page flagged that did not need changing.

False negatives

9.2%

A page that needed changing and was missed.

Draft accepted unedited

71%

Of approved drafts, the share approved with no edit.

Draft accepted with edits

22%

Approved after the reviewer changed something.

Rejected

The reviewer decided the page was fine, or the draft was wrong.

Method

The corpus is 1,180 changes sampled from 42 open-source repositories that keep documentation in the same repository as code, so that “a human later edited this page in response” is observable. That proxy held in 87% of a hand-checked random sample of 200, which is good enough to reason with and not good enough to be smug about.

The corpus is frozen between releases. Every prompt change, routing change and model swap is scored against it before it ships, and a change that looks better and measures worse does not ship.

The sampling frame over-represents projects that care about documentation enough to version it. Private docs sets are almost certainly worse than these numbers suggest.

Your code

Where it goes, and for how long

What is sent: The diff of the change and the text of the candidate pages. Not the repository, not adjacent files, not your git history.
Who receives it: The provider that answered the step. Which provider handled which step is recorded on the sync, so it is auditable after the fact.
Training: Excluded by contract with every provider we route to. Zero-day retention is set where the provider offers it.
Retention here: Diff and page content live in a sync-scoped cache that expires within 24 hours. Business plans can set it to zero, keeping only the audit record.
Staying in your network: Business customers can point routing at a self-hosted open model through a gateway URL, in which case no code leaves your boundary.

The full control list is on the security page.

Get started

Judge it on your own repository.

Thirty syncs a month on the free plan, using the same routing table described above.

Start free Read the security page