Make the model quote the page

For the first eight months, the most common piece of negative feedback we got was some version of: it flagged a page that was fine. Not wrong in an interesting way — wrong in a boring, corrosive way that makes someone stop opening the review queue.

We spent a long time treating this as a model problem. We tried better models, longer prompts, few-shot examples of correct restraint, a second model as a critic. Each one moved the number a little. None of them fixed it, because it was not a model problem. It was an interface problem: we were accepting a claim without requiring the evidence for it.

The shape of the failure

Here is a real example, lightly anonymised. A change renamed an internal helper from formatCurrency to renderMoney. Nothing public. The model returned this finding:

{
  "section": "Formatting amounts",
  "quote": "Use formatCurrency() to display a monetary value.",
  "why": "formatCurrency was renamed to renderMoney in this change.",
  "severity": "breaking",
  "confidence": 0.91
}

Which is a completely reasonable finding, except that the page did not contain that sentence. It contained a paragraph about formatting amounts that never named the function. The model had written the sentence it expected to find, attributed it to the page, and then reasoned correctly about the sentence it had invented.

This is the failure mode people mean when they say hallucination, but describing it that way makes it sound mysterious. It is not mysterious. We asked for a citation in a field called quote, and nothing in the system checked whether the quote was real.

The change

Twelve lines, roughly. Before a finding is shown to anyone, the quote is looked up in the page that was submitted. If it is not there, the finding is dropped.

function normalise(text: string): string {
  return text
    .replace(/\r\n/g, "\n")
    .replace(/[\u2018\u2019]/g, "'")
    .replace(/[\u201c\u201d]/g, '"')
    .replace(/\s+/g, " ")
    .trim()
    .toLowerCase();
}

export function verifyQuote(quote: string, doc: string): boolean {
  const q = normalise(quote);
  if (q.length < 8) return false;
  return normalise(doc).includes(q);
}

What it did to the numbers

On the frozen evaluation set, with no change to the model or the prompt:

Two thirds of our false positives were findings about text that did not exist. Recall got very slightly worse, which is the honest trade: a small number of real problems were described in a paraphrase the check could not match. We took that deal without much agonising. A missed page costs a reader something once. A page flagged wrongly costs us the reviewer's attention permanently.

Why not just ask the model to be careful

We did, at length. The prompt has said "quote exactly, character for character, never paraphrase" for over a year. It helps. It does not eliminate the behaviour, and more importantly it cannot be verified. An instruction is a hope. A check is a guarantee.

The general form of this, which we now apply everywhere: if the model produces a claim about a document you already have, you can check the claim against the document mechanically. Do that. Do not ask the model to self-report its confidence and then threshold on it — a confidence score is another generated token, and in our data it correlated with fluency rather than correctness.

Making the filter visible

One decision we went back and forth on: whether to tell the user that findings had been dropped. The argument against is that it advertises a failure. The argument for is that a silent filter is indistinguishable from a filter that is not running.

We show it. The reasoning panel says how many findings were dropped for an unverifiable quote. Reviewers report that it makes them trust the ones that survived, which is the opposite of what the argument against predicted, and is now our default instinct on questions like this.

Three things that did not work

• Fuzzy matching with an edit-distance threshold. Every threshold we tried either let invented sentences through or rejected legitimate re-wrapping. Normalising the input and requiring an exact substring is strictly better and much easier to reason about.
• Asking the model for character offsets instead of the quote. Models are poor at counting characters and the offsets were wrong far more often than the quotes were.
• A second model to judge whether the first model's quote was faithful. It cost 40% more per sync, added latency, and was less accurate than string matching at the one job string matching is perfect at.