A Confidence Score Is Only Useful if the System May Act on It

Almost every retrieval system has a confidence number in it somewhere. Cosine similarity between the question embedding and each chunk, a reranker score, a log probability on the generated tokens. The number exists. What most pipelines do with it is sort by it and take the top few, which means the number decides ordering and never decides whether to proceed. Something always ranks first. The system answers.

That is a design choice rather than an oversight, and for most applications it is the right one. In a DDQ or an ODD questionnaire it is not. Some share of what arrives cannot be answered from the documents the firm holds, because the consultant asks about a period you never reported, an entity you are not, or a rule you are not subject to. We have not measured what that share is across the industry and we are not going to guess at it here. What we have measured is what a system does when it meets one.

Half the Test Set Was Impossible on Purpose

When we built the hallucination benchmark in March 2026, the test set was 120 questions and 60 of them had no answer in the corpus. The corpus was 15 real DDQs, ODD questionnaires, ESG surveys and APRA guidelines from an Australian equity fund manager. The 60 unanswerable questions were generated by an independent model to be domain plausible but impossible: another manager's proxy voting policy, a Sharpe ratio for a quarter that appears in no document, an SEC rule, executive pay detail that a DDQ never contains, and the outcome of an enforcement action that never happened.

Three systems ran on identical infrastructure so the pipeline was the only variable. DeepSeek R1-0528 via Azure AI Foundry, nomic-embed-text at 768 dimensions, PostgreSQL with pgvector on an IVFFlat index with cosine similarity. System A was a direct model call with a compliance system prompt and no retrieval. System B was standard RAG, a single embedding similarity search with top-k chunks injected into the prompt. System C was our pipeline. The full method and results are public.

The result on the unanswerable half is the one that changed how we talk about this. Standard RAG refused 33 of 60, hedged on 8 and hallucinated on 19. A 55.0% refusal rate and a 31.7% hallucination rate. The plain model with no retrieval at all refused 42 of 60 and hallucinated on 12, a 70.0% refusal rate and 20.0% hallucination.

Adding retrieval made fabrication worse. The hallucination rate went from 20.0% to 31.7%, and the refusal rate fell from 70.0% to 55.0%. The reason is mechanical and it is not a bug in the retriever: retrieval always returns something. Ask an embedding index a question it has no answer to and it will still hand back the nearest k chunks, along with a similarity score that looks respectable because the question was written to be domain plausible. The model then reads genuine fund documents that are topically adjacent to a question about someone else's proxy voting policy, and writes confidently from them. The plain model, with nothing in front of it, had no material to build a plausible fabrication out of, so it declined more often.

The Score Has to Be About the Right Thing

Our pipeline scored 59 proper refusals, 1 hedged and 0 hallucinated on the same 60 questions. A 98.3% refusal rate and a 0.0% hallucination rate on that half.

Three retrieval choices contributed. Hybrid retrieval, text embeddings alongside ColPali visual embeddings that read layout and tables rather than a flattened text stream. Question and answer pair matching, where pairs extracted and verified at ingestion time are returned directly when an incoming question matches one. Recency ranking, so the current edition of a document beats the superseded one. All three raise the chance that the right region of the right document is found. None of them tells you whether it was.

That is the part worth separating out. A retrieval score tells you how close the nearest chunks were, not whether they answer the question, and topically adjacent material sits close to a plausible impossible question by construction. We did not record the similarity distributions on the unanswerable half, so treat that as the argument for the design rather than a result of the run. The design follows from it either way: a threshold on retrieval score filters the cases where retrieval was visibly struggling, which are not the cases that hurt.

So the score the pipeline acted on was computed after extraction, not before. It extracted the specific answer from the located region of the document, guided by document structure and the question. A separate model call then took the question and that extracted answer and asked one thing: does this actually address what was asked. That call returned a confidence score between 0 and 1, and below the threshold, 0.5 by default, the system returned a structured refusal instead of the answer.

That is a second inference call per question, with the cost and the latency it implies. We think it is the single choice most responsible for the 98.3%, because it works on what the first pass produced rather than on the evidence the first pass was handed. This describes the pipeline as the benchmark tested it in March 2026, not a claim about what any particular deployment runs today.

Hedging Is What a Model Does Instead of Refusing

The benchmark judge scored unanswerable responses in three categories: Proper Refusal, Hedged, Hallucinated. The middle one is the interesting category, and it is the one easiest to read as a success. We separated it out because we had read our own early results that way.

A hedge is an answer that gestures at uncertainty in prose while still supplying content. "The fund's approach would typically involve" and then three paragraphs. Standard RAG hedged 8 times, the plain model 6, our pipeline once. To a reviewer working through a questionnaire against a deadline, a hedge is worse than a refusal, because a refusal is a state the workflow can act on and a hedge is prose that a person has to adjudicate one at a time. Hedging is what you get when the decision to answer is made inside the same generation that produces the answer. Moving the decision out of the prose and onto a number is what makes it something other than a tone.

What the Threshold Costs

On the answerable half our pipeline scored 42 correct, 16 partial, 1 incorrect and 1 hallucinated. That is 96.7% accuracy against 30.0% for standard RAG, and the figure counts correct and partial together, so read it as usable rather than complete: 42 of the 60 were fully correct. Sixteen partials against 42 fully correct is a real share of the answerable set, and a threshold at 0.5 is permissive. Raise it and we would expect some of those partials to stop being partials and start being refusals.

Whether that is an improvement depends on the work. For a DDQ, a partial answer an analyst can finish is probably worth more than a clean refusal that sends them back to the source documents. For an ODD response going to an institutional allocator, the calculation may run the other way. The March run used the default and did not sweep the threshold, so we cannot tell you where the curve turns. That is a judgment about the workflow, not a number we measured.

One more thing the numbers say. Of the 60 unanswerable questions, one did not come back as a proper refusal, and on the answerable half one answer was scored hallucinated. The verification pass is a model call, and it is occasionally wrong about whether an extracted answer addresses a question. 98.3% is not the same as never, and that gap is the mechanism failing rather than noise in the scoring. These are March 2026 figures on the stack the benchmark used, and the threshold question above is still open.