Checking a Number Against a Bag of Numbers Is Not Checking

The check that writes itself

A Statement of Advice is generated with a model in the loop, and the platform already holds the client's real numbers: balances, salaries, premiums, fees, the lot. So the check writes itself. Take every dollar amount in the document, compare it against the numbers on file, flag anything that does not appear there. If the model invents a figure, the figure will not be in the client's data, and the reviewer sees it.

We built that. It ran, it fired, it caught things. Then we measured it against the failure it exists to catch, and found it could not see that failure at all. Not rarely. Never, by construction.

How we measured it, and why the method mattered more than the result

The measurement was deliberately boring. Take completed SoAs, real generated text rather than synthetic samples, corrupt a single figure at a time across every money-bearing sentence, re-run the check, record whether it fired. Four kinds of corruption:

A modest drift, a figure nudged enough to be wrong but not enough to look absurd. An order of magnitude, an extra zero. A digit swap, two digits transposed. And a substitution: replace the figure with a different real figure taken from the same client's own file.

The first three were caught at high rates. The ordering was mildly interesting on its own. Drift was caught most reliably and the digit swap least. We did not establish why, and the ordering is not the point.

The substitution was caught zero times. Not a low rate. Zero, across every instance we generated.

Why zero was the only answer the check could give

The trusted set was a flat bag of every number found anywhere in the client record, plus the comparison tables. A substitution draws its replacement from that same bag. So the corrupted figure was, by definition, present in the trusted set, and the check reported no problem. The result was not a tuning failure or a threshold set too loose. There is no threshold that fixes it. The question the check was asking, "is this number one of the client's numbers", returns yes for every substitution that will ever occur.

That is the part that should worry anyone building this. The failure we were most afraid of was not invention. It was a real figure in the wrong sentence. It states the income where the balance belongs, the premium where the sum insured belongs, last year's fee where this year's goes. Every figure in the paragraph is genuine. The document is still wrong, and wrong in a way that survives review precisely because each number looks right when you check it against the file.

We had built a check that scored well against the corruptions we thought to test and scored nothing against the one we most wanted to catch. Which of the four a model commits most often is not something this run measured, and we have not measured it since.

The traversal threw away the meaning and kept the digits

The cause was one line of design, taken without thinking about it. The traversal that collected trusted numbers walked the client record and kept the values while discarding the keys. Sum insured, balance, salary, monthly benefit, employer contributions: dozens of distinct money-bearing paths, each one flattened into an anonymous amount in a list.

The keys are the meaning. Without them, a figure has no claim attached to it, so there is nothing to contradict. With them, a figure in the document binds to the concept its sentence names, and is checked against the value the record holds for that concept. That is the fix, and it is not clever. It is just refusing to throw away the only thing that makes verification possible.

A check that fires is not a check that helps

The second lesson cost more work than the first.

Across the same corpus the original check produced a heavy flag load, and the middling document carried closer to ten flags than to three. The flags clustered near figures that were genuinely on the record. Rounding. A figure stated to the dollar in one place and the nearest hundred in another. A monthly amount stated annually, twelve times a number the system trusted, flagged because the product itself was not in the bag.

All of it rendered identically. A reviewer opening that document sees a wall of amber with no way in. We do not have data on what they did next, and we did not need it to know the report was not doing its job.

The check was rebuilt in September 2026, and flags are now graded by what they actually claim. A contradiction of the record is high: the sentence names a concept, the record holds a value for that concept, and the figure is not that value. That is the only finding worth interrupting someone for, and it is exactly the finding the original check could never make. Below it sits no match, medium, meaning the figure traces to nothing we hold, so we cannot say it is wrong, only that it is not on the record. Then near miss and derived, both low, for figures within a hair of something real or explained by arithmetic on a trusted amount.

The skip list was doing the opposite of its job

One more thing we got wrong, and it is worth its own paragraph because it is the kind of error that hides for a long time.

A skip list existed to keep the check off boilerplate sections. It matched on words in section titles. It was skipping the majority of sections in the corpus, and with them most of the material dollar figures in the document, because the words it matched on appear in the titles of the sections you most want checked. A section on costs and disclosures was skipped for the word disclosure. A section about the client was skipped for the word about.

Deleting it outright was worse. The flag count multiplied to the point where the report was unreadable, which is its own kind of failure. So nothing is skipped now, and a narrower list of general sections reports only the high severity kind. A statutory cap quoted in a strategy section says nothing about the client. The client's salary misstated in that same section says a great deal.

What it still will not do

It flags. It does not edit the document, it does not block generation, and when it errors it contributes no flags rather than failing loudly. What it raises says please verify, not this is wrong. It is a reading aid for the person who signs, and the person who signs is still the control.

What generalises is the method, not our particular bug. A verification layer has to be measured against the failure it exists to catch, and the intuitive test measures a different one. Drift, magnitude errors and digit swaps are easy to catch and comparatively rare. Substitution is common and stays invisible unless the checker understands what each number claims to be. If you are building assurance over generated documents and you have not deliberately tested the substitution case, you probably have this hole, and it will not show up in any metric you are currently looking at.

What we can say now is narrower than what we thought we had. The graded check can represent a contradiction, where the old one could not form the thought. How often it catches one in production is a measurement we have not taken yet.