What Has to Be True Before a Licensee Accepts an AI-Drafted Statement of Advice

Most evaluations of advice software start on model quality. The more useful subject is what the software refuses to do, because that is the part a file review can inspect.

No two licensees run the same file review, so what follows is not a list of their requirements. It describes how SoA automation is built here. It is the list we designed to, and the reasoning is offered so a responsible manager can argue with it. Four things: the template stays the firm's; a document with an unresolved placeholder cannot render; a dollar figure is checked against the client record wherever the sentence names something the record also holds; and a human signs, because the reasoning is theirs and stays theirs.

The Template Is Data, Not a Prompt

The first thing a licensee wants to know is whose document this is. If the answer is "ours, generated to your requirements", the firm has outsourced its own advice format to a vendor and will be asking for a change request every time its process changes.

A firm's Statement of Advice template is read, not rewritten. The bracketed markers already in the template prose are classified into exactly three kinds.

A value marker resolves to a fact on file: a balance, a premium, a fee. It is looked up.

A conditional marker decides whether the surrounding text appears at all. It is evaluated to true or false.

A free text marker is an instruction to the writer, of the "explain why this product was chosen" sort. Only this third kind reaches a language model, and it arrives with the surrounding paragraph, the section title and the section context attached.

The invariant that matters to a responsible manager is that resolution is content driven. There is no hardcoded mapping from a particular marker to a particular answer. Nothing in the system says "when you see this bracket, write this sentence". That is what lets a firm change its template without an engineer, and it is the difference between advice software and a mail merge. It also means the model's territory is small and stated: prose where the template asked for prose, nothing else.

Six Ways a Placeholder Fails

The failures that end a licensee's trust are not subtle. They are a TBD in the strategy section, a $X where a premium belongs, an "insert reason here" that survived into the client copy. A reader spots those in two seconds. A licensee spots them in one.

So every rendered block is scanned on save for six classes of unresolved placeholder: a raw bracketed placeholder never filled, an XXX style stub, a dollar figure left as a literal $X, a TBD, an "insert something here" instruction that made it into the output, and a conditional tag that was never resolved either way.

There is nothing clever in that check. It is pure CPU, no model call, no file access, and it runs on every block save in under a millisecond. That is the point. The cheap deterministic check runs constantly and the expensive probabilistic one runs once. Most of what makes a generated document trustworthy is work of this kind: reading a template the firm already uses, deciding which brackets are decisions and which are facts, and refusing to render a document with a TBD in it. None of it is model capability.

The Figure Check We Had to Rebuild

Here is the part we got wrong, and it is the part worth a responsible manager's attention because the wrong version looks identical to the right one from outside.

The platform already holds the client's real numbers. So the obvious check is: does every dollar amount in the generated document appear in the client's data? We built that. It matched each figure against a flat collection of every number found anywhere in the client record and the comparison tables.

Then we measured it, by mutating a single figure in the real generated text of completed Statements of Advice and re-running the check. It caught small percentage drift nearly always. It caught an order of magnitude error slightly less often than that. It caught digit swaps less often again, and by then the miss rate was no longer a rounding error. Against the fourth corruption, substituting one real figure from the same client's file for another, it caught nothing at all. Not rarely. Never, across every attempt.

The reason is structural. Substitution means stating the income where the balance belongs, or the premium where the sum insured belongs. Every substituted value is already in the trusted collection by definition, so a check that asks only "is this number somewhere in the file" cannot see it. And substitution is how a language model actually gets a number wrong in a document holding scores of amounts. The easy corruptions are rare. The invisible one is common.

What went wrong in the build is that the traversal collecting trusted numbers walked the client record and threw the keys away. The keys are the meaning: sum insured, balance, total fees, salary, monthly benefit, employer contributions. A figure now binds to the concept its sentence names, and is checked against the value the record holds for that concept. That is the only version of the check that can make the finding that matters.

The second lesson cost more work than the first. A check that fires is not the same as a check that helps. The first version produced a handful of flags per document, and every one of them sat near a figure that was genuinely on the record, though most were further off than rounding would explain. Rendered identically, that is a wall of amber with no way into it. Flags are now graded by what they claim. A figure that contradicts the record, where the sentence names a concept, the record holds a value for it and the figure is not that value, is high severity and worth interrupting someone for. A figure that traces to nothing is medium: we cannot say what it claims to be, so we cannot say it is wrong, only that it is not on the record. A figure within rounding distance of something real, or explained by arithmetic on a trusted amount, twelve times a monthly figure stated annually, is low.

A skip list also existed, meant to keep the check off boilerplate. Measured, it was skipping most sections in the corpus and most of the material dollar figures, because it matched on words that appear in the titles of the sections you most want checked. "Costs and disclosures" was skipped for disclosure. "About You" for about. Deleting it outright made the flag volume far worse. So nothing is skipped now, and a narrower list of general sections reports only high severity findings. A statutory cap quoted in a strategy section says nothing about the client. The client's salary misstated in that same section says a great deal.

What Stays the Adviser's

The figure check flags. It never edits the document, never blocks generation, and if it errors it contributes no flags rather than false comfort. A flag says "please verify". It does not say "this is wrong".

That boundary is deliberate, and it is where a licensee should expect the software to stop. The suitability judgement is the adviser's. The reasoning in the free text, even where a model drafted the sentence, is the adviser's once they leave it in. The record of advice, the best interests duty analysis and the sign-off are the adviser's and the licensee's. The best interests duty sits in sections 961B to 961J of the Corporations Act, with ASIC RG 175 as the guidance on how it is met. It attaches to the advice and the adviser, not to the tool that drafted the document, and no amount of deterministic checking moves it.

Which leaves a genuine open question for a responsible manager, and it is not one we can answer from the product side. Our checks catch the mechanical failures at rates we have measured, and they miss some of those too. They catch semantic ones only where the sentence names a concept the record also holds. Everything outside that, an inference that is plausible and wrong, a strategy that is internally consistent and unsuitable, is caught by the adviser reading the document or it is not caught. The honest question to put to any vendor is not how accurate the model is. It is which checks are deterministic, what each one was measured against, and what is left over for the person who signs.