The agent has no write access
I was moving more than fifty tickets a month from open to close by hand on a fraud platform whose findings end up in case files. I specified an agent that runs the whole QA and UAT loop as a real user persona against 55 deterministic correctness checks, and it cannot write to the ticket system without my explicit approval.
Specify an agent that runs persona-driven QA and UAT end to end against the fraud platform, and decide what it is allowed to do without a human.
Role: I own the fraud, waste and abuse product surface at 4L Data. I specified this agent, wrote its oracle library, its persona packs and its operating rules, and I am the approval gate inside it.
Context: A healthcare fraud platform whose findings reach investigators, and whose investigators’ findings can reach a state attorney general. I was moving more than fifty tickets a month from open to close by hand.
What I owned: An oracle library of 55 machine-checkable correctness rules, two personas built from interviews and structured questionnaires with the experts who do the job, a three-loop architecture with a human approval gate between each loop, the findings ledger, and the voice rules that make every drafted ticket read as mine.
Outcome: The QA and UAT loop I was running by hand, specified as an agent that runs end to end as a user and stops at every point where something would be written.
The failures that matter on a fraud platform are confident and wrong.
A tile reconciles to the chart and disagrees with the export. A count of flags is labeled as a count of providers. Elsewhere a provider shows as deactivated while the national registry says active, or an AI summary names a finding that is not in the data and then a different one on the next page load.
Nothing crashes. An investigator sees a clean screen, acts on it, and a named clinician ends up under investigation.
Click-through testing catches the visible half. The rest are reconciliation failures across four surfaces (tile, chart, list, export), and they have to be checked on every module on every release. Nobody does that by hand at the cadence this product ships.
Before an agent could test anything, someone had to write down what correct means on every surface, in a form a machine can check.
The result is an oracle library of 55 checks. Fourteen are universal and apply to every module: a tile must reconcile to the chart, the list and the export; a filter must visibly change what it claims to change; sorting three times must return the starting order; not available must mean not applicable, never unknown; a reload must produce the same screen.
The other 41 are tied to the business logic documents behind each module. An integrity flag total counts distinct providers and can never exceed the provider count. The last sanction date is the newest parseable date and is never blank. A lead’s dollar exposure equals the sum of its findings, or the screen explains the difference.
Alongside the library sits a register of thirteen known, intentional behaviors. The agent reads it before every sweep, because reporting a known limitation is a false positive, and engineers stop reading a tool that produces them.
Every check is deterministic. Most of what goes wrong in a data product is a reconciliation failure, and reconciliation is arithmetic. I applied the same rule to the AI inside the product, where finding titles come from source tables and evidence strength from a fixed check order.
A generic persona produces a generic sweep that finds generic bugs, so the personas came before the agent.
I compiled an evidence digest first, from UAT documents, recorded sessions, the existing ticket history and email threads. Then I built two personas from comprehensive interviews and structured questionnaires with the subject matter experts who do these jobs: a special investigations investigator and a payment integrity manager.
Both are specific enough to drive behavior. The investigator reads the AI summary first and loses trust permanently if it contradicts itself or changes between loads. She checks health system affiliation before accepting any referral pattern as collusion, since referrals inside one system are not collusion. She ranks by what protects the member and what legal will accept, so a provider with ten board cases outranks a fifty thousand dollar self-referral.
The payment integrity manager starts from totals and works down, changes one filter at a time while recording which tiles respond, and asks three things of every field: what it is, where it came from, and why it says not available.
More personas are in production as questionnaire data comes back from other experts, and each is built into the architecture as it lands.
The obvious build is an agent that tests and files. I specified three loops with a human gate between each one.
The exploring loop cannot touch the ticket system beyond read-only queries that check whether something is already known. Triage is a separate loop that drafts tickets without creating them. A ticket is created only when I approve that specific draft. I also wrote a rule for the gap that agents exploit most: silence, "looks good" and "ok" do not count as approval, and if approval is not explicit, the agent asks.
The two possible mistakes have very different costs. A missed defect costs one cycle, and the next sweep catches it. A wrong ticket costs an engineer a day, adds noise to a backlog other people plan against, and teaches the team the tool is not worth reading. In a system of record a bad write costs far more than a missed read, so writes are what I gated.
The agent is also forbidden from proposing features. Its findings are observations. An exploring agent that drifts into product opinion creates work for people who did not ask for it.
The third loop closes the cycle. It retests returned fixes against the exact anchors in the original ticket, checks the neighbors of each fix for side effects, drafts the reply, reassigns to whoever the question is for, and writes the verdict back to the ledger.
Every drafted ticket and comment follows voice rules I wrote: one plain sentence stating the problem, concrete reproduction steps, no more than three numbers in a sentence, and technical content phrased as a question, because the engineers own the how.
By hand, the limit was my attention. More than fifty tickets a month moved from open to close, and the reconciliation checks were the first thing skipped on a short day.
With the loops running, the 55 checks run the same way on every module, release week or not. What remains for me is judgment: whether a finding is real, whether it is worth an engineer’s time, and whether what came back is fixed. That is now the only part of the loop that waits on a person.