Deciding what the AI is allowed to say
A fraud detection platform where model output can end up in a case file. I wrote the interaction specification and drew the line between what the AI generates and what the system asserts as fact.
Interaction specification, AI content boundaries, design, and release priority
Role — Product owner for the fraud detection product surface. I wrote the interaction specification, owned the design, and set the boundaries on AI-generated content.
Context — A prototype built for a conference demo, never productionized. Investigators buried in manual data-gathering, working leads with a low true-positive rate.
What I owned — Fifteen numbered interaction sections across an eight-region modal, seven AI-backed surfaces, four finding-title format rules mapped to four source systems, accessibility specification, performance budgets, and eleven named test conditions.
Outcome — A specification engineering built from directly, with the AI boundaries written as hard rules rather than guidance.
Investigators were spending their time assembling information rather than judging it. The existing lead sources produced enough false positives that the hours they did have went into cases that went nowhere.
An AI layer is the obvious answer, and it was already half-built — a prototype that demoed well and had never been productionized.
The non-obvious problem is what happens when that output lands in a fraud case file. A finding here can become the basis for contacting a provider, opening an investigation, or making a referral. Provider NPIs are public and registry-linked; they identify real, named clinicians. Getting it wrong is not a bad recommendation, it is a reputational event for a person who did nothing.
So the design question was never how to add AI. It was: where does model output stop and asserted fact begin?
I wrote it as a hard rule in the specification: finding titles are data-driven, never AI-generated. No model interpretation of finding names, anywhere.
Titles map directly out of the source systems — the scheme detection tables, the provider data field names, and two separate code tables. Four format rules, four sources, deterministic in every case.
The AI writes the summary. It writes the risk reasoning. It writes the recommended next action. It never writes the label on a finding.
The reasoning is that the name of the allegation is the allegation. “Billing spike” and “phantom billing” are different accusations with different consequences, and the difference between them is a matter of fact, not of phrasing. A model that generates the title is a model that decides what a provider is being accused of. The summary underneath can be regenerated, argued with, or ignored. The title goes in the file.
Evidence strength is binary, not scored. The obvious move is a confidence percentage. I specified Available / Not Available, resolved by a deterministic check order. A number implies a precision the system cannot support and invites an investigator to treat 81% as meaningfully different from 76%. A binary can be verified in one click.
Every AI surface degrades to a working product. Three-second timeout on the summary, the risk score and the recommendation, each with a named fallback. If AI is unavailable the interface renders its complete non-AI equivalent rather than an error state. The failure mode is not a broken screen — it is the product without the assistance.
The assistant is bound to the open lead. Scoped to the client’s own data and the record the user is inside. It is not a general model with a chat box on it.
Output is length-capped by role. Short enough that an investigator reads it instead of skimming, and short enough that the model has no room to wander into inference.
Machine-generated content has its own colour channel. In the working prototypes I built alongside the spec, AI output renders in a reserved hue used nowhere else in the interface. In an evidentiary workflow, being able to see at a glance which words came from a person, which from the platform and which from a model is a safety feature, not a styling choice.
Three proposed tiles died inside the same document I proposed them in — an AI-generated investigation priorities tile, an evidence collection checklist, and an executive summary panel, all replaced by simpler affordances once it was clear they were adding surface rather than answering a question.
I also logged a comprehension gap my own scoping created, in writing, with a stated workaround, rather than leaving it undocumented for someone to discover.
Separately, on the finding-prioritization model, I proposed a multi-category weighting scheme — the idea being that a provider triggering several categories of finding should outrank one triggering a single category.
I tested it against investigator judgment. It produced the wrong investigative priority, ranking a lower-exposure provider above a higher-exposure one. The intuition was reasonable and the result was wrong, so it was cut. A related claim-count factor went the same way.
What shipped instead was a scoring approach with a deterministic override layer on top: certain integrity facts — an active federal exclusion, a deceased provider with live claims — escalate regardless of what the arithmetic says. Some things are not a matter of degree.