Every financial firm has an AI policy. It says the models will be fair, will stay grounded in real data, will not be easy to manipulate, and will refuse what they should refuse. The policy is signed, filed, and accurate as a statement of intent.
Then a model risk officer asks the question a policy does not answer: what is the number, and where is the evidence?
That is the distance between a principle and a control. A principle is a sentence everyone agrees with. A control is a measurement with a threshold, an owner, and a record that can be handed to an examiner. In regulated finance, only the second one counts. Model risk guidance has said for years that a model that cannot be validated should not be run. The same standard now applies to how an AI system behaves, and most teams have no instrument to meet it.
PROBE is one such instrument, built by the author. This article covers the part that matters to a governance team rather than an engineer: how a principle becomes a KPI that can actually be run, where that fits inside the way regulated firms already govern models, and why the design reaches beyond chatbots.
A principle is not a KPI until it has a threshold
"Be fair" is not measurable. It is a value. It becomes a KPI only when four things are committed before anything runs.
- A precise definition of the violation to observe. Not "fairness" but "the model gives a worse answer to one group when the only thing that changed was a name."
- A repeatable test that exposes that violation. For fairness, a set of matched messages identical except for a male or female name.
- A single number with a pass line set in advance. The average of the per-test scores, against a threshold written down before anyone looks at the result.
- The evidence kept alongside the number. The worst cases, side by side, plus how the score was produced.
The last rule matters most to governance teams. A KPI without its evidence is a number someone is asked to trust. A KPI with its evidence is a control that can be defended. PROBE never returns the first kind.
What a governance framework asks, and what an instrument answers
PALO, published at paloframework.org, is a well-designed governance framework. It runs an AI use case through a lifecycle: frame the use case, classify the risk, assess the impact, design the controls, measure and evidence, then prove and review. It is a way to make and record decisions across the life of a model.
PROBE does not compete with that. It plugs into the measure step. A lifecycle framework says that a control must be measured and evidenced. It does not, on its own, produce the number. PROBE produces the number and the evidence for one behavioral principle at a time, in a form the review step can file. The framework is the operating procedure. The instrument is what a team picks up when the procedure says measure.
Naming that split matters, because a common failure is to treat a signed framework as if it were a working measurement. It is not. The gate that reads "measure and evidence" stays a checkbox until someone owns the instrument that fills it.
The number has to meet the standard of validation
A serious risk team raises one objection first: the model is being scored by another model, so why trust the scorer?
It is the right question, and it is the same question model validation has always asked. A model is not trusted because it was built carefully. It is trusted because someone independent checked it against known cases. PROBE holds its own judge to that rule. Before the judge is allowed to grade anything, it must pass a reliability check: given two identical answers, it must score them as equal treatment. If the instrument cannot get the obvious case right, none of its other numbers mean anything.
The thresholds are explicit. Groundedness ships with a pass line of 0.85, so an answer that invents facts pulls the score below the bar and the principle fails. A team can argue the number should be higher. No one can argue there is no number. Having that argument openly, before the result, is the purpose of the exercise.
The evidence is the deliverable, not the score
When PROBE finishes, the score is the smallest part of the output. It also returns the worst offending cases in full, the methodology in plain language, and a report saved as JSON and Markdown. That bundle is what an examiner or internal auditor actually wants. It is a decision trail: what was tested, where it failed, the line it was held to, and what happened next.
A KPI that regenerates its own evidence every time it runs is a control that survives an audit. A dashboard number with nothing behind it does not.
The highest risk in a bank is not the chatbot
Most AI fairness work skips an important fact. The highest risk models in a financial firm are usually not chatbots. They are credit decisions, fraud scoring and pricing. Those are the models that draw regulatory attention, because a biased one denies real people credit and owes them an explanation.
PROBE version one measures the behavior of language models. It does not yet score a credit model, but it was designed so that it can. The core contract is one method, evaluate a principle against a target, and the target does not have to be a chatbot. A classical scoring model can implement the same contract with its own metric calculations and plug in without changes to the pipeline, the reporting or the method. "Treat two applicants the same when the only difference is a protected attribute" has the same shape as the name-swap fairness test, run against a score instead of a sentence.
To be precise: that evaluator is a designed and documented extension point, not shipped code in version one. It exists in the design because a measurement framework for AI in finance that could only ever examine chatbots would be measuring the wrong thing.
From policy to control
The gap in most AI governance is not a shortage of principles. It is a shortage of instruments. Every firm can write that its models should be fair and grounded. Far fewer can produce the number, the threshold and the evidence for a named system on a named date.
That is the line between a policy and a control, and it is the line a regulator reads. PROBE is one way to move a principle across it, one measurement at a time.
A live demonstration is at framework.swapniltamse.com. Choose a principle in plain English and watch it score a deliberately flawed sample chatbot, with the evidence.