At a risk forum in New York in September, one speaker made a short point that matters for every bank deploying agents: large language models are poor at math.
He made it while explaining why the dashboard era is ending. Agents will take over much of the analysis people used to export to spreadsheets, and none of it works in a regulated firm if the model does the arithmetic. His remedy had three parts. Expose the reports a bank already trusts to the agent as tools. Put the expert's way of working into a skill. When the agent finds a gap, improve the skill, not the model.
The author built a working demonstration of that design.
A fictional bank, a real workflow
The demonstration is live at letterreview.swapniltamse.com. Everything in it is synthetic: a fictional Northwind Bank, a supervisory letter about its use of AI tools in operations, and the bank's issue register. The letter has one Matter Requiring Immediate Attention, on automated agents in payments that can initiate and approve payments below $250,000 on their own, and two Matters Requiring Attention, on an incomplete AI tool inventory and on monitoring of a vendor credit model that stopped in March.
The review has three parts.
The skill is a plain text file that sets out how the regulatory affairs lead reviews a letter. Classify each finding exactly as the letter labels it. Quote the obligation word for word. Find the related items in the issue register. Name one accountable owner. The first rule in the file comes straight from the forum: never compute a date, a count or a number of days directly.
The tools are four read-only functions behind an MCP server: get the letter, extract its deadlines, run the open-findings report, and search issue history by topic. All date arithmetic lives in code, with unit tests. If the agent needs to know that the written response is due in 16 days, a tool tells it. If that number is ever wrong, it is a software defect, found once and fixed once.
The harness is whatever agent loads the skill and calls the tools. For this run it was Claude in Claude Code. Any harness that supports skills and MCP could load the same folder.
Run 1 found a gap
The first run followed the skill and got the basics right. The written response was due 2026-10-15. The immediate attention item was due 2026-11-15, in 47 days. A related self-identified issue, agent service accounts not reviewed quarterly, was already 60 days overdue, which is exactly the kind of detail a supervisor notices.
It then met a case the skill did not cover. Two of the findings matched issues from the prior year's letter that the register showed as closed. The vendor model monitoring issue had been closed 215 days before the new letter, and the letter said monitoring had not been performed since March, about a month after that closure.
In a bank, a finding that returns after closure tells the supervisor the fix did not hold, so it goes to the board.
The skill had no rule for repeat findings, so the agent did not invent one. It wrote a section titled "Skill gap detected," described what it found, and proposed an amendment for the skill owner.
The fix is a change anyone can read
The amendment added one step. For each finding, look for a closed issue from a prior supervisory letter. If it covers the same obligation, label the finding Repeat and cite the prior issue, its closure date and the days since closure. If the scope is related but different, label it Possible repeat and ask a person to confirm. For every Repeat, add a required question: why did the earlier closure not hold, and who validated it?
The skill owner approved it, and nothing was retrained. The improvement is a few lines in a file, with a named owner and a date. That is how banks already manage procedures for people.
Run 2: same numbers, better memo
The second run made the same six tool calls in the same order, and the tool results were identical to the first run, byte for byte. The demonstration page checks that in the browser.
The memo was different. The summary now leads with the payments issue and the repeat finding. The vendor model finding is labeled Repeat. The inventory finding is labeled Possible repeat, because the prior issue covered models and the new one covers all AI tools. The open questions start with the one a regulatory affairs lead would ask first: why did the February closure not hold?
Why this design fits regulated work
Another panelist made the point that matters most. No one can supervise an agent that can start 10,000 more agents, so responsibility has to sit with the institution, not with an individual reviewer checking every output.
This design is one practical way to do that. The institution owns the method, in a skill it can read, version and approve. It owns the numbers, in tools under the same change control as any report. The agent drafts and a person signs. Every memo in the demonstration ends with "Prepared by an agent for human review. Not approved."
It also fits current guidance. The model risk guidance the US banking agencies issued in April, SR 26-2, leaves generative and agentic AI out of its scope in footnote 3 and leaves those controls to each bank's own risk management. Owned skills and tested tools are something a model risk team can actually review.
What the demonstration does not prove
It is a recorded run on fictional data, not a live service. The memos were written by Claude following the skill, using only the recorded tool results, and every number in them traces back to a tool response that can be opened on the page. The topic tags and the repeat rule are simplified. A real register needs a controlled taxonomy, and a skill can be wrong, which is exactly why it needs an owner.
The trace, the skill change and both memos are on the demonstration page.