Shipping AI where a mistake costs money

A chatbot that hallucinates a product recommendation embarrasses you for a day. A model that hallucinates a balance, or approves a transaction it should have flagged, costs someone real money that same afternoon. I learned this shipping AI to more than 23 million users at a US fintech, and the lesson never really left me. Money changes the entire risk calculus of what you’re willing to ship.

Most AI demos are built to look impressive in a room. Nobody in that room is checking whether the model’s confidence matches its actual accuracy, because that check is slow and boring and nobody claps for it. In fintech, the gap between “sounds right” and “is right” is exactly where the incident happens. A support bot that gets a shipping date wrong is annoying. A model that’s confidently wrong about whether a transaction is fraud is a liability with your name on it.

Guardrails are the actual product

I used to think of guardrails as friction, the thing that slows a feature down right before launch. That’s kind of backwards. In fintech, guardrails are the actual product, the part that decides whether a customer trusts you with their money at all. We built verifiers that checked a model’s output against ground truth before it ever reached a user, not after a complaint came in. Basically, if the model couldn’t back a number with a real source, it didn’t get to say the number.

That sounds obvious written down. It wasn’t obvious in practice, because building the verifier is slower and far less fun than building the feature, and there’s always a product manager, sometimes me, asking why we can’t just ship it and watch the dashboards. You can watch your way through a bad shopping recommendation. You can’t watch your way out of a wrong balance shown to someone at 2am, because by then they’ve already acted on it.

Teaching the model to shut up

The single biggest improvement we made wasn’t a bigger model or better prompts. It was training the system to say “I’m not sure, here’s a human” instead of guessing. That meant admitting something uncomfortable up front: the model would be wrong sometimes, and pretending otherwise was more dangerous than the errors themselves. A model that’s wrong 2 percent of the time but flags its own uncertainty is safer than one that’s wrong 1 percent of the time and sounds equally sure every single time.

Most teams skip this part because deferring to a human feels like admitting the AI doesn’t work. That instinct is wrong, and expensive. A fintech feature that knows when to escalate survives a regulator asking hard questions, and it survives the one bad week every product eventually has. We tracked how often the model sounded certain and was wrong, and how long it took to get a real person into the loop when it wasn’t. If confident wrong answers weren’t trending toward zero, the feature didn’t ship, no matter how clean the demo looked a week earlier.

What this actually costs you

None of this is free. It’s slower to build. It means shipping something less flashy than a weekend hackathon could produce, and it means telling a stakeholder the feature isn’t ready when it technically works 9 times out of 10. I’ve had that conversation more than once and it never gets easier, because 90 percent sounds like a great number right up until you remember the other 10 percent is someone’s rent, someone’s loan, someone’s overdraft fee they didn’t deserve.

A wrong answer in fintech doesn’t get forgiven the way a wrong answer in a shopping app does. Nobody screenshots it and laughs about it in a group chat. They call support, or worse, they quietly stop trusting the product and never say why. Confidence you can’t actually back up is a liability, and it comes due at the worst possible time, usually right when you can least afford it.

I have shipped both kinds of features in my career. The honest ones are the only ones I don’t still worry about.