The eval you skip is why your AI ships broken

Almost every AI demo I have sat through looked finished. The model answered the question, the screen looked clean, and everyone in the room nodded because it worked in front of them, right then, on that one input. That is the whole problem with treating evaluation as something you bolt on at the end, right before launch, like a compliance checkbox someone has to tick. By the time you are running an eval pass three days before ship day, you are not steering anything anymore. You are just finding out how wrong you already are, too late to do much about it.

I learned this the expensive way while shipping AI to 23 million users at a fintech I used to run product for. A wrong answer there is not an awkward screenshot someone posts and forgets. It is money moving, or not moving, based on something the model said with total confidence. So the eval could not live at the end of a sprint. It had to sit right next to the build, every single day, from day one, or we were not really building anything we could trust.

Build the eval set before you build the feature

Most teams write the eval last because that is how QA has worked for twenty years. Ship the feature, then test it, then fix whatever broke. AI does not behave like normal software though. It does not fail loud. It fails confident. A model will hand you a wrong number in the exact same tone as a right one, and a demo audience has no way to tell the difference unless someone specifically went looking for it beforehand.

What actually works is sitting down before you write much of the product and building fifty to a hundred real cases, the ugly ones especially, the edge cases a demo would never show anyone. You decide upfront what a correct answer even looks like. Then every change, to the prompt, the retrieval, the model itself, runs against that same set before it goes anywhere near a user. It is a bit tedious, honestly. Nobody enjoys building a test harness before the fun part starts. But that tedium is where you actually learn what your system does, instead of what you hoped it does.

Where the guardrails actually belong

A guardrail that only kicks in after something has already gone wrong in production is really just a postmortem generator wearing a different name. The guardrails I trust are the ones built alongside the feature, checking every output before a user ever sees it: does this number reconcile, and is the model actually sure or just guessing with confidence because guessing is what it defaults to.

Basically, the model needs permission to say it does not know, and someone has to build that permission into the system on purpose. It will not happen by itself. Left alone, most models will always answer, because answering is the whole job they were optimized for, and that instinct is exactly what gets expensive once you are past a few million users and a wrong answer stops being a rounding error.

I will admit I did not always believe this. Early on I thought a good enough model plus a careful prompt was most of the job, and the eval was paperwork you did to satisfy someone in compliance. It took watching a small, plausible-sounding error slip through a demo and land in front of a real user to change my mind about that.

What this actually costs you

Building the eval first slows down the first couple of weeks of any project, no question. You will feel behind while some competitor with less discipline demos something flashier and gets applause for it. I have sat in that exact meeting and felt the pull to skip the eval and just ship the version that looks good on a screen. What I have not done, not once, is actually skip it. Because I have also sat in the meeting after launch where someone asks why the model told a paying user the wrong thing with complete confidence, and there is no good answer waiting for you in that room.

The eval is not the part of the project that proves you are done. It is the part that tells you, honestly, whether you are still wrong, and by how much, while there is still time to fix it.

I still catch myself wanting to skip it when a deadline gets tight. Old habits.