Why AI prototypes fail when they reach production

There is a particular kind of disappointment in AI projects: the prototype was genuinely impressive, everyone saw it work, and then the production version is unreliable in a way nobody can quite pin down. The gap is not bad luck. It's a predictable set of differences between the two environments, and most of them are knowable in advance.

You tested on examples you chose

This is the big one. When you build a prototype, you naturally reach for inputs you understand. They're representative in your head. They are almost never representative in distribution — they're cleaner, more typical, and more centred than what real users send.

Real input includes the photo taken in bad light at an angle, the document that was scanned upside down, the question asked in three languages at once, the field left empty because the user didn't understand it. A prototype tested on twenty hand-picked examples tells you almost nothing about the tail, and the tail is where production lives.

The fix is unglamorous: build a fixed evaluation set from real data, including the ugly cases, before you tune anything. Then every change is measured against it rather than against the impression it left in a demo.

You have no idea what your failure rate is

Ask a team what percentage of their AI feature's outputs are wrong and you'll often get a shrug, or a number derived from vibes. That's fine in a prototype. In production it means you cannot tell whether a change made things better, and you cannot tell when something has quietly regressed.

An evaluation harness does not need to be sophisticated to be transformative. A fixed set of inputs, expected outputs, a scoring function, and a number that appears in CI. Once the number exists, arguments about whether a prompt change helped stop being arguments.

A number in CI turns prompt engineering from an opinion into an experiment.

Nothing was designed to fail

Prototypes assume the happy path because the happy path is the point. Production systems spend most of their design budget on everything else: the provider returning a 503, the response that isn't valid JSON, the request that takes 40 seconds, the rate limit hit at 9am on Monday.

• What happens when the model provider is down? A fallback chain to a second provider degrades quality; no fallback takes the feature offline.
• What happens when the output doesn't parse? Validate against a schema and retry with the error, rather than passing malformed data downstream.
• What happens when it's slow? Stream progress, or the user assumes it's broken and refreshes — doubling your load at the worst moment.
• What happens when it's wrong? There should be a route for the user to correct it, and that correction should be visible to you.

In InstaBrain, semantic search falls back to full-text search when the vector service is unreachable. The user gets slightly less relevant results instead of an error page, and usually doesn't notice. That's the shape most of these decisions should take: degrade, don't fail.

Cost was never in the design

At prototype scale, model costs are a rounding error and nobody models them. At production scale they're a line item that can quietly exceed your infrastructure budget. The time to notice is before launch, not on the first invoice.

Most of the fix is structural rather than clever: use the expensive model only for the step that needs it, cache aggressively on inputs that repeat, and check a cheap local source before calling out. Splitting a pipeline into stages with separately-chosen models is usually worth more than any single optimisation.

There is no way to see what's happening

Prototypes are debugged by watching them run. Production systems are debugged by reading what they recorded. If you aren't logging inputs, outputs, latencies, model versions and failures, every production bug becomes an archaeology project.

Log the pipeline stage, the model, the tokens, the latency and the outcome. When someone reports that 'the AI got it wrong yesterday', you want to be able to find that exact request rather than ask them to reproduce it.

The pattern

Every item on this list is a variant of the same thing: a prototype answers 'can this work?' and a production system answers 'what happens when it doesn't?'. Both are legitimate pieces of work. The mistake is assuming the second is a small increment on the first, when it is usually the larger half of the project.