Shipping Agentic AI: From Demo to Production

Agent demos are deceptively easy. A model, three tools, a happy-path task, and it looks like magic in a screen recording. Then you put it in front of real users with real data and the gap between demo and product turns out to be a quarter of work that has almost nothing to do with prompting.

The reason is that a demo optimises for the best case and a product has to survive the worst one. Below is where the time genuinely goes, in the order it usually bites.

Non-determinism breaks your definition of done

The same input can produce a different trajectory on each run. That invalidates the testing model most teams bring with them: you cannot assert on exact output, and a test that passes four times out of five is not a passing test.

What works is asserting on properties rather than strings. Did it call the tool it needed? Did it avoid the destructive one? Is the returned object schema-valid? Did it stay inside budget? Those hold across runs. 'Does the prose match this fixture' does not, and maintaining that fixture will consume your week.

Evals are the test suite, and you own them

Public benchmarks tell you roughly which model is stronger in general. They tell you nothing about whether it handles your customers' malformed CSVs. The only thing that answers that is a set of cases drawn from your own traffic, and building it is unglamorous, mandatory work.

• Harvest real failures continuously, so every production complaint becomes a permanent case, so the same bug cannot ship twice.
• Keep a handful of golden paths that must never regress, and run them on every prompt, tool or model change.
• Include adversarial and empty inputs deliberately; that is where confident nonsense lives.
• Automate the run and put it in CI. An eval suite you execute by hand is an eval suite you stop executing by week three.

Thirty well-chosen cases from your own domain beat any leaderboard for the decision you actually face, which is whether to ship this change on Thursday.

Design for the wrong answer

Your agent will be confidently wrong in production. That is a property of the technology at current capability, not a bug you will prompt away, so the product has to stay safe when it happens. This is where the reversibility boundary earns its keep: reads and drafts can be autonomous, because a wrong draft costs a glance, while sends, deletes and payments need a human in the path.

A pattern I reach for constantly is propose-then-confirm. The agent does the full job and presents the result as a reviewable action rather than executing it. Users get nearly all the leverage, and a mistake costs a rejected suggestion instead of an apology email to a customer.

Latency is a UX problem, not an infra one

A multi-step agent takes tens of seconds and no amount of infrastructure changes that, because you are waiting on serial inference. What you can change is whether the wait is legible. Streaming the reasoning, naming the current step, and showing tool calls as they happen turns dead air into visible progress. The same thirty seconds reads as broken in one design and as working in another.

Cost is architectural

Per-call pricing looks trivial in development, where you are the only user running a handful of requests. At production volume, with multi-step loops and context that grows each turn, it becomes one of your larger line items, and it is driven by choices you make early.

• Route by difficulty: most steps do not need your most expensive model, and classification or extraction rarely does.
• Cache the stable prefix of your context; on a long system prompt this is a large, boring win.
• Cap context growth explicitly instead of letting transcripts accumulate until the window forces your hand.
• Attribute spend per feature in your traces, or you will not know which surface to optimise.

The demo proves it can work once. Production is the claim that it works on a bad Tuesday, on someone else's data, without supervision.

None of this argues against building agents. The leverage is real, and I would not go back. It argues for budgeting the work honestly. The model gets you the demo. Evals, permissions, observability and cost discipline get you the product.