Shipping GenAI that survives production

In enterprise GenAI, most projects never reach production. The models work fine, but the systems around them can’t be operated.

I’ve spent the last two years architecting an enterprise GenAI platform for a global industrial company, a multi-agent system orchestrating 10+ LLMs through a unified chat interface with RAG, tool-use, and streaming. In the process I got tired of seeing competent teams ship demos that never graduated. The rest of this post describes the pattern I see and what to do about it.

The prototype-to-production cliff

The typical GenAI project looks great in a Jupyter notebook. It has a LangChain pipeline wired to GPT-4o, a vector store and a prompt that gets the right answer most of the time, and the team ships it.

Then it reaches production and the team has to answer questions like these:

• Who watches the token spend?
• What happens when the model is down?
• How do you audit which documents fed which answer?
• How do non-engineers change the prompt?
• What’s the rollback if the new prompt regresses?

These questions come up every day, and they’re the reason your prototype dies on its first contact with ops.

The five things I wish more teams did

1. Treat the agent loop like a service

A ReAct agent is a state machine that can loop indefinitely. Give it timeouts, iteration caps and structured error boundaries between tool calls, and log every decision so you can replay failures.

2. Stream end-to-end

Three seconds of perceived latency is often enough to make an AI feature feel broken. Stream the response from the model through your orchestrator to the UI without buffering or serializing it at any layer.

3. Build in observability from the start

Use OpenTelemetry from day one and trace each LLM call, tool invocation and RAG retrieval. When something goes wrong in production, you need to see the exact prompt, the tokens returned and the chain of reasoning. Application Insights, Langfuse and Helicone all work for this, so pick one.

4. Identity per service

Give the API, agent, RAG worker and admin panel separate service accounts, and grant each one only what it needs. Enterprise AI will eventually be a target, and when you get breached, the blast radius stops at the service boundary.

5. Promote verified images to production

Your production container registry should never receive a direct build from a pipeline. Build to a staging registry, run E2E tests there and promote the verified image. Letting CI build prod:latest directly costs you reliability and doesn’t make you any faster.

Closing thoughts

None of this is new, and backend engineers have known it for twenty years. The AI ecosystem currently rewards shipping demos, so if you want enterprise AI to work in production, you have to be willing to do unfashionable work.

---

I take on architecture work for companies shipping enterprise GenAI, including multi-agent systems, RAG pipelines and CI/CD for AI services.