LLMs in production, without fear

Integrating an LLM is easy. Keeping it in production is not. The problem is not the model: it is the missing contract.

The pattern we use

1. Schema first: every model output is validated against a typed schema before touching the system.
2. Continuous evaluation: a dataset of real cases runs on every release; if the metric drops, the deploy is blocked.
3. Explicit confidence: the model returns its confidence level; below the threshold, it escalates to a human.

Costs under control

We cache identical responses, degrade to cheap models for easy tasks, and audit every billed token. The result: stable cost per request even at 10x traffic.

LLMs are not magic; they are just another dependency. And like any dependency, they need types, tests, and observability.