Migrate to a New Model Without Breaking Production

A stronger model can still break your agent, because your agent is fitted to the old one. Prompts that worked partly because the previous model over-followed instructions will slip. Formats that parsed reliably may gain a preamble. None of that shows up in a benchmark comparison.

So treat a model change as a change to your product rather than a dependency bump. It alters behaviour across every path at once, which is a category of change you would normally stage carefully and watch closely.

The corollary is that you cannot evaluate a migration by trying a few prompts by hand. Impressions from a handful of inputs are exactly the evidence least able to detect a regression in the tail, and the tail is where your incidents live.

Tool calling first. How eagerly tools get called, how arguments get formatted, whether calls come in parallel, and how the model behaves when a tool errors. This is where most migrations break, and the breakage looks like an application bug rather than a model change.

Then output discipline: structured output that parsed every time may acquire commentary around it, and enums may gain a plausible sixth value. Then refusals, which shift in both directions and can quietly start declining legitimate work in a regulated domain.

Then the numbers. Latency per call, tokens produced for the same task, and therefore cost per completed task. A model with a cheaper headline rate can be more expensive in practice if it takes more steps or writes more words to reach the same place.