How to choose an AI model for production (and why it’s almost always several)

One agent does many different tasks

Think of an agent that handles orders arriving by email. It has to work out whether the message is an order, a question or a complaint. Then extract data from an attachment: customer, products, quantities. Next, check stock and pricing. Finally, write a clear reply to the customer and, in ambiguous cases, decide whether to call in a person.

Each of these steps has different demands. Classifying the message is simple and happens many times a day. Extracting data from a messy PDF takes precision. Deciding an ambiguous case takes more careful reasoning. Using the most expensive, most capable model for all of them is wasteful. Using the cheapest for all of them is risky.

The criteria that matter

For each of the agent’s tasks, evaluate candidate models on five criteria:

• Quality on your task. Not on public benchmarks, but with your documents, your messages and your vocabulary.
• Cost per task. Work out how much it costs to process a typical task, given the average size of input and output, and multiply by the expected volume.
• Response time. In real-time customer service, a few seconds make a difference. In an overnight batch, they barely matter.
• Where data is processed. Check the provider’s policy on retention and use of the data you send, and whether processing is available in a region that meets your operation’s requirements.
• Provider stability. Rate limits, outage history, advance notice before retiring versions.

The best model for your agent is the one that gets your task right at the lowest acceptable cost, not the one at the top of the leaderboard.

Test with your data, not generic examples

The only reliable way to choose is to test. And the test only counts if it uses real cases from your operation. A practical approach:

• Set aside 50 to 200 real examples of each task, including hard cases and cases that have caused problems before.
• For each example, write down what the correct or acceptable answer would be. That set becomes your answer key.
• Run each candidate model on the same set, with the same instructions.
• Compare accuracy, cost and speed. Look closely at the errors: a model that rarely gets it wrong but gets it badly wrong can be worse than one that makes more small mistakes.
• Keep that set. You’ll use it every time you consider switching models or adjusting instructions.

This test set is one of the most valuable assets of an agent in production. With it, any change becomes an objective comparison instead of a hunch.

For tasks where there isn’t a single right answer, like writing a message to a customer, agree on simple evaluation criteria with the team: is it correct, is it in the company’s tone, does it include everything the customer needs to know. Have two people grade a sample of the responses without knowing which model produced each one.

Why you need a fallback model

AI providers go down. Sometimes for minutes, sometimes for hours. They also change rate limits, adjust prices and retire versions. If the agent depends on a single model, any of those things stops your operation.

Having a fallback model means that if the main one fails or takes too long, the agent automatically switches to another, ideally from a different provider. The fallback doesn’t need to be as good as the main model at everything. It needs to be good enough to keep the operation running while the problem gets resolved.

A few decisions are worth making ahead of time:

• Which tasks can fall back and which should wait or go to a person.
• How long a wait triggers the switch.
• How you find out the fallback kicked in.
• Whether the fallback has been tested on the same set of examples as the main model.

How to switch without rebuilding everything

New models come out all the time, and some are better or cheaper for tasks you already have. Being able to switch quickly becomes an advantage. For that, the agent has to be designed with it in mind:

• A layer between the agent and the models. The agent asks “classify this message” and an intermediate component decides which model handles it. Switching models becomes a configuration change.
• Instructions kept separate from code. Each task’s instructions are stored with versioning and can be tuned for each model when needed.
• Structured outputs. When the model returns data in a defined format, the rest of the agent doesn’t depend on each provider’s writing style.
• The test set always at hand. Before any switch, you run the tests. If the new model is better or equal, you switch. If not, you stay put.
• Gradual switching. Send a small share of tasks to the new model, compare the results in the logs and ramp up from there.

With that structure, changing models stops being a project and becomes routine maintenance.

Common mistakes when choosing

• Choosing from public leaderboards without testing on your own cases.
• Using the most expensive model for everything “to be safe”, and only discovering the cost on the first bill.
• Writing instructions so tied to one model that any switch means rewriting everything.
• Having no fallback and discovering the dependency on the day the provider goes down.
• Switching models because a new version came out, without running the tests first.

All of these mistakes share the same root: treating model choice as a one-time decision. In production, it’s an ongoing decision, revisited with data.