Before you put an AI agent into production: the complete checklist

Every week someone says they built an AI agent over the weekend and no longer need their operations team.

Before that agent touches a real order, invoice or customer, check that you already have everything on this list. It works for people hiring someone to build an agent and for people building one themselves.

You can tick off the items you already have. It’s saved in this browser only.

01 · 4 items

The process comes before the agent

A workflow mapped end to end by someone who understands the work, not one the agent pieced together from three examples
Documented exceptions, because handling the easy cases is the easy part
A process that already works, because automating a broken process just makes things go wrong faster
Clear operating rules, reviewed and signed off by someone on the operations team

02 · 5 items

When the agent needs to stop

A list of situations where the agent must stop: disputes, legal threats, requests to speak to a person, or any amount above a set limit
A way to hand the case to a person with the full conversation history, so the customer doesn’t have to start over
The ability for a person to step into the middle of a task and hand it back to the agent without losing what was already done
Permission for the agent to say “I don’t know” instead of making something up
A minimum confidence level below which the case goes to a person

03 · 5 items

Answers you can trust

Sources alongside answers, so someone can check where the information came from
Financial answers that match the system of record to the cent
An agent that admits a mistake when challenged, and fixes it
Consistent answers when a customer asks the same question more than once
A sensible way out for questions nobody anticipated

05 · 6 items

Test environments and least privilege

Test versions of every tool, so testing a refund flow doesn’t refund a real person
Test accounts on external services, so a test doesn’t email a real customer
A limited set of tools, because you don’t want the agent discovering permissions it never needed
A separate service account for each agent, with access limited to what it actually does
An agent that reads invoices without permission to pay them
Human approval before the agent moves money, sends formal correspondence or closes an account

06 · 5 items

Execution that survives failure

Operations that can run twice without damage, because retries happen and paying twice is no fun
Timeouts on tool calls, so an unresponsive API doesn’t freeze the whole workflow
Progress saved at every step, so a failure at step nine doesn’t mean starting from scratch
A cap on retries, so a stuck agent doesn’t spend the night trying the same thing
A maximum run time before the case is escalated

07 · 5 items

Code where you can, the model where you must

Regular code handling the steps regular code can verify, leaving the model only what takes judgment
Rule-based extraction for dates, amounts and account numbers, so the model doesn’t rewrite critical details
Structured outputs validated before they move on to another system
Response parsing that doesn’t break when the model decides to add a friendly intro before the JSON
Independent verification that the task was completed, instead of trusting the agent’s own report

08 · 3 items

Agents that use the browser

Browser automation that works within the constraints of your customers’ real portals
A plan B for when a third-party site changes its layout or moves a button
Someone available to approve two-factor authentication when an external portal asks for it

09 · 9 items

Evaluation before and after every change

An evaluation set built from real, recent customer cases, not examples the developer made up
Business experts reviewing results, because most engineers can’t tell whether an insurance denial code is right
Every wrong answer investigated and added to the evaluation set, so the same mistake gets caught next time
Hard test cases nobody is allowed to quietly remove just to improve the score
Evaluations rerun whenever the prompt, the model or the data changes
Regression tests, so fixing one problem doesn’t create another
AI-based grading checked against human judgment from time to time, because the grader makes mistakes too
A comparison between models using your business’s real questions before choosing one
Reasoning settings tested against your actual workload, because paying for more reasoning doesn’t guarantee better results

10 · 6 items

A gradual rollout into operations

A period where the agent works in parallel with the team, without acting on its own
A phase where the agent recommends before it earns the trust to execute
Memory of false alarms already dismissed, so nobody has to reject the same wrong alert every morning
Frequent human review of full customer conversations, not just dashboard averages
Security testing that tries to break the agent before real users do
A baseline measured before launch, so you know what actually improved

11 · 7 items

Models and prompts under control

Pinned model versions, so an unexpected vendor update doesn’t change the agent’s behavior without anyone noticing
Model retirement dates tracked well in advance, not discovered when something stops working
A fallback model that has already been tested, in case the main provider goes down
The ability to switch models without rebuilding the whole workflow
Prompts stored in version control, with changes reviewed before they reach production
A log of prompt changes, so you can investigate what happened instead of hearing that someone “tweaked something”
Independent ways to roll back prompts, models and tools

12 · 5 items

Volume and peaks

API rate limits accounted for before the Monday morning rush
Real queues, because hundreds of cases arriving at once need to be processed reliably
Concurrency limits, so parallel tasks don’t all hammer the same service at once
An emergency stop button that someone outside engineering knows how to use
A gradual release, starting with a few customers rather than everyone at once

13 · 7 items

Costs

Cost per case measured before launch, not discovered on the first monthly bill
A monthly operating budget that was actually approved
Daily spending limits that stop execution when needed, instead of just sending an alert
A token budget per task, because one unusually large document can make a simple case surprisingly expensive
Caching for the information the agent looks up again and again
Cheaper models for simple tasks and more capable models where they make a measurable difference
A cost estimate before running a year of historical cases through the system

14 · 9 items

Security against attacks

Protection against malicious instructions hidden in the emails, PDFs and websites the agent reads
A strict separation between external content and the instructions the agent is allowed to follow
Restrictions on where the agent can send data, so it doesn’t send customer records to an address it found in a document
Isolated environments, with no exposed credentials, for any code the agent runs
Credentials stored and used by secure tools, instead of shown directly to the model
Security review of MCP servers and plugins, because even a tool’s description can influence how the agent behaves
A response plan for when someone manages to trick the agent into trying to expose sensitive information
Logs of every tool call, who started it and what information or systems it touched
Safety filters that stop truly dangerous requests without blocking legitimate work in sensitive areas

16 · 9 items

Voice agents

Secure payment links, instead of asking customers to read their card details out to a voice agent
Response times fast enough for the conversation to sound natural
Interruption handling, because real people don’t wait for the robot to finish every sentence
Voicemail detection, so account information isn’t read out to an answering machine
Call transfers that don’t drop the customer
Correct pronunciation of names, addresses and technical terms
Testing with different accents, background noise and bad connections
Replies in another language when the customer speaks another language
The right tone for customers who are angry, grieving or going through financial hardship

17 · 7 items

Monitoring in production

Detailed traces of the steps, tool calls and token usage of every run
Transcripts searchable by customer, case and outcome
Dashboards that track accuracy, escalation rate, cost per case and uptime together
Alerts when the escalation rate suddenly spikes, because that’s the first sign something broke
Alerts when the escalation rate unexpectedly drops to zero, because silence is a warning sign too
Monitoring for performance drift as cases and business conditions evolve
A feedback channel that takes real user problems to whoever is responsible for fixing them

18 · 6 items

The human side of every handoff

Human review queues with enough capacity to handle escalations
Clear response times for the human side of every handoff
One reliable agent before you start connecting five, because every handoff between agents is another chance to lose context
A named owner for every agent
A playbook for when the agent suddenly starts escalating every case
Documentation explaining why the important rules exist, so nobody removes a critical safeguard six months from now

At the end of the day

And after building all of that, you’ll still need a good number of the people you were planning to replace:

The operations team, to explain how the work really happens, with every exception
The operations team again, to build evaluation cases from real customer work
An operations leader, to decide which cases the agent must never touch
Experienced people, to take over when the agent gets stuck in the middle of serving a customer
The operations team, to review real conversations and find the problems dashboards didn’t show
A backend engineer, to connect the agent to your systems without giving it unrestricted access
Security specialists, to make sure sensitive information stays where it belongs
Compliance specialists who can explain to an auditor what the agent did and why
An architect who understands the real cost of running each case
Quality engineers who notice when accuracy drops after a model or prompt update
An engineer who knows the infrastructure well enough to fix it when everything starts failing at 3 a.m.
Someone senior enough to tell the CEO that the second agent can wait until the first one works properly
And someone who remembers why a particular refund was banned long ago, long after everyone else has forgotten

Good luck. 🫡

← All articles

Before you put an AI agent into production: the complete checklist

The process comes before the agent

When the agent needs to stop

Answers you can trust

Systems, integrations and data

Test environments and least privilege

Execution that survives failure

Code where you can, the model where you must

Agents that use the browser

Evaluation before and after every change

A gradual rollout into operations

Models and prompts under control

Volume and peaks

Costs

Security against attacks

Privacy, compliance and audit

Voice agents

Monitoring in production

The human side of every handoff

Business, team and ownership

And after building all of that, you’ll still need a good number of the people you were planning to replace:

Keep reading

AI agent observability: what to log to understand every decision

The AI proof of concept that never leaves the pilot: why it happens and how to reach production

The forward deployed AI engineer: who builds the agent inside your operations

Want to put an agent into production the right way?