There’s a difference between a working AI model and production-ready AI. Before launch, most teams put their effort into the model itself, adjusting prompts and checking outputs until the results look right. The decisions that matter after launch get less attention: how risky the asset is, what it's allowed to access, who confirms its controls work, and what happens when something goes wrong. Answering those questions before go-live is easier than answering them during an incident while under pressure.
Below are six things to have in place before any AI asset goes live. They apply whether the asset is a simple chatbot or an agent with write access to the ERP. Each one should be confirmed before the asset reaches production.
Traditional software tests check for an exact answer. AI models can give different outputs for the same input, so exact-match tests either fail all the time or get loosened until they don't mean much. Acceptance criteria for AI should describe the range of behavior that's acceptable. For an invoice extraction tool, one criterion might say the model leaves the vendor field blank when the name is unreadable instead of guessing. Another might set a minimum accuracy on a fixed set of sample invoices.
Write these before testing starts. Criteria written afterward tend to describe whatever the model already does.
Every AI asset needs a risk tier before launch, because the tier decides how much of the rest of this list applies. It sets the testing bar, decides if the rollback has to be rehearsed, and determines how many layers of kill switch the asset gets.
Companies without tiers usually apply one standard to everything. If that standard is heavy, a tool that summarizes meeting notes waits weeks for a review it never needed. If it's light, an agent that posts journal entries goes live with the same oversight as the summarizer. Both can happen in the same company.
Two or three tiers are usually enough to start. Base them on what the asset can touch, such as customer data, financial records, or systems it can write to. Each tier assignment also needs an owner. Someone in IT governance or risk should make the call and revisit it whenever the asset's scope changes.
Security review for AI assets should run automatically every time an asset is promoted, whether from development to test or from test to production. Manual reviews work if a company has three AI tools. Once it has 30, reviews get skipped when a deadline is close. The rushed builds that get skipped are typically the ones that need review most.
Build the security scan into the pipeline so promotion fails when the scan fails. At a minimum, it should check for exposure to prompt injection, data access beyond what the asset needs, and credentials or API keys stored in prompts or configuration files.
Any AI asset that touches a financially controlled process falls under the same SOX requirements as the process itself. If an agent drafts reconciliations or a model flags transactions for review, auditors will expect documented evidence that the controls around it were tested before go-live.
The harder part comes after launch. Changing the model or its configuration can change how the asset behaves, so the control evidence from launch may stop being valid. Repeat control testing after any such change, including model updates a vendor pushes on its own schedule. Teams should know which vendor models their assets rely on and how they'll learn when one changes.
Before launch, the team should be able to answer three questions in writing. What is the exact sequence of steps to revert the asset? Who has authority to approve a rollback? How long does it take from decision to completion?
Rolling back an AI asset involves more than code. A regression can come from a reworded prompt, a new model version, a changed retrieval source or a configuration setting. Each needs a version history and a known-good state to return to. If no one can say what the last good version of the prompt was, the team can't roll back to it.
Then rehearse it. Run the rollback in a test environment and time it. The first attempt shouldn't happen in the middle of an incident while the business waits.
A kill switch lets someone stop an AI asset quickly without deploying new code. While rollback brings back a previous version, a kill switch stops the asset completely so the team can figure out what went wrong.
Every asset should have at least one. The minimum is a flag that turns off that single application. Higher-risk assets need more layers. An agent with write access to financial systems might need one switch that cuts its access to specific integrations and a broader one that pauses every agent sharing the same model or credentials. Size the switch to the level of damage the asset could do if it misbehaved. The risk tier from section 2 should already describe that.
The switch also needs an owner and a documented trigger. People should know who can flip it and when. Test it the same way you test the rollback.
A launch checklist assumes the version checked is the version that stays in production. With traditional software, that's mostly true, but not for AI assets. A vendor can update the model, someone can edit a prompt, or a data source can change, and the asset may start behaving differently. Don’t let those changes skip the review a code release would get. You don’t want an unchecked version running in production. Testing, security scans, and control validation should run again after any of these changes, and the risk tier, rollback plan and kill switch are worth a look to confirm they still fit.
At Trenegy, we help companies build the governance and testing practices they need to put AI into production safely. To chat more about AI deployment, emailinfo@trenegy.com.
Trenegy is a non-traditional consulting firm based in Houston, Texas.
