How to evaluate AI agents for customer service: Testing agents pre-production

Pre-production testing is the launch gate between an impressive pilot and real customer traffic.

Executives saw the agent resolve a clean call and sound natural doing it, and now leadership wants a launch date. Your contact center still has peak-hour queues and compliance scripts, plus authentication steps and customers who interrupt because they are already frustrated. You have to decide how the agent should respond when a caller changes direction or asks a question the script did not anticipate.

A weak gate turns call volume into uncontrolled experimentation. The launch decision requires evidence strong enough to demonstrate that the agent can act safely before customers become the test environment.

What is pre-production testing?

Pre-production testing is the structured evaluation phase between a working AI agent pilot and the introduction of live customer traffic. It validates behavior across representative scenarios, edge cases, adversarial inputs, and compliance requirements before real callers reach the system.

Unlike a demo, which proves the agent can handle a cooperative user on a clean path, pre-production testing measures how the agent performs when customers interrupt, change direction, or push against policy. It captures reasoning trajectories, tool calls, and handoff decisions across repeated runs, not just final answers.

The output is an evidence packet that lets business, technical, compliance, and risk owners judge whether the agent is safe to launch, with thresholds calibrated to the risk of each individual use case.

Why pre-production testing decides whether a pilot ever ships

Most GenAI pilots produce impressive demos and never survive contact with production traffic. The reason, most often, is the absence of evidence that executives, compliance owners, and risk teams must accept the residual risk of deploying an autonomous system in front of real customers. Pre-production testing is the discipline that closes that gap, and the pressure to close it is only getting sharper.

Several forces make pre-production testing the deciding factor between a stalled pilot and a live agent:

• Executive pressure without proof of value: Only 11% of customer service leaders say GenAI investment outcomes have met their primary business objective, even as executive pressure to deploy keeps climbing. Without evidence of launch readiness, most projects stall in the same place.

• Demos hide failure modes: A pilot shows the agent succeeding on a cooperative path. Pre-production testing surfaces how it behaves when a caller interrupts, switches intent, or asks for something the script never anticipated.

• Governance requires an evidence packet: Business, technical, compliance, and risk owners each need artifacts they can defend before signing off, and none of those artifacts come out of a demo environment.

• Cancellation risk is real: Loose gates turn call volume into uncontrolled experimentation, and the resulting incidents become the reason projects get pulled shortly after go-live.

The pilot proves the agent can work. Pre-production testing proves it can be trusted to work at scale under real customer behavior, and that shift is what turns AI agent lifecycle management into a defensible launch decision.

What changes when you test agents instead of scripts

Testing an AI agent requires a different discipline than testing a scripted Interactive Voice Response (IVR) flow. Scripted flows repeat fixed steps, making one-time validation feasible. AI agents evaluate context and decide when to act, so testing must measure both the reasoning path and the final answer.

Agent testing covers whether the agent made the right decision and carried the required context across repeated runs; following the IVR menu tree is no longer enough.

The test plan has to measure behaviors that scripted QA misses:

• Probabilistic behavior: Repeated identical inputs can produce variable responses, so a single passing run proves little; No Jitter says that agentic systems require testing the range of behavior rather than a single instance.

• Multi-turn context: Real conversations span many exchanges where context carries forward, and validation has to catch when the agent loses the thread by turn seven.

• Continuous validation: Teams cannot rely on one-time validation for a system whose behavior can change over time, so pre-production establishes the reference point.

The test record needs the transcript and tool-call record, including handoff decisions and state changes, because the launch decision depends on whether the agent followed the required path.

Defining "correct" also changes. For a simple application, ground truth means the expected answer. For an agent, ground truth must include expected tool calls and reasoning steps. Required paths include identity verification before refund processing.

Customer satisfaction does not prove that the agent completed the required back-office steps. A refund agent might issue the refund correctly but fail to log the transaction in the compliance database. Single-turn checks score that as success because the customer got the money. A trajectory check catches the missing write, which is exactly the failure that surfaces in an audit six months later.

Probabilistic behavior and multi-turn context make trajectory failures harder to test in the phone channel. A voice agent must recognize intent through accents and background noise while remaining fast and consistent throughout the call. Building agents on conversational AI best practices gets you to a working demo. Deciding what "acceptable" looks like requires risk-calibrated thresholds before go-live.

Setting risk-calibrated thresholds before go-live

There is no universal "good enough" score, because the right readiness threshold depends on the risk of the specific use case. Business leaders set the readiness thresholds for each flow.

Start from the cost of being wrong. A product frequently asked questions (FAQ) agent can tolerate a higher error margin than a billing-dispute or claims agent, because a wrong answer about store hours and a wrong answer about a denied claim differ by an order of magnitude in terms of consequences. Holding both to the same benchmark score makes the FAQ threshold too strict or the claims threshold too loose.

Each threshold should define whether the flow ships, changes, or routes to a human.

Containment and resolution rate

High containment can mask unsafe resolutions when teams prioritize fewer handoffs. The readiness gate should measure the share of contacts the agent resolves without a human handoff and set the threshold at which resolved calls remain safe. That outcome gives leaders actionable evidence without suppressing the escalation customers still need.

Hallucination ceiling

Unsupported answers pose a launch risk because a single invented response can lead to financial or regulatory exposure. The hallucination ceiling sets the maximum acceptable rate of fabricated responses, with lower ceilings for high-stakes flows. Passing the gate means the agent stays inside approved knowledge before customers hear it.

Escalation quality

Escalation failure turns automation into a longer queue when the agent sends a customer to the wrong team or drops context. The gate should measure whether the agent hands off to the right human, with enough context for the human agent to continue. A passing result protects customer effort and prevents automation from shifting work downstream.

Authentication accuracy

Identity-sensitive flows fail if the agent acts before it verifies who is speaking. Authentication accuracy measures how reliably the agent completes that check before taking action. Passing the gate protects every downstream action that depends on verified identity.

In voice, authentication accuracy and intent recognition are critical thresholds, because a misrecognized identity or intent cascades through the entire call and every action that follows. Schwäbisch Hall shows what governed thresholds look like at real volume: their voice AI agent handled 500,000 calls in six months with an authentication rate above 80% and 98% intent recognition accuracy across 16 live use cases. That production result shows why thresholds must protect sustained accuracy under real load across multiple use cases.

Once numeric gates exist, the next question is whether policy holds when a user pressures the agent to break them.

Building red-team and compliance testing into the gate

Functional QA validates expected behavior with cooperative users; the production gate also has to test prompt pressure, required disclosures, unsafe requests, and regulated scenarios that rarely appear in a clean demo.

The test set should include attacks that pressure the agent to ignore policy or complete regulated actions without authorization.

• Prompt injection and jailbreaks: Attempts to override the agent's instructions or elicit behavior it should refuse, including inputs crafted to cause it to ignore its guardrails.

• Policy and disclosure violations: Cases in which the agent omits a required disclosure, misstates a policy, or gives advice that the policy does not permit.

• Unsafe decline handling: Scenarios the agent should refuse outright, tested to confirm it declines cleanly instead of improvising a harmful or non-compliant response.

• Regulated-industry breaches: Failures specific to the vertical, where the law draws the pass/fail line.

A passing result means the agent refuses unsafe requests and escalates with the required disclosure when the user pushes beyond the happy path.

Pass/fail criteria vary by industry. Disclosure and fair-treatment rules vary by sector, so the same agent behavior can pass in one industry and fail in another. New regulation raises compliance expectations. The AI Act requires the development and use of AI systems to ensure appropriate traceability and explainability, and its diversity and non-discrimination provisions require that systems avoid discriminatory impacts and unfair biases.

Compliance testing before launch must demonstrate that the agent answered correctly and that you can explain why it made the choice it did. Designing guardrails for AI agents is what makes those tests something the agent can actually pass.

In voice, red-teaming must include adversarial inputs delivered by phone because an attacker on the phone operates differently than one in a chat window. The agent has to deliver required disclosures audibly in the call flow, at the moment the customer needs them. A disclosure that exists only in the log is a disclosure the regulator will treat as absent.

Compliance and red-team results matter only when leaders with the right authority can accept or reject the residual risk. Go-live requires governance sign-off from the business, technical, compliance, and financial owners named in the sign-off sequence.

Building the sign-off sequence and drift baseline

Go-live is a sequenced, cross-functional decision in which different stakeholders hold veto authority at different gates. When launch plans under-specify sign-off authority, teams discover too late that no one knows who can block go-live and why.

If sign-off happens out of sequence, the launch meeting can approve a compliant agent that still strands customers in the wrong queue, or a reliable integration that fails a disclosure requirement. The sign-off packet should give each veto owner the evidence they control: CX operations reviews transcripts and escalation outcomes; engineering reviews integration and load results; legal and compliance review disclosure evidence; and risk and finance review residual-risk economics.

• CX operations: They sign off on customer-experience outcomes and escalation quality. Their gate confirms the agent meets contact center standards when real customers get impatient or confused.

• Information technology (IT) and engineering: They sign off on integration and system reliability, including load performance and dependent system connections that cannot fail mid-call.

• Legal and compliance: They sign off on required disclosures and regulatory obligations that govern the industry, so a polished resolution does not create exposure after launch.

• Risk and finance: Risk and finance approve threshold acceptance and the business-value case. They decide whether the residual risk matches the expected return.

During testing, capture the model drift baselines so post-launch monitoring has something to compare against. Model drift is harder to spot than a software crash because a wrong action may not trigger a clear error. An action completed correctly ten times can fail unpredictably on the eleventh, and the symptom surfaces only after the agent has already acted. Without a baseline, drift becomes something you discover through customer complaints rather than your own instruments.

For phone deployments, the baseline includes intent recognition accuracy and authentication rates measured under simulated peak call volume, so the organization can catch drift early in the channel that carries the most sensitive interactions.

Related: A Bayesian framework for A/B testing AI agents

Turn AI agent evaluation into governed launch evidence

Pre-production testing is the operating muscle that determines whether every future release reaches customers safely, and it is what separates agents that impress in a demo from those who scale.

A launch-ready team knows exactly which failures route to design, engineering, or a risk owner before another customer hears the agent, and it carries drift baselines forward so problems surface on instruments instead of in complaint queues. Without that discipline, the first live conversation becomes the test environment, and the cost of being wrong lands on customers first.

Parloa's AI Agent Management Platform maintains that rhythm across Design, Test, Scale, and Optimize, with security as a governance layer, support for 140+ languages, and evidence connected to enterprise systems. Support for ISO 27001:2022, ISO 17422:2020, SOC 2 Type I & II, PCI DSS, HIPAA, GDPR, and DORA gives compliance teams reusable launch evidence.

Book a demo to see AI agents that hold up in production and give your first live conversation the best chance of solving the customer's problem without making them repeat themselves.

FAQs about pre-launch AI agent evaluation

What is pre-production testing for AI agents?

It is the evaluation phase before an AI agent handles real customers, where you validate behavior against representative scenarios, edge cases, and compliance rules. Unlike a demo, it measures whether the agent holds up under adversarial and multi-turn conditions across more than a single cooperative exchange.

How long does pre-production testing take?

It depends on the use case's risk and the number of scenarios, but a structured cycle typically spans several weeks and covers validation, simulation, stress testing, and a monitored canary release. The highest-risk flows should stay in test until policy failures and authentication misses have clear owners. Higher-risk use cases warrant more time on threshold calibration and compliance testing.

What is a good hallucination threshold for a customer service AI agent?

There is no universal number. Use-case risk determines the acceptable rate. A product FAQ agent tolerates a larger error margin than a billing-dispute or claims agent, where a wrong answer carries regulatory or financial costs.

Who signs off on launching an AI agent?

Go-live is a sequenced cross-functional decision. CX operations, IT and engineering, Legal and compliance, and Risk and finance each hold authority over different gates.