AINovember 17, 202610 min read

Beyond IVR performance testing: Pre-launch voice AI tests

Joe Huffnagle

VP Solution Engineering & Delivery @Parloa

A caller in a menu-driven interactive voice response (IVR) system (opens in a new tab) can only go where the menu allows: press 2 for billing, say “agent,” wait. A caller talking to an AI agent can say anything, in any order, with a train announcement behind them and a half-finished sentence they correct midway.

The load-test playbook contact centers built for menus answers one question well: does the phone system hold up at volume? An AI agent needs more proof than that. Before go-live, you also need evidence that it understands open-ended requests, stays fast and accurate when hundreds of people call at once, and hands callers to humans with the conversation intact.

Traditional IVR performance testing still matters, because every AI agent runs on telephony that can fail under load. The rest of the work happens before launch, on the agent itself. Post-launch monitoring (opens in a new tab) and continuous improvement belong to a separate discipline.

What is IVR performance testing?

IVR performance testing is the practice of placing simulated calls against an IVR at realistic volume to measure how many calls it can carry at once, how quickly it responds, how good the audio sounds, and how it behaves when overloaded. It matters because a phone system that works for a handful of callers can fail at peak, when most customers call. Best practice is to ramp up traffic gradually (opens in a new tab) and hold it at set levels up to full load.

This testing catches failures a configuration review can miss: one company found during testing that only half its port capacity (opens in a new tab) was working because of a configuration issue.

Before any AI agent work starts, confirm that your last full load test reflects the current carrier, session border controller (SBC), and IVR configuration.

Types of IVR performance testing

Each type of IVR performance testing checks a different failure mode: capacity at expected peak, behavior under overload, sudden surges, and faults that build under sustained load.

Load testing

Load testing runs the IVR at the number of simultaneous calls you expect at your busiest time, to find where performance slips before capacity runs out. For example, a utility that expects its heaviest calling on the first business day after bills go out would test at that day’s volume and watch for calls that fail to connect, busy signals before the lines should be full, and queues that keep growing.

Stress testing

Stress testing pushes the IVR past its rated capacity to find the breaking point and see how it fails. The Session Initiation Protocol (SIP), the standard that sets up internet phone calls, has limited built-in protection: when overloaded, it mainly turns new calls away (opens in a new tab) with a message that the service is unavailable. Failures are not always in the phone layer either, because speech recognition can become less accurate under heavy load even while calls still connect.

The stress question to answer is whether the system rejects excess calls cleanly or fails in ways that affect calls already in progress.

Spike testing

Spike testing sends a sudden surge of calls to the system with no gradual ramp, like a storm, an outage, or a product recall. It shows whether lines and queues sized for average traffic hold when calls arrive all at once. Load testing builds traffic up step by step, so it answers a different question. A phone line can run smoothly for months and still fail the first time a service outage sends many callers in together.

Test at volumes well above your normal level, and include surges that follow predictable triggers such as a marketing campaign or a billing run.

Soak testing

Soak testing holds steady load for hours to find faults that build slowly, such as memory a system uses and never releases. In one problem Cisco documented, a speech-recognition component on a voice gateway stopped working (opens in a new tab) after a few hundred calls, and every later call failed until the gateway was restarted. The logs pointed to a memory problem.

Hold full load for several hours and watch for the system running out of database connections, memory that grows with every call, and limits on how many files or connections can stay open. Then drop the load to zero and check that the system returns to normal with no connections or processes left hanging. Add that recovery check to every soak run; Cisco documented a fault that only a restart cleared.

Metrics and benchmark thresholds to measure

Voice quality and network metrics have published thresholds; AI agent response time and carrier capacity do not have standard ones. The table below lists each metric with its source, population and scope:

Metric

Definition

Benchmark

Source, scope and date

Mean opinion score (MOS)

Mean of opinion scores on a 1–5 scale for a telephone transmission system (ITU-T P.800.1)

R-factor 80 maps to MOS 4.03, “Satisfied”; R 70 maps to MOS 3.60, “Some users dissatisfied”

ITU-T G.107 (opens in a new tab) (06/2015) Table B.1, a provisional planning guide, conversational MOS estimated from the E-model

One-way latency (mouth-to-ear)

Delay from speaker’s mouth to listener’s ear, all sources included

Below 150 ms gives “essentially transparent interactivity” for most applications; 400 ms should not be exceeded for general planning

ITU-T G.114 (opens in a new tab) (05/2003), in force

Jitter

Variation in packet interarrival time, computed per RFC 3550

30 ms or less one-way

Cisco QoS design guidance (opens in a new tab) for voice

Packet loss

Share of packets that fail to arrive

1% or less

Cisco voice QoS guidance

Post-dial delay (PDD)

Time from dialing to ringback

Most carriers treat under 7 seconds as acceptable; business VoIP generally expects under 2 seconds

No standard target exists

Call attempts per second (CAPS/CPS)

Rate of new call setups

No standard value; provider defaults range from 1 CPS per trunk per region (Twilio Elastic SIP Trunking) to 20 CPS per source IP (Telnyx)

Provider documentation from Twilio (opens in a new tab) and Telnyx (opens in a new tab)

AI agent response time

Time between end of user input and return of the result

No standard threshold

Definition from ITU-T F.748.25 (opens in a new tab)

The voice-quality score bands are planning guides and should not be treated as pass or fail lines. Different scoring models can rate the same call differently, so choose thresholds that match the model (opens in a new tab) you measure with.

On the concurrency side, IETF RFC 7502 (opens in a new tab) benchmarks the maximum SIP session establishment rate a device sustains without failures and requires reporting session duration with results. Set your pass thresholds per codec, per carrier and per call path and write the definition next to each number in the test plan.

Performance testing vs functional and regression IVR testing

Performance testing answers whether the system holds capacity; functional and regression testing answer whether it behaves correctly. Industry sources agree on goals and timing, and say much less about who owns each test:

Test type

Goal

Typical timing

Owner

Functional

Confirm every menu path, prompt and dual-tone multi-frequency (DTMF) input reaches the correct destination

Before go-live on a first launch; each flow tested before integration testing

QA analysts or IT staff; vary by business

Regression

Confirm existing call flows, recordings, lookups and routing still work after a change

After every change or release

QA analysts operating automated regression suites

Performance and load

Confirm capacity and response speed under concurrent calls

Ahead of known high-volume events; months before peaks; before go-live on migrations

No clear owner in IVR sources; many leaders believe the technology vendor or cloud provider (opens in a new tab) should own it

Load testing is the test type where sources name no clear owner, and Contact Center Pipeline warns (opens in a new tab) that “CCaaS providers have been known to fail unanticipated load tests.” Assign a named owner to each test type in your launch plan, including load testing, and do not assume the platform provider runs it.

Why menu testing falls short for voice AI agents

Testers script every path through a known menu call flow; an AI agent has no fixed call flow to script against. An LLM produces its reply at runtime from the caller’s words, the agent’s instructions, and the tools it can call. Four properties break the scripted approach.

An InfoQ platform engineering playbook (opens in a new tab) names the gap directly: “Unit and contract tests pin down the deterministic path, but they cannot answer the question that matters most for an LLM system: Is the model still behaving?” Pre-launch testing for AI agents therefore has to score behavior statistically across many runs. Checking one path once falls short.

Traditional IVR testing vs pre-launch voice AI agent testing

The two disciplines share the telephony layer and diverge everywhere above it. The comparison below sets them side by side:

Dimension

Traditional IVR testing

Pre-launch voice AI agent testing

What is measured

Concurrent call capacity, port and trunk usage, prompt playback, DTMF and grammar recognition, MOS, PDD

Everything on the left, plus intent coverage, task completion, tool-call correctness, hallucination, end-to-end speech-to-LLM-to-speech latency, handoff quality

Failure modes

SBC collapse, SIP table errors, port exhaustion, queue overflow, memory leaks, dead-end branches

All of the left, plus TTFT growth under load, run-to-run variance, drift after model updates, multi-turn degradation, accent and noise errors, lost context at handoff

Tooling

Dedicated IVR platforms, SIP traffic generators, scripted call-flow tests

AI-driven caller simulation (opens in a new tab), LLM-as-judge and rule-based scoring, plus telephony load generation

Pass criteria

Each path reaches the right destination; metrics within threshold at target load

Pass rates across repeated runs; p95 latency, task completion and escalation accuracy within tolerance at peak plus margin

Test design

A menu map defines a finite set of paths

Scenario sets, personas and adversarial cases sample an open-ended input space

Treat the left column as a prerequisite for the right one, not a substitute.

A pre-launch testing plan for a voice AI agent

A pre-launch plan for an AI agent runs seven steps, from conversation simulation through telephony capacity. Run them in order: behavior first, then load, then the infrastructure that carries the load.

1. Simulate conversations across every intent

Simulate full conversations for every request type (intent) the agent handles, using test callers that behave like real people. Build the scenarios around these points:

  • Vary the phrasing and the order in which callers give their information.

  • Include difficult callers who are frustrated, terse, or skeptical, since simulated callers tend to be more patient than real ones.

  • Run each scenario several times, because the agent may answer each time differently.

Check the simulated results against a sample of real test calls before you trust the pass rates, and count the step as passed only when the two agree closely.

2. Test edge cases, accents and background noise

Test the situations that most often trip up a voice AI agent, because real calls are rarely clean. Speech recognition accuracy can differ across accents and dialects, so check how the agent performs for the callers you actually serve. Use the checklist below to build the test set:

  • Accents and dialects that match your callers’ regions and languages.

  • Background noise such as traffic, crowds, and households.

  • Interruptions, long pauses, and callers who change their request partway through.

  • Upset or confused callers.

Report results for each caller group so a weak group cannot hide inside a blended average, and count the step as passed when every group meets the same standard.

3. Measure response time at full volume

Time the gap between a caller finishing a sentence and the agent starting to reply, with the system handling as many calls as it will on your busiest day. Callers notice a pause long before they notice a wrong word.

  • Time each part of the reply separately: understanding the caller, writing the answer, and speaking it.

  • Look at the slowest calls, not the average, because those are the calls callers remember.

  • Include calls where the agent looks something up in a back-end system, since lookups slow down most at peak.

Set a maximum reply time before testing starts, and hold the release if load pushes the slowest calls past it.

4. Prove handoffs to human agents

Test every reason the agent hands a caller to a person, and confirm the person receives the full context. A 2026 Gartner survey of customers found 87% say reaching a human (opens in a new tab) is essential when companies use generative AI for service. Check these handoffs:

  • A caller who asks for a person reaches one.

  • The agent hands off when it cannot tell what the caller wants.

  • An upset caller, or a request that fails twice, gets handed off.

  • The person sees the transcript, the handoff reason, and the steps already done, including any verified identity (opens in a new tab) checks.

A handoff passes if the person can continue without asking the caller to repeat anything.

5. Run regression checks before each release

Rerun the full set of test conversations against every change before it reaches callers. Changes include prompts, instructions, knowledge, and the underlying model. Put these controls in place:

  • Keep every version of the agent’s setup so each result maps to an exact build.

  • Change one thing at a time and replay the same scenarios.

  • Block a release that misses its agreed threshold.

  • Keep rollback ready so a failed release returns to the last approved version.

Run the suite on a schedule even when nothing changed on your side, because model providers change their models without notice, and the step passes when every scheduled run meets the agreed threshold.

6. Build the capacity model and traffic ramp

Work out how many simultaneous calls the agent must handle before you write the load test, because a test aimed at the wrong number proves little. Start from your own call records, then plan the test around three ideas:

  • Size the test for your busiest hour.

  • Add headroom for unusual days and for callers who redial after a failed call, since redials add traffic.

  • Raise traffic in steps, holding each level, up to and past your peak.

The step passes when call quality holds at peak plus headroom, and the system recovers fully once traffic drops back to normal.

7. Confirm telephony prerequisites

Confirm the phone carrier can carry your test traffic before you generate any, because an unplanned burst of test calls can fail at the carrier and tell you nothing about the agent. Work through these four checks:

  • Confirm the carrier’s limits on new calls per second and on simultaneous calls.

  • Ask whether calls over the limit are rejected or queued.

  • Use a separate caller ID for test calls so they stay out of production reports.

  • Raise limits before the test starts.

The step passes when the carrier, platform admin, and AI vendor have confirmed the limits and are on the test.

Generic load tools need voice protocol support to test telephony at all.

When and how often to test

Test on every change, on a fixed schedule, and ahead of every predictable peak. Each trigger catches a different risk:

  • Configuration and prompt changes: Run the affected scenarios on every change and the full regression suite before release; Forrester notes that larger synthetic suites are moving toward automatic, continuous testing instead of running only after a noteworthy change.

  • Model and provider updates: Run the golden suite on a schedule even without code changes, since providers update models without notice.

  • Releases: Gate each build on simulation pass rate, task completion and escalation rate, as Bluejay recommends.

  • Seasonal peaks: Run load tests months before high-traffic events to confirm the environment has not changed since the last peak.

InfoQ cautions that canary releases (opens in a new tab) surface only problems that appear within limited traffic slices and are less effective for low-frequency or region-specific issues. Use phased deployment as a final gate that follows pre-launch simulation.

How Parloa runs pre-launch testing at scale

Parloa’s AI Agent Management Platform (opens in a new tab) runs pre-launch testing through Performance Lab, whose lifecycle runs three stages: Build (Agent Builder) → Optimize (Performance Lab) → Observe (Parloa Lens).

Teams build agents in Agent Builder (opens in a new tab) from natural-language briefings, simulate and score them in Performance Lab before go-live, and observe them in production with Parloa Lens (opens in a new tab). One AI model plays the caller while another runs the configured agent.

The benefits map to the plan above:

  • Scenario and edge-case testing: Each simulation runs a defined scenario and evaluates how the agent handles edge cases before go-live.

  • Quality scoring: Performance Lab (opens in a new tab) measures latency and evaluates edge-case handling alongside conversation quality before deployment.

  • Side-by-side comparison: A/B testing in Performance Lab compares two agent configurations, so teams can measure a prompt or model change before release.

  • Release discipline: Version control, regression testing and approval workflows support controlled changes before callers hear them.

  • Handoffs that keep context: Configurable escalation triggers route callers to human agents, and the transcript and context are preserved for the human who picks up.

  • Model choice without rebuilding: Model-agnostic orchestration lets teams test a different STT, TTS or LLM provider against the same scenarios without rebuilding the agent.

Parloa integrates with contact center platforms including Genesys, NiCE, and Five9, supports SIP trunks or direct PSTN forwarding, and supports more than 140 languages. Berlin Brandenburg Airport went live in under six weeks across four languages.

Next step: prove readiness before go-live

An AI agent needs evidence beyond load testing before go-live: proof that it understands callers, answers fast at scale, and hands off cleanly. Pre-launch testing for AI agents adds repeated conversation simulation, accent and noise coverage, end-to-end latency at volume, escalation checks, and regression gates on every release, on top of the telephony tests contact centers already run.

The Parloa (opens in a new tab) AI Agent Management Platform puts that work in one lifecycle: in Optimize, Performance Lab simulates callers and scores agents before deployment, and A/B testing compares agent configurations; in Observe, Parloa Lens observes performance once agents are live.

Callers decide whether to trust an AI agent on their first call. Book a demo (opens in a new tab) to see how Performance Lab tests your agents before go-live.

FAQ

How does functional testing differ from performance testing?

Functional testing confirms each path, prompt, and input reaches the right destination; performance testing confirms the system holds capacity and response speed under concurrent calls. Functional tests run before integration and go-live, while load tests are scheduled ahead of known peaks and after major infrastructure changes.

How do LLMs affect voice agent performance under load?

An AI agent’s language model takes longer to start answering as more calls arrive at once; in one lab test (opens in a new tab), the delay before the model started responding grew from 42 milliseconds with one request to 759 milliseconds with 32 at once. The same model can also give different answers to the same request, so a single test run does not show how the agent will behave. Pre-launch tests should run each scenario several times and check the slowest calls at full production volume.

How do you test IVR systems step by step?

Start with a load test at expected peak volume to confirm the phone system holds up, then push past capacity, add a sudden surge of calls, and hold sustained load for hours. Check that every menu path and prompt reaches the right destination, and rerun those checks after every change. For an AI agent, also test simulated conversations, accents, and background noise, and confirm handoffs to human agents work before go-live.

Which tools are used for IVR and voice agent load testing?

Dedicated IVR testing platforms such as Infovista VistaCX Load generate telephony load at scale. Voice AI testing tools simulate conversations and score agent behavior. Some AI agent platforms include testing themselves, for example, Parloa’s Performance Lab, which simulates callers and scores AI agents before go-live. Open-source SIPp generates Session Initiation Protocol (SIP) and Real-time Transport Protocol (RTP) traffic; JMeter does not list SIP or RTP among its supported protocols.

How often should a voice AI agent be tested?

Test on every prompt, configuration, or model change; on a fixed schedule even without changes; before each release; and ahead of seasonal peaks.

Ready to turn conversations into lasting loyalty?

Let's build your next great customer experience.