Word error rate: How to benchmark voice AI
Callers from one region keep landing in the wrong queue because the voice AI benchmark used cleaner audio than the contact center receives. The AI agent misreads a regional vowel as the wrong intent and hands the call to a team that cannot help. Those callers authenticate twice, escalate to a human agent more often, and leave a poor customer satisfaction score (CSAT).
A vendor deck might quote a very low word error rate, yet a production report can still show lower CSAT.
What is word error rate?
Word error rate (WER) is the ratio of recognition errors to the total number of words in the reference transcript, expressed as a percentage. It measures how accurately an automatic speech recognition (ASR) system converts spoken language into text, which is the first thing an AI agent must do on a call before it can match intent or look up an account.
Calculate WER by adding substitutions, deletions, and insertions, then dividing that sum by the number of words in the reference transcript. Take a caller who says "I need to move funds from my savings account today." The reference contains 10 words. The transcript reads "I need to move refunds from my savings account today please." That is one substitution and one insertion, so two errors over 10 reference words gives 20% WER.
Where word error rate misleads
A single WER percentage flattens very different failures into one number. The formula counts three kinds of errors, and each carries a different cost on a phone line:
Substitution: Swaps one word for another, as when "fund" arrives as "refund" and the AI agent opens a refund flow instead of a transfer.
Deletion: Drops words the caller said, such as one digit of an account number, and authentication fails on the first attempt.
Insertion: Adds a word nobody said, most often when the recognizer transcribes line noise or a hold tone as speech.
A dropped digit and a misheard "the" weigh the same in that formula, which is why WER alone cannot certify a phone channel. Normalization compounds the ambiguity: if the reference reads "twenty" and the transcript reads "20," a scorer that has not normalized numerals counts a substitution no caller would notice. Two vendors quoting different WER on the same audio may differ only in how each prepared the reference, not in what the recognizer actually did.
Set production acceptance criteria
A voice AI benchmark becomes acceptance criteria when every metric has a threshold, an audio segment it applies to, and an owner who signs off on it. The steps below move from defining the metrics through building a representative test set, measuring perceptual quality on both sides of the call, and combining the results into one scorecard that governs vendor selection and every post-launch re-run.
1. Define the production metrics before testing
Before scoring any audio, name what the benchmark measures and why. A voice AI benchmark is a controlled test of how accurately an AI agent hears, how natural it sounds, and how quickly it responds. Three metrics carry the weight:
Word error rate (WER): The ratio of recognition errors to reference words at the ASR layer, the component that turns caller speech into text before the AI matches intent or looks up an account.
ASR accuracy: A broader view of recognition quality that includes exact-match entity accuracy for numbers, names, and alphanumeric codes, plus intent recognition accuracy on live traffic.
Mean opinion score (MOS): A subjective listening-quality rating from poor to excellent that scores how the audio comes across, applied to synthesized outbound speech and inbound channel quality.
Fix definitions and scoring rules before the first sample is graded so vendors cannot re-label the same audio into a better number.
2. Set thresholds by business risk
Thresholds have to reflect what a mistake costs the caller, not what looks presentable on a slide. Set a stricter WER threshold on clean audio and a separate, more forgiving threshold for noisy or accented telephony audio, then calibrate both to the business risk of misrouting and failed authentication.
Ask whether the vendor measured its unusually low WER on clean audio before applying it to codec-compressed calls, because the same model rarely delivers the same number on both. Even a low aggregate WER fails the caller when an error lands on a claim reference or an account digit, so define additional acceptance criteria for high-risk entities and score them by exact character match rather than folding them into WER. Set diagnostic thresholds that trigger remediation the moment any segment indicates a codec, sample, or model issue rather than waiting for aggregate drift.
3. Build the test set from real customer call audio
The audio in the benchmark decides whether the result predicts production. Public benchmark corpora often rely on read speech recorded into a good microphone in a quiet room, while real calls arrive through a telephony codec that strips upper frequencies and land amid background noise, cross-talk, and callers who start before the prompt ends.
Assemble the test set from consented, anonymized production calls, stratify it into segments, and score each separately:
Telephony codec compression: Pass every sample through the same codec chain callers use.
Background noise: Tier samples by signal-to-noise ratio and set a threshold for each tier.
Regional and non-native accents: Oversample lower-volume groups when needed and score each against the common target.
Numbers and alphanumeric codes: Score by exact character match.
Each production language: Give every language its own segment and threshold.
Hold voice activity detection settings constant across every vendor tested so the audio, not the configuration, drives the result.
4. Measure MOS on both sides of the conversation
MOS covers what the AI agent sounds like outbound and what it hears inbound, and both matter. Outbound, the score moves with intonation and pacing, which is what prosody means in practice. A synthesized voice with every word correct can still pause in the wrong place when reading back an account number, or run a question and a statement together at one pitch, and lose points with listeners.
Inbound quality also needs a score. The audio the AI agent hears has already crossed the caller's handset, the carrier network, and the codec, and that quality caps what any recognizer can do. Human panels do not scale across thousands of segments, so controlled benchmarks use the Perceptual Objective Listening Quality Analysis (POLQA) algorithm to produce MOS-LQO without a listener in the room.
Run inbound MOS-LQO against codec bitrate alongside WER on the same segments: if a lower-bitrate codec lowers MOS-LQO and raises WER together, check inbound channel quality before renegotiating with the model vendor.
5. Combine the metrics into one scorecard and re-run it in production
A benchmark becomes governance when its results sit on one scorecard with a threshold beside each. Vendors sign against that scorecard in the proof of concept, and teams use it as the baseline for later re-runs and post-go-live service levels.
The scorecard covers four dimensions, each with its own acceptance threshold:
Recognition quality: WER and exact-match entity accuracy for each accent and language segment against the common service-quality target.
Perceptual quality: Inbound MOS-LQO and outbound MOS together to distinguish channel degradation from model errors.
Latency: P50 median response time and P95 95th-percentile response time, because averages hide the pauses where agentic AI latency and cost trade-offs surface.
Task success: Routing accuracy, containment, and CSAT, because recognition is only instrumental.
Between quarterly re-runs, contact center AI observability catches drift on the day it starts by scoring every production conversation against the same thresholds the scorecard set.
Govern word error rate with production call audio
Word error rate is a diagnostic, not a verdict. A single aggregate percentage on clean audio has never certified a phone channel; the number that matters is WER on the codec-compressed, accented, noisy audio your callers actually produce, segmented so no failure hides inside an average and paired with the MOS reading that explains why the recognizer struggled.
Parloa brings testing and monitoring together across Build, Optimize, and Observe, with historical-data testing, multilingual deployment across 140+ languages, and continuous observability that scores every production conversation against the thresholds the scorecard set. Compliance covers ISO 27001:2022, ISO 17442:2020, SOC 2 Type 1 & 2, PCI DSS, HIPAA, and DORA.
Book a demo to benchmark WER, ASR accuracy, and MOS on your own call audio, then monitor them against the same thresholds after launch.
Get in touch with our teamFAQs about benchmarking voice AI
How much call audio does a WER benchmark need?
There is no universal call-audio duration for a WER benchmark. Set the sample size per language to represent continuous speech across every segment the benchmark must compare. Calibrate that starting point to the business risk and the number of segments in the test. Every accent, noise tier, and language segment needs enough samples for teams to compare its WER with the others separately instead of folding it into one aggregate.
Can vendor leaderboard WER be used for shortlisting?
Yes, for shortlisting only. Public benchmarks using read speech, not conversation, can show that a model is worth testing. A leaderboard position never replaces a run on your own codec-compressed telephony audio, which is the only figure that belongs in acceptance criteria.
Is MOS worth measuring if WER is low?
Yes. A low WER on clean input says nothing about how callers perceive the synthesized voice. Inbound MOS-LQO is also one way to separate a network or codec problem from a model problem when production WER climbs.