What does it mean for a speech-to-speech model to understand a caller?

On September 10, OpenAI made GPT-Live-1 available through its API. The full-duplex speech-to-speech model can listen and speak at the same time without waiting for strict turns. It separates the fast conversational “talker” from deeper reasoning, tool use, or backend workflows that can be handled elsewhere when needed.

This architecture can make conversations faster and more natural, but natural conversation is only one part of a good enterprise voice interaction. AI agents also need to reliably understand names, numbers, booking references, account details, and other exact information across different languages and scenarios. 

With speech-to-speech models, evaluating that understanding is less straightforward than with a traditional speech pipeline. Listening, reasoning, and speaking are more tightly connected, and there is no longer necessarily a single speech-to-text output that tells us what the system understood. Previous evaluations showed us that speech recognition can vary by language and information type. Speech-to-speech models also combine more of the pipeline inside one model, leaving fewer individual components to adjust.

The potential for natural conversation, together with those open questions about comprehension, made GPT-Live-1 worth evaluating against dedicated speech-to-text systems.

But before comparing recognition scores, we first had to answer a more basic question: What should we actually measure when we ask whether a speech-to-speech model understood the caller?

What does a speech-to-speech model actually “understand”?

With traditional speech-to-text (STT), the measurement target is clear: audio goes in and a transcript comes out. We can compare that transcript with the expected text.

Speech-to-speech models blur that boundary. Information from the caller can appear in several places: in a transcript, in data passed to another system or used for a downstream action, and in what the model says back.

We tested GPT-Live-1 throughout the preview phase. In earlier preview versions, invoking delegation passed a separate text-content field to the client, which we evaluated explicitly. This gave us three signals to compare: the direct transcript of the caller’s speech, the information used for a downstream action or passed to another system, and the model’s spoken repeat-back.

We initially used repeat-back (asking the model to reproduce what it heard) as one measure of whether the model had understood exact information. On some structured inputs, particularly codes, it performed materially worse than the direct transcript.

To locate the failure, we also evaluated the information used downstream. In our tests, that value contained the same captured information as the direct transcript, although formatting and punctuation could differ.

Imagine a caller gives an airline agent the booking reference AB7K42. The model captures AB7K42 correctly, and the correct value is available to the backend workflow. But when the agent repeats the reference, it says AB7K4.

Looking only at the spoken confirmation could make this appear to be a recognition error. The other signals show that the model captured information the workflow could use but failed when verbalizing it.

Both outcomes matter in enterprise applications. Some workflows use captured information directly. Others confirm it with the caller, especially when the input is unclear or requires stricter verification. Capture, downstream use, and verbal reproduction are separate failure points.

For this benchmark, we wanted to measure whether GPT-Live-1 had captured the information that an enterprise workflow could use, without also measuring errors introduced when the model turned that information back into speech. The direct transcript fit that purpose: in our English and German code tests, it matched the downstream payload after we normalized case and punctuation. It also required no special repeat-back instructions and was available in every run, so we used it as our measure of task-relevant speech understanding in GPT-Live-1.

Not every speech-to-speech model exposes the same interfaces. GPT-Realtime, for example, didn’t provide an equivalent native input transcript in our evaluation setup. We used repeat-back as its recognition measure, and separate tool-calling tests produced similar results.

How we tested GPT-Live-1

We benchmarked GPT-Live-1 against Azure STT and Deepgram STT across three input categories, focused on recognition rather than the full conversational experience:

1. General conversational speech, such as “I want to book a restaurant in Cooperdale for five people.”

2. Numerics, such as “9 2 2 2.”

3. Alphanumeric sequences, such as “Q H I V P 3.”

We tested English, German, Spanish, French, and Japanese. We also included GPT-Realtime, an earlier OpenAI speech-to-speech model, as an additional reference point.

We evaluated the systems on the same telephony-like audio and reference labels. For general speech, we measured word-error rate. For numbers and codes, we measured whether the complete value was captured correctly. The English and German evaluations were broader, with around 20,000 samples each, while the Spanish, French, and Japanese tests were smaller directional checks with around 3,000 samples each.

Recognition quality depends on what callers say

GPT-Live-1 came closest to dedicated STT on general conversational speech under the telephony-like conditions of our benchmark. These conditions are important because they reflect real-world phone interactions, where audio quality is often constrained.

In English and German, its word-error rate was within roughly three percentage points of the dedicated STT systems we evaluated. In our smaller Spanish, French, and Japanese tests, GPT-Live-1 remained within roughly 3.5 percentage points of the strongest dedicated system.

For conversational speech, the recognition gap was relatively small across all five languages. The picture changed when information had to be captured exactly.

Conversational understanding can tolerate some approximation. Structured values are less forgiving because one incorrect character may prevent a booking reference, postcode, customer ID, or account number from matching the expected record. To measure this, we test whether a complete code was understood correctly with what we call the entity (recognition) accuracy.

On numeric inputs, GPT-Live-1 was within roughly one to two percentage points of dedicated STT in English and German, with Spanish showing a similarly strong result. But in our smaller French and Japanese tests, it trailed the strongest dedicated system by roughly 11–12 percentage points.

Alphanumeric sequences exposed larger differences. In English, GPT-Live-1 was competitive with the dedicated systems, sitting between Azure and Deepgram. In German, the gap to dedicated STT ranged from roughly 9–21 percentage points. Across our smaller Spanish, French, and Japanese tests, GPT-Live-1 trailed the strongest dedicated system by roughly 11–18 points.

Taken together, the results show that speech-understanding quality depends on the combination of language and information type. Strong performance on conversational French, for example, doesn’t guarantee equally strong performance when a caller provides exact numbers or codes.

Even as directional checks, the smaller Spanish, French, and Japanese tests show why multilingual evaluation needs to go beyond asking whether a model “supports” a language.

The score depends on the configuration and the run

The model’s repeat-back performance changed materially with the instructions we gave it. With earlier, simpler prompts, repeat-back accuracy was worse by as much as 40 percentage points in some tests, particularly on numbers and alphanumeric sequences.

This doesn’t mean that one prompt eliminates repeat-back errors. It shows that a score based on the model’s generated response measures the task configuration as well as the information the model captured. That is another reason to distinguish repeat-back from the direct transcript.

Aggregate scores can also hide variation between runs. When we repeated parts of the evaluation with the same inputs and configuration, the headline results remained consistent, but around 20% of evaluated items changed correctness state.

An input recognized correctly in one run could fail in another, while a different input moved in the opposite direction. Because those changes can cancel each other out, two runs can report the same 90% accuracy while failing on different inputs.

Our repeated-run evidence showed that dedicated STT was more consistent than GPT-Live-1 at the item level. It also showed why aggregate accuracy and item-level consistency need to be reported separately: one tells us how often a system is right overall, while the other tells us whether it succeeds and fails on the same inputs each time. Both can matter in enterprise workflows.

Evaluate speech understanding in context

GPT-Live-1 stayed close to dedicated STT on general speech across all five languages we tested. English performance was strong across all three input categories, while structured-input performance in the other languages was more mixed. Compared with GPT-Realtime, GPT-Live-1 also showed a more balanced pattern between English and the other evaluated languages on structured inputs.

But a model ranking answers only part of the practical question. An enterprise voice application needs to capture what the caller said, use or pass that information correctly, and communicate it accurately when confirmation is required. It must do so under the actual instructions and workflow, consistently, and for the languages and information types the application depends on.

Word-error rate and entity recognition accuracy remain the right metrics for speech-understanding. What changes with speech-to-speech models is how we calculate them and what we can infer from the result. We first have to identify which output represents the information the model captured and distinguish recognition errors from errors introduced later in the interaction. As model architectures and interfaces evolve, that signal may change or disappear, so the measurement approach needs to be checked again for each model and version. The challenge isn’t finding new metrics, but ensuring that established ones are calculated from the right evidence.

Methodology note: The primary benchmark was run during the preview phase of GPT-Live-1 with an earlier API version. We repeated the English and German benchmark on the newer version using the same audio and task configuration and observed no material overall change. The smaller Spanish, French, and Japanese results remain directional.