How Parloa benchmarks models for enterprise AI agents

Picture an AI agent that handles six customer flows: billing questions, address changes, a payment dispute, and a few others. You move it onto a newer model, rerun your tests, and the overall pass rate goes up. Five flows got better. The payment dispute flow, though, now skips a verification step it used to handle reliably.

On a dashboard, that pass rate looks like an improvement. The customer calling about a disputed charge never sees that average. They experience the broken flow.

That's the kind of change we need to catch when evaluating models at Parloa. Before a model becomes available to build on in our AI agent management platform (AMP), we benchmark it against the model it would replace, examining how agents perform across individual use cases.

Why changing models can require different prompts

Public benchmarks give us useful information about a foundation model's broad capabilities. Inside an enterprise agent, though, the model has a particular job: follow the agent’s instructions and carry a conversation through to the intended outcome.

The agent's prompt contains the instructions that shape how it does that job, including when and how to call tools. A prompt written for one model can carry assumptions about how that model interprets instructions. Move it to another model, and those assumptions may no longer apply. Tool sequencing can change, or a step that previously worked may need more explicit wording.

Take a call transfer. One model may confirm the transfer to the caller and call the required tool. Another may confirm it verbally but skip the tool call unless the prompt spells out the sequence more explicitly. The caller hears the same confirmation either way, but only one of those transfers actually happens.

This makes the comparison harder than swapping models and checking the score. If the new model performs worse with the existing prompt, how much comes from the model, and how much comes from instructions written for its predecessor?

Our benchmark answers that question with two separate comparisons, both run on the same setup.

What we evaluate in the benchmark

We benchmark a broad range of available models using synthetic data. Only models from subprocessors we have signed contracts with can become available to build on in AMP. We compare each of those candidates against the model in Parloa’s stack it would replace.

Each benchmark runs on a maintained set of production-representative agent designs and their use cases. We favor complex designs: agents with branching logic, many subtask agents, numerous skills and tools, long prompts, or agents operating in regulated industries.

Simulated callers work through each use case in full, multi-turn conversations. We repeat those conversations many times to examine behavior across a population of runs. A few successful examples won't tell us as much as seeing how consistently an agent handles the same kind of work.

The tools are mocked during these simulations, so benchmark runs don’t reach live systems. Every model in a comparison uses the same versioned mocks and receives the same response for the same input. Models may still call different tools or pass different arguments, and those differences are part of what we measure.

We report on two performance dimensions:

Instruction-following measures whether the agent does what its prompt asks: takes the right steps in the right order, calls the appropriate tools, and reaches the intended outcome. We report results by use case, so individual workflows stay visible throughout the comparison.

Latency measures how long the caller waits for the agent’s next response. This is end-to-end agent latency: it includes orchestration, network, and tool execution, not just the model, so we only compare it under equivalent infrastructure and concurrency. We report it per turn, from the caller’s message to the agent’s next reply. Because tool execution is part of that time, turns that require a tool call are reported separately from those that don’t.

Giving the new model a fair comparison

We run the same benchmark twice, keeping the tools, scenarios, and scoring unchanged.

First, a drop-in swap. We change only the model and leave the agent’s existing instructions as they are. This measures the immediate risk of migrating: how well the new model performs as a drop-in replacement, before any changes to the prompt. It can’t separate the model’s capability from prompt fit, though, because the instructions were developed for the previous model.

Then, an adapted run. We adapt the instructions for the new model and run the benchmark again, refining and rerunning where the results show the prompt needs work. This shows how the new model performs when the agent is configured for it.

Reporting both results lets us distinguish immediate migration risk from performance after adaptation. The adaptation work also surfaces wording and instructions that worked in testing, which feed into the guidance for builders.

How we score and validate results

Some requirements can be checked directly. Did the agent call a particular tool? Did its output match the required format? Did the response arrive within the latency threshold? Deterministic rules handle these, giving the same result for the same recorded conversation. The underlying events can still vary from run to run, especially latency.

Other requirements need judgment. Deciding whether an agent followed an instruction or reached the correct outcome can require interpreting the conversation.

For these judgement-based requirements, a model acts as the judge. We calibrate the judges against human labels, and when a decision depends on a small margin, we re-judge the same conversations and use the per-conversation majority.

We also review results rule by rule. Two models can reach roughly the same overall score while failing different requirements. Breaking the score down by rule shows which requirements each model misses.

We define before each run which failures count as invalid. A runtime error or empty response is excluded only when the benchmark infrastructure caused it. If the model, provider, or agent configuration caused it, it counts as part of the result, since excluding it could make an unreliable model look stronger. Exclusions are reported for both sides of every comparison. Out-of-scope conversations are reported separately: scenarios a production deployment wouldn't route to that agent stay out of the overall pass rate.

Each benchmark records everything needed to reconstruct and rerun it: the agent definition, prompt variant, model and provider, sampling settings, simulations, tool mocks, scoring rules, judge versions, and seeds.

What it takes to clear the benchmark gate

All of this feeds the benchmark gate, the first checkpoint a model has to clear before it can become a recommended model. To clear it, a model must perform equivalently to or better than the current model within predefined acceptance margins, accounting for uncertainty in the results rather than comparing single scores. Every critical use case and rule group has to meet the bar on its own, so a strong average can’t hide a regression in any one of them.

A model clears the gate as part of the full setup we tested: its adapted prompt, the provider, and the serving configuration. Cost and latency also factor into the internal decision. Beyond the gate, later stages add prompting guidance, applied validation, production capacity, and testing on real calls.

These results provide evidence about performance under the benchmark conditions. Clearance doesn’t automatically validate the model for other agents, languages, channels, or use cases; those configurations still need their own testing.

What builders get from the work

Prompt refinement produces useful findings beyond the comparison itself. It shows where a model needed explicit instruction in the agents we tested, which wording worked there, and which patterns produced poor results.

We capture those findings in model-specific prompting guidelines. The guidelines identify the exact model version, the configurations tested, the evidence behind the findings, including the number of conversations and use cases and the comparison model, and any known limits. That way, builders can judge how well the advice fits their own agents. The guidelines also describe what the model handled without explicit instruction in testing, and include a validation procedure to follow before deployment.

For builders moving an existing agent, the migration audit is the place to start. It highlights assumptions in prompts written for the previous model that the new model interpreted differently. These are patterns observed in testing rather than a complete list of differences.

Our agent architects add best practices based on applying the guidelines while building agents. Both are published in Parloa’s product documentation. Builders should still validate any recommended wording on their own agents before deployment.

Models will keep changing. The standard should not.

As new models arrive and existing ones evolve, the ones agents rely on today will eventually be replaced. Each transition brings familiar questions for builders: Which instructions still work? Where does the prompt need to change? Does the agent still handle each tested workflow as reliably as before?

A consistent benchmarking process gives us a way to work through those questions. For the hypothetical agent handling six customer flows, that means keeping the missed verification step visible even when the overall score improves.

As models change, the findings and prompting guidance will change with them. The standard remains the same: a recommendation needs evidence from the tested use cases, accompanied by guidance that helps builders adapt and validate their own agents.