Who improves the agent that improves the agent? Harness engineering at Parloa

When Parloa first launched Parloa Navigator, I wrote an article about why Parloa doubled down on the agent harness, explaining why oftentimes, when an agent misbehaves, the model isn’t the problem. The harness is. But if the harness is what ensures the AI agent acts appropriately, who or what ensures the harness works correctly? That’s where humans come in. 

When agent builders use Navigator, Navigator makes recommendations for how they can improve their agents. Then, they review the recommendations and ultimately decide whether or not they want to implement the change. 

Having launched Navigator, our engineers are now focused on ensuring the product is continuously improving the quality of recommendations it provides. For that, we have adopted the discipline of harness engineering.

Researcher Vivek Trivedy describes harness engineering as the practice of turning a system that behaves like a distribution into one you can trust like a guarantee, through a loop where every correction becomes a permanent fix rather than a one-off patch.

This is our approach to engineering Navigator, and it’s not dissimilar to how Navigator works for agent builders. 

Why you can’t “fix” AI models

The first thing to note is that non-deterministic models are inherently unfixable.

With deterministic code, failure is obvious. The product either works, or it doesn’t work.

Non-deterministic models don’t offer such black and white resolution. The same prompt can behave differently from one run to the next, making a clear cut “problem” harder to diagnose. So instead of tracking a fixed output, you’re tracking a range of behavior. This means that you can’t rely on one-off prompt edits. If you do, the same problem will surface again in a different conversation, phrased just differently enough that the prompt edit no longer applies. Additionally, piling edits on top of edits creates larger prompts that ultimately confuse the LLM and can accidentally introduce contradicting bits of information. 

The goal for improving non-deterministic models, then, is to narrow the range of what “good behavior” looks like, so it’s specific enough that the outputs are correct the majority of the time. This was the impetus for Parloa’s Subtask Agents, which introduce two layer routing into voice AI agents that increase governance while not compromising on latency.

Harness engineering in practice

Harness engineering treats agent failure as a signal about the harness and aims to close the gap at that level. Currently, the improvement loop is highly manual. As it works, we are also working to automate more and more of the loop, so it becomes more autonomous and reliable over time. It’s the same approach we take in Studio for our customers, where customers are using Parloa’s Sims & Evals to power their own harness engineering. 

While the end goal is for this loop to become more automated, it’s important to not shortcut the manual effort. Teams that do tend to never learn what the failure patterns were in the first place, causing them to struggle to make the right automation decisions down the line.

There are three stages to the harness engineering loop:

Stage 1: What needs to be improved?

First, we receive direct builder feedback on what Navigator produced. A human engineer inspects each piece of feedback against the live agent prompt and manually fixes the harness. The process is slow and unscalable, but it’s also the only way to develop the pattern recognition that later automation will depend on.

Stage 1 is where we get the insights dashboards don’t provide: which categories of recommendation builders ignore, which failures are continuously happening (just in different conversations), and where the prompt is doing work that would be better managed in the configuration.

Stage 2: Enrich and analyze

In Stage 2, the feedback loop moves inside of Navigator, enriching human feedback with the context Navigator had available when it made the recommendation. 

At this stage, live agent configurations are fetched, and the transcript is pulled via API for full context. Then, an LLM reads the live Navigator prompt, classifies the root clause, and drafts a generalized point of view on what the gap is. The product manager adds in additional human points of view, and all feedback is captured inside of Navigator to help it understand which recommendations were approved, and which didn’t work. 

Stage 3: Manage and validate

Next, proposed fixes are consolidated, and a relevant test case is created. This way, when we look to further optimize the system, we can validate that the future fix doesn’t regress towards an earlier state. Regression cannot be ignored, as in this looped approach, there’s heightened risk for an added fix to accidentally break what was already working in the harness. Additionally, with each edit comes more opportunities for the change to collide with or contradict other parts of Navigator’s prompt, making the harness harder to dissect if something goes wrong. That’s why we always define a set of evaluations for different scenarios and test the agent against all possible scenarios before accepting a change. 

At the end of the test phase, a human makes the final decision on whether or not the fix is deployed.

Harness engineering, all the way down

Ultimately, the way we improve Navigator is not dissimilar to how Sims & Evals in Studio works for our customers. A builder runs simulation and evaluation sets against their own agent, finds the failure patterns behind a bad score, and confirms a fix. 

The future state is one where automation owns more of the optimization process, but it won’t change the loop itself. Whether human or AI, the loop will always follow the same stages: feedback, enrichment, fix, validation. Automation will make the harness tighter, lower the risk gaps, and deliver better results for our customers.