Reranking in RAG: How to sharpen retrieval accuracy

On a high-volume home insurance line, compliance teams cannot review every answer, so reranking in RAG reduces the risk that an AI agent answers from the wrong passage when retrieval returns several plausible matches. A caller asks what the cancellation fee is on a policy taken out in March. The knowledge base holds policy wordings published at different times that differ in one clause: the fee.

Retrieval-Augmented Generation (RAG) pulls conflicting versions, and the older version scores higher and reaches the model first. A second scoring pass can promote the correct version before it replies. The caller does not hear a confident, out-of-date number. That avoids a repeat contact or escalation and gives the compliance team an approved clause to review.

How the second scoring pass works in retrieval

Retrieval-Augmented Generation (RAG) grounds an AI agent's answer in a pre-processed vector database built by indexing a knowledge base. When a caller asks a question, the system searches that database for relevant passages and hands them to the model. But first-stage speed can leave a plausible but incorrect passage at the top, which is where a second scoring pass comes in. Reranking in RAG reorders the passages a first-stage retriever returns, so the most relevant ones reach the model first.

Here is how the two passes divide the work:

• First stage (built for speed): Searches the entire index as soon as the caller stops speaking and accepts likely relevant candidates.

• Second stage (built for precision): Scores each candidate more carefully because it handles only a limited set of passages.

• Trade-off: The reranker's additional precision costs compute and time, but prevents a plausible, incorrect passage from shaping the answer.

The answer an AI agent gives is only as good as the passages at the top of the list. By adding a careful second look at a shortlist, reranking closes the gap between fast retrieval and accurate answers, so the model reads the clause that actually applies to the caller.

Where reranking sits in a two-stage retrieval pipeline

A two-stage retrieval pipeline is how most RAG systems balance speed and accuracy. A fast first stage sweeps the whole knowledge base and returns a shortlist of likely passages. A slower, more precise second stage called the reranker reorders that shortlist so the best match reaches the AI agent first.

Two things can still go wrong. If the correct passage falls outside the shortlist, the reranker never sees it and cannot promote it. If the shortlist is too wide, the reranker wastes time scoring passages that were never plausible, and the caller waits longer.

The lever CX teams control is top-k, the size of that shortlist: tune it against real call transcripts so the right clause consistently reaches the model within the response time callers expect.

Choosing a reranker within a voice latency budget

A chat deployment's tolerance for a slower second stage doesn't transfer to a phone call. On a call, speech detection, transcription, the model's first token, speech synthesis, network transport, and retrieval all consume the same real-time response target. A team that sizes a reranker for chat can consume the retrieval slice's limited headroom and stretch the response into the pause that makes a caller ask whether anyone is there.

Teams can allocate that limited retrieval time among three reranker classes:

• Cross-encoders: Read the query and each candidate together, trading higher candidate-level compute for stronger precision over lighter rerankers; each additional candidate increases that compute.

• Lightweight rerankers: Use smaller or distilled scoring models that run with lower latency than larger cross-encoders. Smaller models reduce latency and typically sacrifice some precision.

• Large language model (LLM) rerankers: Prompt an LLM to judge relevance. They provide flexible relevance judgments at a higher cost, so teams should test them on the organization's policy wording.

Teams should benchmark reranker-stage latency on their own candidate pool and infrastructure, then verify the resulting end-to-end turn time, including transcription and synthesis. Reporting both measurements separates retrieval regressions from delays elsewhere in the voice pipeline.

After measuring the same query mix on phone and chat traffic, teams can use a lighter reranker for voice when the full turn runs long and keep the cross-encoder for chat. Record full-pipeline latency and cost for each choice so teams do not assess the lower-latency voice model on response time alone.

Best practices to sharpen retrieval accuracy

A reranker only sorts what the first-stage retriever gives it, so the passages entering the pipeline must already be the right ones. The following practices shape a clean candidate pool and a verifiable answer, so precision earned in the second stage translates into accurate responses callers can trust.

1. Filter by jurisdiction and product

Apply metadata filters for jurisdiction and product line before retrieval runs, so a regional variant or a policy the caller does not hold never reaches the shortlist. This keeps the reranker focused on passages that legitimately apply to the caller and prevents a close-but-wrong clause from outranking the customer's own policy wording.

2. Enforce access rights

Tie the retrievable set to what the caller has authenticated for, so a retail banking customer never retrieves a private-banking policy, no matter how closely it matches the question. On a call, the AI agent should set the authentication state early in the conversation to gate which passages exist for the rest of the interaction.

3. Apply version and effective-date rules

Tag every document with its version and effective dates, then filter or rank on those tags at query time. This resolves near-identical wording, like the cancellation-fee clause from the opening scenario, because only one version applies on the date the customer signed, and the reranker never has to choose between them.

4. Ground the answer with citations

During answer generation, tie every response to an approved source so a reviewer can see exactly which clause the AI agent read. Citation grounding turns an opaque answer into an auditable one, giving compliance a fast path to verify accuracy and giving operations a clear trail when a caller disputes what they heard.

5. Keep the knowledge base clean

Reranking cannot rescue a stale corpus. Delete or replace superseded documents on a regular cadence, then verify that current versions remain retrievable and rerun transcript tests to catch new gaps. A clean knowledge base provides a reliable foundation for every downstream stage: filtering, retrieval, reranking, and grounding.

A confidently stated wrong fee is compliance exposure: a regulated insurer or bank has quoted a customer a price regulations do not permit it to charge, with no human agent in the loop to catch it. Build evaluation sets only from passages these practices make eligible, preserving jurisdiction, product, and effective-date distinctions.

Measuring reranking against contact center key performance indicators (KPIs)

A reranker earns its place when it raises containment or routing accuracy after release without increasing wrong answers or latency beyond the approved budget. Retrieval metrics matter only when teams map each one to an operational number the business can report upward as a KPI; otherwise the engineering team tunes toward a leaderboard and the business sees nothing change.

Three retrieval metrics can test that contribution against labeled transcripts and production KPIs:

• Recall@k: Whether the right passage entered the pool of k candidates. It indicates the retrieval ceiling for first-contact resolution because an unseen answer cannot resolve the call.

• Mean Reciprocal Rank (MRR): How high the right passage ranks across queries. It can signal wrong-answer and repeat-contact risk when the correct clause loses to a near-duplicate.

• Normalized discounted cumulative gain (nDCG): The quality of the whole ordering, with top positions counting most. It helps assess questions that require several leading passages.

The links between retrieval metrics and operational KPIs set the release gate. A reranker swap or a top-k change ships the way a prompt change does: through simulation against real call transcripts, where the team can see whether the new ordering resolves the cancellation-fee question on the recordings the old one got wrong, and then through a containment check in production against the previous baseline. A change that raises MRR on a test set and lowers containment on live traffic has failed.

Retrieval precision can drift as knowledge changes each quarter and policy versions multiply, so a team that tuned a reranker on last year's pool may see different performance against this year's. Rerun the release gate after quarterly corpus growth or after adding new policy versions, rather than assuming the earlier result still holds.

Govern reranking configurations in production

Unowned production reranker changes can leave teams unable to trace or reverse performance shifts, so assign a named owner for each production reranker configuration and authorize that owner to roll it back when live performance shifts.

Record the channel, model class, candidate count, knowledge filters, transcript evaluation, latency result, and containment baseline. This history gives technical and operational teams a shared timeline that compliance can use during incident review and to compare against the previous configuration.

Clear ownership helps compliance identify the settings behind a disputed answer and helps operations quickly restore the last known configuration.

Turn retrieval accuracy into a competitive advantage

Reranking is an operational discipline that ties retrieval quality to the KPIs contact center leaders already report on: containment, first-contact resolution, and compliance exposure. Teams that treat it that way ship changes through transcript simulation and live containment checks, not leaderboard scores, and every caller hears the clause that actually applies to them.

Parloa helps enterprise contact centers put that discipline into production. The AI Agent Management Platform supports the full lifecycle across Build, Optimize, and Observe, so teams can govern reranker configurations, simulate changes against real call transcripts, and monitor containment and latency in one place, with the ownership and audit trail compliance requires.

Book a demo to see how Parloa can help you sharpen retrieval accuracy so every caller hears the policy answer that applies to them.

FAQs about reranking in RAG

Does reranking add latency to voice AI agents?

Yes. Caching repeated questions can avoid rerunning the same scoring work, making it most useful where repeat questions are common.

When should a team skip reranking altogether?

When first-stage retrieval already puts the right passage on top for the questions callers actually ask, which happens with small, well-structured knowledge bases covering distinct topics. Reranking pays off when passages are near-duplicates, such as policy versions or regional variants, and simulation on real transcripts shows the first stage promoting the wrong one.

Cross-encoder or LLM reranker for a contact center knowledge base?

Compare both on the same domain-specific policy queries. Prefer a fine-tuned cross-encoder or lightweight reranker when it meets the required relevance level within the channel's response time. Test an LLM reranker when relevance depends on reasoning the smaller models miss and the channel can absorb its higher per-call cost.

How many candidates should the first stage pass to the reranker?

Use a fixed reranker and transcript set to select the smallest k that meets the recall target within the approved full-turn budget. Retest that value whenever the corpus, query mix, model, or latency budget changes.