LLMs have made relevance assessment substantially easier to scale. Teams can use them to grade query–document pairs, compare retrievers, expand incomplete judgment sets, and rerun evaluations without waiting for a large human-labeling effort. Recent IR research and production implementations show this moving into repeatable search-evaluation infrastructure.[1][2][3] That makes it more important to understand how the resulting measurements are produced.
In brief: A reliable LLM judge can still sit inside a fragile retrieval evaluation. Before trusting an important comparison, test whether the system decision survives changes to the relevance reference, query sample, presentation, assessor, and candidate pool.
An offline relevance score such as nDCG@10 is produced by an evaluation setup. When an LLM supplies the relevance judgments, the judge is one component of that setup. The queries selected for evaluation, the candidates exposed for judgment, the relevance criteria, the evidence shown to the model, and the way judgments are aggregated all contribute to the final system comparison.[1][4]
A simplified view of the pipeline looks like this:
query sample
↓
retrieval runs
↓
candidate pool
↓
document / passage presentation
↓
LLM judge ← relevance criteria / rubric
↓
relevance judgments
↓
metric / aggregation
↓
system decisionThe distinction matters because a reliable judge does not automatically imply a reliable system comparison. The comparison can still depend on choices elsewhere in the pipeline. For practitioners, the central question is how stable the retrieval decision remains when reasonable parts of the evaluation setup are changed.
Relevance Reference Drift
Consider a controlled comparison in which the query set, corpus, and retrieval runs are held fixed while the relevance reference changes.
same queries
same corpus
same retrieval results
different relevance reference
↓
does the system decision change?Relevance Reference Drift describes a change in the apparent quality or ordering of retrieval systems under this intervention. Retrieval has remained fixed; the measured conclusion has moved because the evidence against which the systems are scored has changed.
This matters when judgments are incomplete or were collected under an earlier retrieval regime. Suppose an evaluation collection was built when production retrieval was predominantly lexical. A later semantic retriever may surface relevant documents that use different vocabulary and were rarely exposed during the original pooling and judging process. If those documents are missing from the qrels, the benchmark may underrepresent what the newer system retrieves well.
Information retrieval has dealt with variation in relevance judgments, incomplete judgments, pooling bias, and test-collection reusability for decades.[5][6][7] Recent work tests the same issue when neural retrieval systems return different relevant documents from the systems used to construct an older pool.[8] LLMs make it cheaper to extend judgment sets, but the relationship between the reference and the systems that helped construct it remains.
The issue appears in lexical-to-semantic transitions, new embedding models, multi-vector retrieval, multilingual retrieval, RAG evaluation, and search-platform migrations. Elastic has reported the practical symptom: apparently non-relevant results can turn out to be relevant documents absent from the original judgments.[9]
Migration is a particularly instructive case. Historical judgments may have been accumulated while an Elasticsearch or OpenSearch implementation shaped which documents were routinely exposed for assessment. A migration to a different retrieval architecture can therefore change both the system being measured and the space of relevant evidence the historical reference represents. Preserving performance against the old reference and preserving relevance for users are closely related objectives, but they are not automatically identical.
A useful robustness question follows directly: would the system ranking hold under another reasonable relevance reference?
When the answer changes, the sensitivity provides evidence about the strength and scope of the original conclusion.
What Happens to Unjudged Documents?
Incomplete judgments raise a practical scoring question: how does the evaluation treat retrieved documents for which no relevance judgment exists?
An evaluation that effectively maps every unjudged document to non-relevant can penalise a system for retrieving useful material that happens to be absent from the judgment set. Classical work on evaluation with incomplete relevance information showed that standard effectiveness measures can become unreliable as the judgment set becomes substantially incomplete.[6]
Evaluation procedures handle incompleteness differently. Condensed-list evaluation removes unjudged documents before computing the metric, while measures such as Judged@k quantify how much of the returned ranking has actually been assessed.[10]
Practical analysis therefore benefits from separating two concerns:
- the relevance reference: which documents have judgments, and what do those judgments say?
- the evaluation rule: what happens when a retrieved document has no judgment?
A measured advantage that changes materially with the treatment of unjudged documents deserves further investigation before it is interpreted as a general statement about retrieval quality.
Judge–Pool Coupling
New relevance judgments are often constructed from candidates contributed by several retrieval systems.
BM25 \
dense +--> candidate pool --> LLM judge
hybrid /
↓
relevance referenceThe judge can assess only the candidates that enter the pool. Any document missed by all participating retrievers remains outside the evidence available for judgment.
Judge–Pool Coupling describes this dependency: the relevance reference depends on the candidate pool, while the candidate pool depends on the retrieval systems selected to construct it. The assessor may be operationally separate from those systems while still receiving evidence that they determined.
The underlying issue is familiar from classical pooling: systems contributing to a pool can influence the resulting relevance set and later evaluation reliability.[7][8] Vespa makes the operational point for LLM-judging workflows: judged documents from one retrieval technique are unsuitable for evaluating sufficiently different techniques because they can retrieve relevant query-document pairs absent from the original labels.[2]
LLM-based workflows add a wrinkle: a large automatically generated judgment set can create an impression of broad coverage while retrieval still bounds the candidate universe.
Now consider adding a substantially different retriever:
BM25 \
dense |
hybrid +--> candidate pool
ColBERT /
↓
LLM judgeIf the new system contributes relevant material absent from the original pool, the reference expands along with the systems being compared. That can change measured relationships among systems that were already present. A pool with poor coverage of a challenger's distinctive relevant evidence can reward compatibility with the historical pool alongside retrieval effectiveness.
Two-Level Stability
Questions about judge consistency often combine two different kinds of stability.
The first is document stability: whether individual documents receive similar judgments, or occupy similar positions in a judge-produced preference ranking, across repeated or alternative assessments.
The second is system stability: whether variation in those individual judgments materially changes the comparison between retrieval systems.
Together, these form Two-Level Stability. An evaluator can move substantially at the document level while producing comparatively small changes in the system-level conclusion. The opposite pattern is also possible.
This distinction has deep roots in IR evaluation. Voorhees found that substantial differences in individual relevance judgments could coexist with highly correlated rankings of retrieval systems.[5] Recent LLM work makes the same point at the evaluator level: instance-level judge quality does not fully characterise behaviour after judgments are aggregated into system rankings.[4][1]
Consider two credible judging runs that disagree on a noticeable fraction of individual documents. The disagreement may reveal procedural or assessor sensitivity. After aggregation across queries, however, both runs may preserve the same system ordering and the same important pairwise conclusions.
A different pattern can occur when two judging runs agree on almost every document while disagreeing on a small set of queries that separates two closely matched systems. Strong document-level agreement can then coexist with an unstable system-level decision.
The significance of disagreement depends on how it propagates into the system comparison. Teams ship rankers, replace embedding models, adopt retrieval strategies, or migrate architectures at the system level. Document-level variation remains useful diagnostic evidence because it shows where uncertainty enters the measurement.
The practical rule is straightforward: measure stability at the level at which you intend to act. A useful evaluation reports both how much individual judgments move and whether that movement changes the system comparison.
Stress the Sample, Order, Judge, and Pool
Treating an LLM-based evaluation as a measurement system makes robustness testable. Four useful perturbations are Sample, Order, Judge, and Pool.
Sample
The query set determines which parts of the search problem contribute to the aggregate result. Retrieval effectiveness varies by query, so sample size and composition are major sources of uncertainty. Classic TREC analysis quantified how often changing the topic sample could reverse a comparative conclusion.[11]
A sample dominated by navigational or brand queries may conceal the behaviour of a method developed primarily for exploratory or long-tail search. Aggregate performance can similarly obscure variation across markets, languages, query lengths, or other product-relevant slices.
Sampling analysis should show where the measured advantage originates and how sensitive the aggregate conclusion is to query composition.
Order
Presentation becomes part of the judging protocol whenever it can alter the judgment. For pairwise or listwise assessment, candidate position can directly influence preferences; recent work has shown position effects in both protocols.[12] Swapping sides or permuting candidate slates is therefore a natural robustness check.
Pointwise assessment has a different presentation surface, where relevant variables may include rubric ordering, field ordering, title-versus-passage placement, surrounding context, or the serialization of structured metadata.
The general principle is to vary what the model actually observes and test whether those changes propagate into judgments or system-level conclusions.
Judge
A repeated run of the same model can expose run-to-run variation. A different capable model tests dependence on a particular assessor. A controlled prompt variation probes sensitivity to judging instructions. A targeted human audit provides an external comparison for consequential or ambiguous cases. Studies of LLM relevance labelling show why these distinctions matter: capable LLM assessors can correlate strongly with human system rankings while still exhibiting model-specific biases.[1]
These interventions answer different questions and are most useful when interpreted separately.
Perfect agreement is usually too strict. The operational concern is whether reasonable assessor variation changes the decision.
Pool
The candidate pool determines the evidence available for judgment. A useful pool perturbation changes the contributing retrievers and measures whether the judged reference or existing system conclusions move materially.
This is especially informative when a new retrieval architecture is designed to surface a different candidate population from the incumbent systems.[7][8]
The four perturbations can be summarized as follows:
| Stress | Question |
|---|---|
| Sample | Would another reasonable query sample change the decision? |
| Order | Would another presentation of the evidence change the judgments? |
| Judge | Would another credible assessor change the decision? |
| Pool | Would different retrieved evidence change the relevance reference? |
The appropriate depth of testing depends on the decision. An inexpensive evaluator used to discard weak ranking ideas has a lower reliability requirement than an offline comparison used to justify a production migration.
Freeze the Ruler
Perturbation is valuable while an evaluation setup is being validated. Regression testing and longitudinal comparison require a stable measurement procedure.
The relevant parts of the setup should be frozen and versioned once important sensitivities are understood: query set, corpus snapshot, retrieval runs, pool construction, relevance criteria, document representation, ordering policy, judge model and revision, prompt, relevance scale, decoding settings, metric, and treatment of unjudged documents.
Production implementations are already treating parts of this evaluator configuration as versioned infrastructure. DoorDash, for example, describes versioning its relevance rubric and recalibrating against a golden set when evaluation criteria change, while tagging historical measurements with the rubric version used to produce them.[3]
Evaluation can move while retrieval remains unchanged. A model provider may update a model, a prompt may change, or an additional metadata field may be shown to the judge. Without versioning, metric movement may reflect retrieval changes, evaluator changes, or an interaction between the two.
A useful operating principle is therefore: stress before freezing; freeze before comparing over time.
When Is the Evaluation Good Enough for the Decision?
A team exploring many ranking ideas may use a fast evaluator to narrow the design space. Some judgment noise is acceptable when the evaluation is an early-stage filter.
A small offline relevance advantage being used to justify a production retrieval migration has a different evidentiary burden. Closely matched systems leave less room for evaluator variation, while the cost of a wrong conclusion is materially higher.
Two factors should drive the level of scrutiny: the distance between the competing systems and the consequence of the decision. As systems become harder to distinguish, or the decision becomes harder to reverse, sampling uncertainty, judgment coverage, assessor sensitivity, candidate-pool dependence, and online validation matter more.
A practical criterion is that the system decision should remain stable under perturbations corresponding to how the judgments were produced: sample, presentation, assessor, and pool. The closer the systems are, and the more consequential the decision is, the more stability the evaluation should demonstrate.
LLMs make relevance assessment easier to scale and move more of evaluation into software. That makes the evaluator itself subject to engineering discipline: control its inputs, measure its sensitivities, version its configuration, and test its conclusions under perturbation.
At Searchplex, we treat this as part of relevance engineering: before attributing a measured gain or regression to retrieval, establish how much confidence the evaluation setup supports and where that confidence can break down.
References
On the Use of LLMs for Relevance Labelling. ACM Transactions on Information Systems
Improving retrieval with LLM-as-a-judge. Vespa Blog
LLM-as-a-Judge: Evaluating natural language search. DoorDash Engineering
JuStRank: Benchmarking LLM Judges for System Ranking. ACL 2025
Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. Information Processing & Management
Retrieval Evaluation with Incomplete Information. SIGIR 2004
Bias and the Limits of Pooling for Large Collections. Information Retrieval
On the Reliability of Test Collections for Evaluating Systems of Different Types. SIGIR 2020
Evaluating search relevance part 2 — Phi-3 as relevance judge. Elasticsearch Labs
On Information Retrieval Metrics Designed for Evaluation with Incomplete Relevance Assessments. Information Retrieval
The Effect of Topic Set Size on Retrieval Experiment Error. SIGIR 2002
Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. IJCNLP-AACL 2025