It is tempting to use an LLM to evaluate how your search system is performing.
Give it a query, show it the results, ask it to act as a judge, and have it rate their relevance. The labels look clean. The explanations sound reasonable. You can aggregate the scores and compare one version of search with another.
But if you are diagnosing a search problem or testing a change, this is often the wrong place to start.
The LLM can judge only the results it receives. It can tell you whether those results look relevant. It cannot establish that the search system worked correctly before producing them.
Relevant documents may have been indexed incorrectly, filtered out, or missed during retrieval. Those failures require different tests and different engineering fixes.
In production search, several failures can produce the same visible symptom: a result looks wrong. A filter may have removed the right candidate, the candidate window may be too shallow, or the right documents may have reached ranking and been ordered poorly. The result page does not identify which boundary failed; diagnosis requires separate checks.
This article organizes the checks into five diagnostic layers: Data + Configuration Correctness, Candidate Recall, Relevance Judgments, Ranking Evaluation, and Evaluation Audit. Together, they form the Search Evaluation Ladder.
╭────────────────────────────────────╮
│ Data + Configuration Correctness │
╰──────────────────┬─────────────────╯
│
╭───────────▼──────────╮
│ Candidate Recall │
╰───────────┬──────────╯
│
╭─────────────▼────────────╮
│ Relevance Judgments │
╰─────────────┬────────────╯
│
╭────────────▼───────────╮
│ Ranking Evaluation │
╰────────────┬───────────╯
│
╭───────────▼──────────╮
│ Evaluation Audit │
╰──────────────────────╯This ladder focuses on offline diagnosis and evaluation. Production changes may also require online validation, such as an A/B test.
Each higher layer assumes that enough of the lower layers are working. Starting at the judge can produce a precise answer to the wrong question.
Search Must Behave as Configured
Data + Configuration Correctness can usually be tested deterministically: is the search system processing the corpus and queries as intended?
This covers field types, tokenization, case and Unicode normalization, language analysis, stemming, synonyms, numeric and date handling, missing fields, filters, and permissions. A defect at this layer changes what the system can retrieve before ranking quality is measured.
Consider a product identifier such as AB-1234. If the configured analyzer splits it into ab and 1234, a search for the exact identifier may behave like a broad two-term query. An LLM can judge the returned products accurately while saying nothing about the configuration error that produced them.
Linguistic processing can create the same problem even when the implementation is behaving as designed. The original Porter stemmer, for example, can collapse distinctions between terms such as universe and university; later versions introduced handling to avoid such conflations. The algorithm may be operating correctly while collapsing a distinction the product needs to preserve.
Deterministic fixtures are the right instrument here:
╭──────────────────────────────────╮
│ Known document + known query │
╰─────────────────┬────────────────╯
│
╭───────▼──────╮
│ Search │
╰───────┬──────╯
│
╭──────────────────────▼─────────────────────╮
│ Expected match and non-match behaviour? │
╰──────────────────────┬─────────────────────╯
│
╭────────▼───────╮
│ PASS / FAIL │
╰────────────────╯A deterministic fixture checks both sides of the expected search behaviour. The intended document must match; documents with the wrong identifier, tenant, language, or permissions must not. Analyzer output, indexed fields, query parsing, filters, and access-control predicates should be inspectable without a relevance judge.
A common diagnostic sequence is to inspect the indexed representation first, verify that the expected field is searchable, and only then test whether the query admits the document into the candidate set. This keeps a data or configuration failure from being mistaken for a candidate-retrieval problem.
An LLM judgment is an expensive and indirect way to discover a deterministic configuration error.
Can the Right Document Reach Ranking?
Once the data path is correct, the next question is candidate recall: can a known relevant document reach the stage where ranking can act on it?
Candidate generation may involve lexical retrieval, dense nearest-neighbor search, hybrid branches, query expansion, filters, permissions, routing, and a fixed candidate depth. The previous layer checks whether filters and permissions behave as configured; this layer asks whether eligible relevant documents still receive adequate coverage. Approximate retrieval adds another variable: the engine may trade recall for latency by examining only part of the vector index.
An exact-versus-approximate comparison measures ANN recall: whether approximate search recovers the neighbors that exact vector search would return. Even 100% ANN recall does not show that the embedding places relevant documents near the query. Relevance Recall@K separately measures whether known relevant documents reach the candidate stage.
╭──────────────────────────╮
│ Known relevant document │
╰─────────────┬────────────╯
│
╭────────────▼───────────╮
│ Candidate generation │
╰────────────┬───────────╯
│
╭──────────▼─────────╮
│ Present in top K? │
╰──────────┬─────────╯
│
┌────────no─────────┴────────yes────────┐
│ │
│ │
╭───────────▼──────────╮ ╭───────────────▼──────────────╮
│ Ranking cannot help │ │ Ranking can still improve it │
╰──────────────────────╯ ╰──────────────────────────────╯A candidate-recall test does not ask whether the document finished first. It asks whether the document was available at the candidate depth consumed by the next ranking stage. For a single fixture, this is an inclusion check. Across a labeled query set, the same check becomes Recall@K at the candidate stage.
This exposes failures caused by shallow candidate windows, low ANN recall, restrictive filters or permissions, routing errors, or a retrieval branch that never ran.
Ranking cannot recover a document that candidate generation never admitted.
Separate Recall Failures from Ranking Failures
A poor top-ten result set can reflect failures at two different boundaries, sometimes at both:
╭────────────────╮
│ Poor top-10 │
╰────────┬───────╯
│
╭────────────────────▼───────────────────╮
│ Known-relevant coverage adequate │
│ at candidate depth? │
╰────────────────────┬───────────────────╯
│
┌───────no─────────┴────────yes───────┐
│ │
│ │
╭───────────▼──────────╮ ╭───────────▼──────────╮
│ Investigate retrieval│ │ Investigate ranking │
╰──────────────────────╯ ╰──────────────────────╯The visible symptom may be the same: relevant results are absent or too low. The corrective action differs.
Inadequate candidate coverage may require changes to query expansion, filters, ANN settings, retrieval routing, or candidate depth. Poor ranking means useful candidates were available but ordered poorly, and may require different features, weights, training data, or a stronger reranker. Both can occur in the same result set.
The candidate-recall test therefore does more than measure retrieval coverage. It identifies which side of the retrieval–ranking boundary needs investigation. Tuning ranking when the necessary candidates were never available spends effort at the wrong boundary.
Evaluate Ordering After the Candidate Path Is Healthy
Deterministic tests establish whether expected search behaviour holds. They cannot show whether one ranking configuration is better across a representative query set.
That requires offline relevance evaluation:
╭────────────────────────────────────────────────╮
│ Representative queries + relevance judgments │
╰────────────────────────┬───────────────────────╯
│
╭──────────────────▼─────────────────╮
│ Configuration A / Configuration B │
╰──────────────────┬─────────────────╯
│
╭───────────────────────▼──────────────────────╮
│ nDCG / MRR / MAP + slice-level analysis │
╰──────────────────────────────────────────────╯A candidate-recall fixture can prove that document D was retrievable for query Q. It cannot prove that a new ranking function improved ordering across navigational, exploratory, exact-match, multilingual, and long-tail queries. Ranking evaluation aggregates evidence across that workload and makes regressions visible by query or slice.
This is the layer served by tools such as Rated Ranking Evaluator, Quepid, pyvespa, OpenSearch Search Relevance Workbench, and custom evaluation harnesses. They differ in workflow and platform support, but all support the same basic pattern: run controlled search configurations against a query set and compare the returned rankings using a defined set of relevance judgments.
Verification tells you whether an invariant holds. Relevance evaluation tells you whether one ranking is better against a chosen reference.
Where LLM Judgments Help
LLM judging becomes useful when the bottleneck is relevance labeling. A new corpus may have few judgments. A semantic retriever may surface candidates absent from an older judgment set. Human review may be too expensive to cover enough queries and documents for useful iteration.
An LLM can generate labels for those query-document pairs and make repeated offline comparison practical. OpenSearch Search Relevance Workbench, for example, retrieves the top results from a configured search, sends selected document fields and a prompt to an LLM, and stores the resulting judgments for later experiments.
That workflow solves a real problem by scaling judgment generation. It does not establish that the index is correct, that candidate recall is sufficient, or that the evaluation set represents the production workload.
The resulting judgments are also limited by the candidate set used to generate them. If candidates came from one retrieval method, the LLM can label only the evidence that method exposed. A different retriever may surface relevant query-document pairs that were never available for judgment.
The API call made judgment generation easier. The surrounding search and evaluation problems remain.
When Evaluation Becomes Theater
LLM evaluation becomes evaluation theater when sophisticated judgments and precise-looking metrics are used to support a claim the experiment was not designed to establish.
Consider this chain:
╭────────────────────╮
│ Wrong analyzer │
╰──────────┬─────────╯
│
╭───────────────────▼──────────────────╮
│ Relevant document is not retrieved │
╰───────────────────┬──────────────────╯
│
╭─────────────────────▼────────────────────╮
│ LLM judges only the surfaced results │
╰─────────────────────┬────────────────────╯
│
╭───────────▼──────────╮
│ nDCG@10 = 0.61 │
╰──────────────────────╯The score may be calculated correctly. It still does not diagnose the analyzer failure. Repeating the run with a stronger model, a more detailed rubric, or more judged examples improves the top layer while leaving the underlying defect untouched. Those repeated runs consume inference budget and engineering time without producing evidence about the boundary that failed.
The test does not support the claim being made. A final-ranking metric can show that relevant documents are missing or poorly placed. It cannot tell you whether a missing document was excluded before ranking or simply ranked below the evaluated cutoff. A judge can label what it sees. It cannot audit evidence that never reached it.
LLMs remain useful here. The failure begins when judgment generation is treated as a substitute for retrieval diagnosis and evaluation design.
Audit Results That Carry Consequences
Once the lower layers have been checked and judgments exist, test the evaluation setup itself. A result used to discard an obviously weak idea needs less evidence than a small offline gain used to justify a production migration.
For consequential decisions, test whether the winner survives defensible changes to the query sample, candidate pool, evidence presentation, judging prompt, or judge model. Version the corpus, queries, retrieval runs, pool construction, judging configuration, and metric so a later score change can be attributed to something specific.
Our companion article, What Is Your LLM Judge Actually Measuring?, examines this layer in detail: relevance-reference drift, unjudged documents, Judge–Pool Coupling, judgment stability, and practical robustness tests.
The question here comes one step earlier: should this problem have reached a relevance judge yet?
The Search Evaluation Ladder
The complete sequence is:
╭──────────────────────────────────────────────────────╮
│ Data + Configuration Correctness │
│ Is search behaving as intended? │
╰───────────────────────────┬──────────────────────────╯
│
╭───────────────────────────▼──────────────────────────╮
│ Candidate Recall │
│ Can the right documents reach ranking? │
╰───────────────────────────┬──────────────────────────╯
│
╭───────────────────────────▼──────────────────────────╮
│ Relevance Judgments │
│ Do we have the labels needed for comparison? │
╰───────────────────────────┬──────────────────────────╯
│
╭───────────────────────────▼──────────────────────────╮
│ Ranking Evaluation │
│ Is the ordering better against the judgments? │
╰───────────────────────────┬──────────────────────────╯
│
╭───────────────────────────▼──────────────────────────╮
│ Evaluation Audit │
│ Can we defend the result? │
╰──────────────────────────────────────────────────────╯The ladder shows which checks depend on earlier ones. A mature evaluation system may cover several layers, but each test answers a specific question and supports a limited set of conclusions.
One Change Across All Five Layers
Suppose a team introduces the synonym nyc → new york city for searches over municipal guidance. The same change creates a different question at every layer:
| Layer | Question |
|---|---|
| Data + Configuration Correctness | Was the synonym loaded, applied to the intended field, and kept away from fields where expansion would be harmful? |
| Candidate Recall | Can known documents about New York City enter the candidate set for relevant nyc queries? |
| Relevance Judgments | Do we have enough labels for newly surfaced documents to evaluate the two configurations? |
| Ranking Evaluation | Across representative nyc queries, did the new configuration improve ordering without creating regressions elsewhere? |
| Evaluation Audit | Does the measured improvement survive another query sample, candidate pool, or credible judging setup? |
These tests are complementary, not interchangeable. A passing synonym fixture does not establish ranking quality. A higher nDCG score does not prove the synonym was applied correctly. An LLM label does not show whether a better document was excluded before judgment.
Match the Tool to the Question
Each failure calls for a different instrument:
| Evaluation Job | Typical Mechanisms |
|---|---|
| Data + Configuration Correctness | Fixture documents, positive and negative assertions, analyzer and index inspection |
| Candidate Recall | Expected-document tests, Recall@K, exact-versus-ANN comparisons, filter and permission checks |
| Relevance Judgments | Domain experts, behavior-derived labels, LLM judging workflows |
| Ranking Evaluation | RRE, Quepid, pyvespa, OpenSearch Search Relevance Workbench, custom harnesses |
| Evaluation Audit | Sample, pool, judge, prompt, and presentation sensitivity tests; versioned runs |
A mature tool may support several evaluation jobs. The layer determines what its output can prove.
- Is the corpus represented and indexed correctly?
- Does query and linguistic processing behave as intended?
- Can known relevant documents reach the candidate depth available to ranking?
- Is the observed failure in candidate recall or ordering?
- Does the query set represent the workload affected by the change?
- Are additional relevance judgments actually needed?
- If an LLM supplies them, which documents and fields can it see?
- If the decision is consequential, has the evaluation itself been stress-tested and versioned?
LLMs make a costly part of relevance assessment easier to scale and repeat. That makes broader judgment coverage and faster offline iteration practical.
Search quality still starts below the judge: in how data is represented, how queries are interpreted, which candidates survive retrieval, and how those candidates are ranked. Strong evaluation programs use deterministic verification, candidate-recall tests, relevance judgments, ranking metrics, and evaluation audits at the appropriate layer.
At Searchplex, this is part of relevance engineering: start at the lowest layer that can explain the failure, then use the judge for the question it can actually answer.