Evaluation & Operations
Know where quality is lost. Know whether a change helped. Keep that evidence connected to the system in production.
One bad result. Many possible causes.
The final result is where a failure becomes visible, not necessarily where it originates.
The correct source may be unavailable, structure may be lost during ingestion, the wrong unit may compete for retrieval, relevant candidates may be filtered out, or a sound candidate pool may be ranked poorly. In RAG, good evidence can still be selected or used incorrectly downstream.
Attribution determines the remedy. Source access, preparation, representation, retrieval, ranking, and downstream use are different decisions. Measuring them separately prevents a downstream fix from masking an upstream failure.
Evaluate each layer for the job it performs.
A single end-to-end score can show that behaviour changed. It cannot identify which system boundary improved or failed.
Source Connectivity & Governance
Was the correct, authorised, current source available?
- Source coverage
- Permission correctness
- Version correctness
- Freshness
- Authority
Check source coverage before ranking: no ranker can recover a source it was not allowed to see or an update that never arrived.
Content Preparation
Did ingestion preserve the information retrieval depends on?
- Parsing completeness
- Structural retention
- Metadata quality
- Extraction quality
- Duplication checks
Compare indexed content with the source. Headings, tables, clauses, identifiers, language, and relationships can disappear even when ingestion succeeds.
Retrieval Modeling
Can the right object compete for retrieval?
- Unit eligibility
- Granularity analysis
- Field coverage
- Sparse representation
- Dense representation
A document can be indexed while the useful article, product variant, case, paragraph, or clause remains too coarse or fragmented to retrieve reliably.
Retrieval
Did the right candidates enter the pool?
- Recall@K
- Filter correctness
- Lexical/vector contribution
- Fusion behaviour
- Missed candidates
Measure what each lexical, vector, filtered, or federated path admits and misses. Evaluate RRF or weighted fusion by the candidates it promotes and suppresses.
Ranking
Did the strongest eligible candidates rise to the top?
- nDCG
- MRR
- Precision@K
- Reranking impact
- Stage contribution
Judge ranking against the candidates and features it received. Better ordering of an incomplete pool may not improve the results users see.
Evaluation & Operations
Can evidence be compared, traced, regression-tested, and released safely?
- Evaluation sets
- Segmentation
- Release comparisons
- Regression checks
- Traceability
This layer records the evaluated workload, system version, change, and outcome so the comparison can be repeated before release and investigated afterwards.
When retrieval feeds downstream systems
RAG and agentic systems introduce decisions after ranking that need their own evidence.
Keep evidence retrieval, context selection, grounding, and final-answer behaviour separable. One answer score cannot show whether generation failed or the required evidence never reached it.
Enterprise RAG →Evaluation may span repeated retrieval calls, source routing, changing context, and downstream actions. Preserve enough of the trajectory to see how one retrieval decision affected the next.
Agentic AI →Build a baseline around the workload.
Reproduce the failure, separate likely causes, and measure the proposed change. Do not attempt to represent every future query.
Segment where the contract changes.
Known-item lookup and exploratory discovery do not make the same promise. Neither do current-version and historical research, public and permissioned content, or monolingual and cross-language retrieval. Segment around differences that materially change expected behaviour, not every field available in the analytics system.
Sample common behaviour by traffic. Add rare, high-impact failures such as permission leaks, stale versions, and failed identifier lookup. Do not draw conclusions about query classes the set does not cover.
Collect enough evidence for the decision.
- 01
Reproduce the failure the team is trying to explain.
- 02
Distinguish plausible causes rather than confirm the first hunch.
- 03
Measure the behaviour the proposed change should improve.
- 04
Expose important adjacent regressions and production constraints.
The baseline also identifies what was measured: query processing, filters, candidate depth, retrieval configuration, rank profile, and model version where relevant. Without that system state, a later score may be repeatable as a calculation but not reproducible as retrieval behaviour.
A useful baseline may contain dozens of carefully chosen queries rather than thousands of labels. Add coverage only when it can change the diagnosis, release decision, or regression boundary.
Carry the evidence into production.
Offline evaluation tests the retrieval change under a controlled comparison. Production validation tests its effect under real workload conditions.
Offline comparison
Compare the change with the baseline by relevant query class. Inspect improvements, regressions, and unexpected movement rather than relying on one aggregate delta.
Controlled validation
Use A/B testing to measure a product outcome, interleaving to compare ranked lists sensitively, or shadow evaluation and phased rollout when exposure carries more risk.
Traceable release
Keep the retrieval path, configuration, model or rank-profile version, and release identity reconstructable so a production result can be tied to the system that produced it.
New baseline
Monitor the intended product outcome together with relevance, latency, cost, and policy guardrails. A validated release becomes the comparison point for the next change.
Keep each release traceable through query interpretation, permissions, retrieval, ranking, result, latency, and release ID. Add production regressions to the evaluation set with the configuration that produced them.
Evidence from production systems
Nextens: versioned knowledge and inspectable retrieval
The redesign addressed version handling, metadata, duplicate suppression, query assistance, and ranking predictability in a high-trust tax research system.
Read case study →CuratedAI: exact and exploratory retrieval
Hybrid multilingual retrieval and finer retrievable units improved known-item lookup and original-language discovery where semantic-only retrieval was insufficient.
Read case study →