Retrieval-Augmented Generation is often discussed as an LLM architecture. In practice, however, many weak RAG answers originate before the language model receives a prompt.

The retriever may return a document about the right subject but the wrong fact. Chunking may separate a rule from its exception. Metadata filters may select an obsolete version. A reranker may promote a topically similar but insufficient passage. The generator then turns this evidence into an answer that can sound coherent even when its factual basis is incomplete.

This creates a diagnostic problem. When retrieval and generation are evaluated only through the final response, a retrieval failure can look like a prompt failure, and a generation failure can be incorrectly blamed on the index.

The main argument of this article is that RAG should first be evaluated as a sequence of information retrieval decisions. Answer quality remains the final objective, but it should not be the first and only measurement.

The practical problem is evidence delivery

Consider an internal assistant that answers questions about product documentation, operating procedures, or company policies.

A user asks:

Can a customer cancel the service during the first 30 days without paying a termination fee?

The knowledge base contains:

• a current policy describing the 30-day cancellation rule,
• an older policy with different conditions,
• a general contract page that mentions termination fees but not the exception,
• a support article that paraphrases only part of the rule.

The system must do more than retrieve text about cancellation. It must retrieve the current passage containing the applicable rule, its conditions, and any relevant exceptions.

If the correct passage does not reach the prompt, the generator has several bad options. It can answer from incomplete evidence, rely on information stored in its parameters, combine incompatible passages, or refuse despite the answer being present elsewhere in the corpus.

The business cost depends on the application. A weak answer may create additional support work, expose an outdated policy, generate an unsupported recommendation, or cause a user to make the wrong decision.

A fluent answer can conceal a retrieval failure

Language models are optimized to produce plausible continuations. Fluency is therefore weak evidence that the retrieved context was correct.

A generator can summarize an irrelevant passage accurately. It can combine fragments that should not be combined. It can add a plausible statement that is absent from the supplied evidence. It can also ignore a relevant passage when the context is long, redundant, or badly ordered.

Retrieval-Augmented Generation does not remove the model's parametric knowledge, meaning information encoded in model parameters during training. Retrieval determines which external evidence is supplied at inference time, but the generator may still use information learned during pretraining or instruction tuning.

This combination of parametric and non-parametric knowledge was part of the original RAG formulation proposed by Lewis and colleagues. The non-parametric component provides retrieved documents, while the generator still retains knowledge represented in its parameters.

For a grounded application, this distinction matters. A factually correct answer can still violate the system contract when it is unsupported by the approved knowledge base. Conversely, a context-faithful answer can still be wrong if the retrieved context is obsolete, incomplete, or irrelevant.

At least three questions should therefore be evaluated separately:

A single end-to-end score cannot reliably identify which component caused the failure.

The retrieval pipeline is more than vector search

A production retrieval layer is usually a pipeline rather than one model. A simplified process is:

1. Parse source documents and preserve their structure.
2. Split the content into retrievable units.
3. Attach identifiers, timestamps, permissions, and other metadata.
4. Build one or more searchable representations.
5. Retrieve a broad candidate set.
6. Apply filters and access-control rules.
7. Merge results from multiple retrieval methods.
8. Rerank the candidates.
9. Select and order the final context.
10. Send that context to the generator.

Each stage changes the evidence available downstream.

Calling the entire process "vector search" hides several independent design choices and makes failures harder to diagnose.

Define retrieval success before selecting a metric

Let \(q\) denote a query and let \(C\) be the collection of retrievable passages. A retrieval system produces an ordered list:

$$R_k(q) = \left(c_1, c_2, \ldots, c_k\right)$$

where \(c_1\) is the highest-ranked passage and \(k\) is the number of candidates retained for evaluation or generation.

Let \(G(q)\) be the set of passages judged relevant to the query. Relevance should be defined at the same unit that the system retrieves. If the system retrieves chunks, document-level labels are often too coarse.

A document-level hit may look successful even when the returned chunk does not contain the answer. This is especially common in long manuals, policy collections, and API documentation where many passages share the same document identifier.

Hit Rate at k

Hit Rate at \(k\) checks whether at least one relevant passage appears in the first \(k\) results:

$$Hit@k(q) = \mathbf{1}\left\{R_k(q) \cap G(q) \neq \emptyset\right\}$$

This metric is easy to interpret. It answers whether the candidate set contains any known relevant evidence.

It is also easy to saturate. A query with five required passages and only one retrieved passage still counts as a hit.

Recall at k

Recall at \(k\) measures how much of the annotated relevant evidence was retrieved:

$$Recall@k(q) = \frac{|R_k(q) \cap G(q)|}{|G(q)|}$$

This is useful when several passages are needed to construct a complete answer. It requires reasonably complete relevance annotations, which can be expensive and ambiguous.

Mean Reciprocal Rank

Reciprocal rank uses the position of the first relevant result:

$$RR@k(q) = \begin{cases} \frac{1}{rank_q}, & \text{if a relevant passage occurs in the top } k \\ 0, & \text{otherwise} \end{cases}$$

Mean Reciprocal Rank, or MRR, averages this value across queries. It is useful when finding one relevant passage early is the main objective. It does not reward retrieving additional relevant passages after the first.

nDCG at k

Normalized Discounted Cumulative Gain, or nDCG, evaluates ranking quality by assigning more value to relevant passages near the top. It can also support graded relevance, for example distinguishing a directly answering passage from one that provides only background information.

nDCG is often more informative than Hit Rate when the generator receives only a small number of top-ranked chunks. Its value still depends on the quality and consistency of the relevance judgments.

A perfect hit rate can still hide a weak ranking

A small retrieval test may produce a perfect Hit Rate at 3 even when relevant passages are consistently returned at the bottom of the candidate set or only part of the required evidence is present.

Consider four retrieval runs that all place at least one relevant chunk in the top three for every query:

• a BM25-style lexical run,
• a dense retrieval run,
• a hybrid run,
• a reranked hybrid run.

Hit Rate at 3 cannot distinguish these systems when every run scores 1.0. Recall, reciprocal rank, and nDCG can still reveal differences in evidence coverage and ordering.

A retrieval evaluation

The following example uses six synthetic queries and manually defined passage-level relevance judgments. It does not evaluate a real retriever and should not be interpreted as evidence that one retrieval architecture is generally superior to another.

The example has a narrower purpose: to show why multiple retrieval metrics are required even when Hit Rate is perfect.

The rankings are fixed inputs. The code evaluates them exactly as written and uses only the Python standard library.

from math import log2


qrels = {
    "q1": {"rag-evaluation", "context-recall"},
    "q2": {"vector-indexing"},
    "q3": {"hybrid-search", "score-fusion"},
    "q4": {"metadata-freshness"},
    "q5": {"chunk-boundaries", "table-chunking"},
    "q6": {"cross-encoder-reranking"},
}

runs = {
    "bm25": {
        "q1": ["rag-evaluation", "prompt-design", "context-recall"],
        "q2": ["faiss-notes", "vector-indexing", "embedding-models"],
        "q3": ["score-fusion", "bm25", "hybrid-search"],
        "q4": ["metadata-freshness", "document-versioning", "filters"],
        "q5": ["table-chunking", "chunk-overlap", "tokenization"],
        "q6": ["bert-ranking", "cross-encoder-reranking", "bi-encoder"],
    },
    "dense": {
        "q1": ["faithfulness", "rag-evaluation", "answer-relevance"],
        "q2": ["embedding-models", "ann-search", "vector-indexing"],
        "q3": ["semantic-search", "hybrid-search", "reranking"],
        "q4": ["stale-content", "metadata-freshness", "document-versioning"],
        "q5": ["chunk-boundaries", "semantic-chunking", "layout-parsing"],
        "q6": [
            "bi-encoder",
            "candidate-generation",
            "cross-encoder-reranking",
        ],
    },
    "hybrid": {
        "q1": ["rag-evaluation", "context-recall", "faithfulness"],
        "q2": ["vector-indexing", "faiss-notes", "embedding-models"],
        "q3": ["hybrid-search", "score-fusion", "bm25"],
        "q4": [
            "metadata-freshness",
            "document-versioning",
            "stale-content",
        ],
        "q5": ["chunk-boundaries", "table-chunking", "layout-parsing"],
        "q6": [
            "bert-ranking",
            "cross-encoder-reranking",
            "candidate-generation",
        ],
    },
    "reranked": {
        "q1": ["context-recall", "rag-evaluation", "faithfulness"],
        "q2": ["vector-indexing", "embedding-models", "ann-search"],
        "q3": ["score-fusion", "hybrid-search", "reranking"],
        "q4": [
            "metadata-freshness",
            "document-versioning",
            "temporal-retrieval",
        ],
        "q5": ["table-chunking", "chunk-boundaries", "layout-parsing"],
        "q6": [
            "cross-encoder-reranking",
            "bert-ranking",
            "candidate-generation",
        ],
    },
}


def evaluate_run(
    qrels: dict[str, set[str]],
    run: dict[str, list[str]],
    k: int,
) -> dict[str, float]:
    hit_rates = []
    recalls = []
    reciprocal_ranks = []
    ndcgs = []

    for query_id, relevant_chunks in qrels.items():
        ranking = run[query_id][:k]

        hit_rates.append(
            float(
                any(
                    chunk in relevant_chunks
                    for chunk in ranking
                )
            )
        )

        retrieved_relevant = sum(
            chunk in relevant_chunks
            for chunk in ranking
        )
        recalls.append(
            retrieved_relevant / len(relevant_chunks)
        )

        reciprocal_rank = 0.0
        for rank, chunk in enumerate(ranking, start=1):
            if chunk in relevant_chunks:
                reciprocal_rank = 1.0 / rank
                break
        reciprocal_ranks.append(reciprocal_rank)

        dcg = sum(
            float(chunk in relevant_chunks) / log2(rank + 1)
            for rank, chunk in enumerate(ranking, start=1)
        )

        ideal_relevant = min(len(relevant_chunks), k)
        idcg = sum(
            1.0 / log2(rank + 1)
            for rank in range(1, ideal_relevant + 1)
        )
        ndcgs.append(dcg / idcg)

    return {
        f"HitRate@{k}": sum(hit_rates) / len(hit_rates),
        f"Recall@{k}": sum(recalls) / len(recalls),
        f"MRR@{k}": (
            sum(reciprocal_ranks) / len(reciprocal_ranks)
        ),
        f"nDCG@{k}": sum(ndcgs) / len(ndcgs),
    }


for system_name, run in runs.items():
    metrics = evaluate_run(qrels, run, k=3)

    formatted = " ".join(
        f"{metric_name}={value:.3f}"
        for metric_name, value in metrics.items()
    )

    print(f"{system_name:9s} {formatted}")

The following output was produced by executing the code:

bm25      HitRate@3=1.000 Recall@3=0.917 MRR@3=0.833 nDCG@3=0.786
dense     HitRate@3=1.000 Recall@3=0.750 MRR@3=0.528 nDCG@3=0.503
hybrid    HitRate@3=1.000 Recall@3=1.000 MRR@3=0.917 nDCG@3=0.938
reranked  HitRate@3=1.000 Recall@3=1.000 MRR@3=1.000 nDCG@3=1.000

What the synthetic result demonstrates

All four runs achieve a perfect Hit Rate at 3. Based only on this metric, they are indistinguishable.

The remaining metrics show different behavior.

The dense run retrieves at least one relevant passage for every query, but it often places that passage at rank two or three. It also misses some secondary relevant passages. Its Recall at 3, MRR at 3, and nDCG at 3 are therefore lower.

The BM25 run performs better in this constructed example because several rankings place exact technical names near the top. This result follows from the manually defined inputs. It is not a benchmark comparison between a real BM25 implementation and a real embedding model.

The hybrid run retrieves all annotated evidence within the first three positions and generally ranks it near the top. The reranked run places a relevant chunk first for every query and orders all annotated relevant chunks before the irrelevant candidates.

The example supports a limited conclusion:

A perfect retrieval hit rate does not imply that the retrieved context has complete coverage or optimal ordering.

It does not support the stronger conclusion that hybrid retrieval or reranking will always improve a production system. That claim requires evaluation on representative queries, documents, relevance judgments, latency constraints, and actual retrieval implementations.

Choose k according to the system decision

The value of \(k\) should not be selected only because a larger candidate set improves recall.

There are usually at least two different cutoffs:

• Candidate k: the number of passages passed from initial retrieval to a reranker.
• Context k: the number of passages finally passed to the generator.

A large candidate set can improve the reranker's opportunity to find relevant evidence. A large final context can increase token cost, latency, redundancy, and contradiction risk.

More context is not automatically better. Research on long-context language models has shown that performance can depend on where relevant information appears in the input. In the experiments reported by Liu and colleagues, models often performed better when relevant information appeared near the beginning or end of the context than when it appeared in the middle.

The correct value of \(k\) is therefore an empirical choice. It should be evaluated jointly with reranking, deduplication, context ordering, answer quality, and latency.

Chunking defines the retrieval unit

Chunking is sometimes treated as a preprocessing detail. In a RAG system, it defines the objects that can be retrieved and therefore changes the retrieval problem itself.

A chunk that is too small may omit the qualifications needed to interpret a fact. A chunk that is too large may contain several topics, reducing ranking precision and consuming unnecessary context tokens.

There is no universally optimal chunk length. The appropriate unit depends on document structure and query behavior.

Structure can matter more than token count

A fixed token window may be adequate for uniform prose. It is often a weak default for documents containing:

• section hierarchies,
• tables and captions,
• API methods and parameter descriptions,
• legal clauses and exceptions,
• code blocks,
• questions and answers,
• versioned procedures.

For these documents, structure-aware chunking can preserve relations that arbitrary token boundaries destroy.

A useful chunk may contain a section title, the local passage, source metadata, and a link to a parent section. Overlap can reduce boundary failures, but excessive overlap creates near-duplicate candidates and can make retrieval metrics appear better without increasing evidence diversity.

Evaluate chunking through queries

I would not select chunk size using only document statistics. I would compare chunking strategies on a fixed query set and measure:

• passage-level Recall at k,
• ranking quality,
• duplicate rate in the selected context,
• answer-fact coverage,
• context token count,
• latency and index size.

The best strategy is the one that supports the required decisions under the system constraints, not the one that produces the cleanest distribution of chunk lengths.

Lexical and dense retrieval fail differently

Lexical retrieval preserves explicit terms

BM25 and related lexical methods score documents using term occurrence, document length, and corpus-level term statistics.

They remain useful for queries containing:

• error codes,
• product identifiers,
• legal clause numbers,
• function names,
• table and field names,
• rare domain terminology.

A lexical system can fail when the user and the document express the same concept with different vocabulary.

Dense retrieval connects semantic paraphrases

Dense retrieval represents queries and passages as learned vectors. It can retrieve semantically related passages even when the query and passage share few exact words.

This is useful when users paraphrase documentation, use informal language, or describe a concept rather than naming it directly.

Dense retrieval can still fail on exact identifiers, poorly represented domain terminology, distribution shift, or distinctions that are small linguistically but important operationally.

The performance of a dense retriever also depends on its training data, embedding objective, input length, language coverage, and similarity function. "Using embeddings" does not define one reproducible retrieval method.

The BEIR benchmark illustrates why broad claims about retrieval architectures should be treated cautiously. Retrieval methods can behave differently across question answering, fact verification, citation prediction, duplicate detection, and other retrieval tasks.

Hybrid retrieval needs an explicit fusion rule

Hybrid retrieval combines signals from lexical and semantic systems. The objective is usually to preserve exact-match behavior while recovering semantically relevant passages.

The combination is not as simple as adding raw scores. BM25 scores and vector similarities have different ranges and distributions. Their numerical values are not automatically comparable.

Common approaches include:

• normalizing scores before combining them,
• learning a weighted scoring function,
• merging ranked lists with Reciprocal Rank Fusion,
• taking the union of candidates and delegating final ordering to a reranker.

The fusion method should be treated as part of the evaluated system. A hybrid label without a documented fusion rule is not enough to reproduce a result.

Reranking trades throughput for precision

A first-stage retriever must search a large collection efficiently. It often scores the query and passage independently or uses relatively inexpensive lexical calculations.

A cross-encoder reranker evaluates the query and candidate passage together. This allows richer token-level interaction, but the model must run separately for each candidate.

A common architecture is therefore:

$$\text{broad retrieval} \rightarrow \text{candidate set} \rightarrow \text{reranking} \rightarrow \text{final context}$$

Reranking is useful when the initial retriever has adequate recall but poor top-rank precision. It cannot recover a relevant passage that never entered the candidate set.

The evaluation should report both quality and cost:

• Recall at the candidate cutoff,
• nDCG or MRR after reranking,
• context-level recall at the final cutoff,
• p50 and p95 latency,
• throughput,
• compute or API cost.

A reranker that improves nDCG but violates the latency budget may not improve the actual system.

Freshness and permissions are retrieval correctness

Relevance is not only semantic similarity. In many systems, a passage is valid only when it satisfies additional constraints.

A document may be topically relevant but unusable because it is:

• obsolete,
• superseded by a newer policy,
• outside the user's permissions,
• associated with another tenant,
• written for another product version,
• outside the requested time period or jurisdiction.

These conditions should be represented in the relevance judgments and test cases. Otherwise, a retrieval system can score well while repeatedly returning evidence that must not be used.

Filtering errors also have two directions. A missing filter may expose invalid evidence. An overly strict or inaccurate filter may eliminate the only relevant passage.

For time-sensitive corpora, the evaluation set should include queries where older and newer sources conflict. The expected result should identify not only the relevant topic but also the valid version.

Retrieval metrics do not prove answer quality

Good retrieval is necessary for many grounded questions, but it is not sufficient.

A retrieved passage may be relevant without containing every fact required by the answer. The generator may misread a negation, combine incompatible sources, omit an important condition, or produce an unsupported conclusion.

End-to-end evaluation should therefore add generation-level dimensions after retrieval has been measured.

Frameworks such as RAGAS and ARES formalize several of these dimensions and can accelerate evaluation. Their outputs should not be treated as objective ground truth. Automated evaluators can have domain bias, prompt sensitivity, calibration errors, and failure modes of their own.

A small, carefully reviewed human-labeled set remains useful for validating whether an automated evaluator agrees with the judgments that matter in the application.

Build the evaluation set around failures

A retrieval test set should not contain only clean questions with one obvious answer passage.

I would include several query classes:

• Exact identifier queries: error codes, function names, SKUs, or clause numbers.
• Paraphrase queries: the user uses different vocabulary from the document.
• Multi-passage queries: the answer requires evidence from more than one section.
• Temporal queries: a newer source supersedes an older one.
• Contradictory-source queries: the corpus contains incompatible statements.
• Table queries: the answer depends on headers and cell relationships.
• Permission-sensitive queries: only part of the corpus is accessible.
• Unanswerable queries: the requested fact is absent.
• Ambiguous queries: clarification is preferable to immediate retrieval.
• Adversarial wording: the query contains misleading assumptions or copied instructions.

Each case should define what successful retrieval means. Depending on the application, the annotation may include:

• required passages,
• acceptable alternative passages,
• forbidden or obsolete passages,
• expected answer facts,
• the intended answerability decision,
• metadata constraints,
• the reason the case is difficult.

The evaluation set should also record meaningful segments such as query type, source type, language, product, customer group, and time period. An aggregate score can hide a severe failure in one operationally important segment.

A practical evaluation sequence

I would evaluate a RAG system in the following order:

1. Define the system contract. Specify the corpus, eligible sources, freshness rules, answerability policy, citation requirements, latency budget, and allowed use of parametric knowledge.
2. Create a versioned query set. Include ordinary questions, difficult cases, unanswerable cases, and known production failures.
3. Annotate retrieval targets. Record passage-level relevance, required facts, invalid sources, and metadata constraints.
4. Establish simple baselines. Compare lexical, dense, and straightforward hybrid retrieval before adding complex orchestration.
5. Measure candidate retrieval. Use Recall at a relatively broad cutoff to determine whether reranking has access to the required evidence.
6. Measure final ranking. Use MRR, nDCG, and context-level recall at the cutoff sent to the generator.
7. Inspect errors by segment. Separate exact-match, paraphrase, table, temporal, multi-passage, and unanswerable queries.
8. Evaluate generation. Measure correctness, claim-level faithfulness, completeness, citation support, and abstention behavior.
9. Measure operational cost. Record latency distributions, token use, index size, throughput, and reranking cost.
10. Run controlled experiments. Change one major component at a time and preserve the same evaluation protocol.
11. Validate online. Monitor production failures and add reviewed cases back to the evaluation set.

This sequence does not eliminate end-to-end testing. It makes end-to-end results interpretable.

The main conclusion

RAG quality does not begin with prompt wording. It begins with a precise definition of acceptable evidence and a retrieval pipeline that can deliver that evidence reliably.

The language model still matters. It must interpret, combine, cite, and sometimes reject the retrieved material. Those capabilities cannot compensate reliably for a missing passage, an obsolete source, or an incorrect access filter.

The most useful evaluation design therefore preserves the structure of the system:

$$\text{corpus} \rightarrow \text{retrieval} \rightarrow \text{context} \rightarrow \text{answer}$$

Measure each transition, then evaluate the complete result.

Key takeaways

• RAG should be evaluated as retrieval, context construction, and generation rather than as one opaque response function.
• Retrieval controls the external evidence supplied to the generator, but it does not remove the model's parametric knowledge.
• Document-level Hit Rate can conceal missing answer passages, incomplete evidence, and poor ranking.
• Recall, MRR, and nDCG answer different retrieval questions and should be selected according to the intended context cutoff.
• Chunking, metadata, fusion, reranking, and context ordering are modeling and system-design decisions.
• Automated RAG evaluators are useful tools, but their judgments should be calibrated against reviewed examples.
• A reliable production system must include stale-source handling, permissions, abstention, latency, and citation correctness.

Sources

Retrieval-Augmented Generation

1. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Kuttler, H., Lewis, M., Yih, W., Rocktaschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474.

Lexical and Dense Retrieval

1. Robertson, S., and Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333-389.
2. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of EMNLP 2020, 6769-6781.

Retrieval Evaluation and Long Context

1. Thakur, N., Reimers, N., Ruckle, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. Proceedings of the NeurIPS 2021 Datasets and Benchmarks Track.
2. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157-173.

RAG Evaluation

1. Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. (2024). RAGAS: Automated Evaluation of Retrieval Augmented Generation. Proceedings of the EACL 2024 System Demonstrations, 150-158.
2. Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. Proceedings of NAACL 2024, 338-354.
3. Stolfo, A. (2024). Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study. Findings of NAACL 2024, 1537-1552.