Embedding drift is a quiet failure mode. The application may continue returning results, the vector index may respond within its latency target, and the language model may still produce fluent answers. At the same time, the retrieval layer can begin selecting different and less useful evidence.

This matters for RAG, semantic search, recommendations, deduplication, clustering, and any workflow in which vector distance influences a downstream decision.

The difficulty is that embedding drift does not describe one specific failure. The document population may change while the encoder remains fixed. Preprocessing or chunking may change. A new embedding model may produce a different representation space. A migration may also update document vectors without updating the query encoder.

These cases require different responses. Distribution movement can be legitimate, while a small integration error can make query and document vectors incompatible.

The main argument of this article is that embedding monitoring must combine representation diagnostics with fixed-query retrieval evaluation. Centroid movement, vector norms, and corpus-neighbor stability are useful signals, but none of them establishes that users still receive the correct results.

The production problem is preserving retrieval behavior

Consider an internal RAG assistant that retrieves technical policies before generating an answer.

The retrieval service stores document chunks as vectors. Each user question is encoded into the same representation space, compared with the stored vectors, and used to select the top passages.

A team later upgrades the embedding model. The corpus is re-embedded during a scheduled migration, but one application instance continues encoding queries with the previous model version.

The vector database remains healthy. Every query returns five passages. Latency remains unchanged. The generator receives valid text and produces a well-formed answer.

The failure appears only in relevance. Query and document vectors now belong to different representation spaces, so their similarity scores no longer have the intended meaning.

The monitoring objective is therefore not to prove that vectors still exist. It is to verify that the representation and retrieval path still support the intended task.

Dense retrieval depends on a shared representation space

Dense retrieval commonly uses separate encoders for queries and documents. The resulting vectors are compared with an inner product, cosine similarity, or another distance function.

Let \(f_q(q)\) be the query encoder and \(f_d(d)\) the document encoder. A retrieval score can be written as:

$$ score(q,d) = f_q(q)^\top f_d(d) $$

The exact implementation depends on the model and its training objective. The important assumption is that the query and document encoders produce compatible representations.

Dense Passage Retrieval uses a dual-encoder architecture to represent questions and passages as dense vectors for open-domain retrieval.

Retrieval-Augmented Generation connects a generator with an external dense vector index. In such a workflow, retrieval errors affect the evidence supplied to generation.

The representation space is therefore part of the runtime contract. The index version, query encoder, document encoder, normalization rule, distance function, chunking policy, and preprocessing version must remain compatible.

Embedding drift describes several different changes

The term embedding drift is often used too broadly. A useful monitoring plan separates three primary sources.

Corpus drift can be legitimate. A policy archive should change when new policies are published. Pipeline changes can also be intentional when they remove boilerplate or improve chunk boundaries.

A representation migration is different. Vectors produced by different model versions should not be assumed to be directly comparable. The fact that two arrays have the same dimension does not establish that their coordinates describe the same space.

Centroid movement is only a distribution signal

For document embeddings \(\mathbf{x}_1, \ldots, \mathbf{x}_n\), the centroid is:

$$ \mu = \frac{1}{n} \sum_{i=1}^{n} \mathbf{x}_i $$

In this notation, \(\mu\) is the centroid vector of the embedding collection.

A simple shift statistic compares the current and reference centroids:

$$ D_{\mathrm{centroid}} = \left\| \mu_{\mathrm{current}} - \mu_{\mathrm{reference}} \right\|_2 $$

A large value shows that the average representation moved under the selected coordinate system.

It does not show why the centroid moved, whether document relationships changed, or whether retrieval quality declined.

The value also depends on the representation coordinate system. Two spaces can encode the same pairwise relationships while having different coordinates and different centroids.

Geometry can move while retrieval remains unchanged

Suppose every embedding is transformed by the same orthogonal matrix \(Q\), where:

$$ Q^\top Q = I $$

For vectors \(\mathbf{x}\) and \(\mathbf{y}\), the inner product after transformation is:

$$ (Q\mathbf{x})^\top(Q\mathbf{y}) = \mathbf{x}^\top Q^\top Q\mathbf{y} = \mathbf{x}^\top\mathbf{y} $$

The pairwise scores are preserved. If queries and documents are transformed together, their ranking can remain identical even though the coordinates and centroid have changed.

If only the documents are transformed while queries remain in the old space, document-to-document geometry can still remain stable. Query-to-document retrieval can nevertheless fail.

This construction is an analytical example. Real embedding-model migrations are not generally exact orthogonal rotations. It isolates why distribution and corpus-geometry metrics cannot replace an end-to-end query benchmark.

Useful monitoring signals answer different questions

The final signal is the most directly connected to application quality. It is also the most expensive because it requires a maintained evaluation set.

A synthetic migration experiment

The following experiment creates six synthetic topics, fifty documents per topic, and four benchmark queries per topic.

Documents and queries are generated around topic-specific centers and normalized. A document is considered relevant when it belongs to the same synthetic topic as the query.

The experiment compares the reference representation, a coordinated migration in which documents and queries receive the same orthogonal transformation, and a partial migration in which only document vectors are transformed.

The coordinated migration preserves all query-document similarities. The partial migration keeps document-to-document geometry intact but breaks compatibility between query and document representations.

The data and representation changes are synthetic. They demonstrate monitoring behavior and are not measurements from a real embedding model.

import numpy as np

rng = np.random.default_rng(42)
n_topics = 6
docs_per_topic = 50
queries_per_topic = 4
dimension = 24
top_k = 5

centers = rng.normal(size=(n_topics, dimension))
centers /= np.linalg.norm(centers, axis=1, keepdims=True)

documents = []
document_topics = []

for topic in range(n_topics):
    vectors = centers[topic] + 0.18 * rng.normal(
        size=(docs_per_topic, dimension)
    )
    vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
    documents.append(vectors)
    document_topics.extend([topic] * docs_per_topic)

documents = np.vstack(documents)
document_topics = np.array(document_topics)

queries = []
query_topics = []

for topic in range(n_topics):
    vectors = centers[topic] + 0.10 * rng.normal(
        size=(queries_per_topic, dimension)
    )
    vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
    queries.append(vectors)
    query_topics.extend([topic] * queries_per_topic)

queries = np.vstack(queries)
query_topics = np.array(query_topics)

rotation_seed = rng.normal(size=(dimension, dimension))
rotation, _ = np.linalg.qr(rotation_seed)


def top_k_ids(query_vectors, document_vectors):
    similarities = query_vectors @ document_vectors.T
    return np.argsort(-similarities, axis=1)[:, :top_k]


def retrieval_metrics(query_vectors, document_vectors):
    ids = top_k_ids(query_vectors, document_vectors)
    relevant = document_topics[ids] == query_topics[:, None]
    reciprocal_ranks = []

    for row in relevant:
        positions = np.flatnonzero(row)
        reciprocal_ranks.append(
            0.0
            if positions.size == 0
            else 1.0 / (positions[0] + 1)
        )

    return {
        "ids": ids,
        "hit_rate": relevant.any(axis=1).mean(),
        "precision": relevant.mean(),
        "mrr": np.mean(reciprocal_ranks),
    }


def document_neighbors(vectors):
    similarities = vectors @ vectors.T
    np.fill_diagonal(similarities, -np.inf)
    return np.argsort(-similarities, axis=1)[:, :top_k]


def overlap_at_k(reference_ids, current_ids):
    return np.mean([
        len(set(reference_row) & set(current_row)) / top_k
        for reference_row, current_row
        in zip(reference_ids, current_ids)
    ])


reference = retrieval_metrics(queries, documents)
reference_neighbors = document_neighbors(documents)

scenarios = {
    "reference": (queries, documents),
    "coordinated_migration": (
        queries @ rotation,
        documents @ rotation,
    ),
    "partial_migration": (
        queries,
        documents @ rotation,
    ),
}

for name, (scenario_queries, scenario_documents) in scenarios.items():
    metrics = retrieval_metrics(
        scenario_queries,
        scenario_documents,
    )
    centroid_shift = np.linalg.norm(
        scenario_documents.mean(axis=0)
        - documents.mean(axis=0)
    )
    query_stability = overlap_at_k(
        reference["ids"],
        metrics["ids"],
    )
    neighbor_stability = overlap_at_k(
        reference_neighbors,
        document_neighbors(scenario_documents),
    )

    print(
        f"{name:22s} "
        f"centroid_shift={centroid_shift:.3f} "
        f"query_top5_stability={query_stability:.3f} "
        f"corpus_neighbor_stability={neighbor_stability:.3f} "
        f"HitRate@5={metrics['hit_rate']:.3f} "
        f"Precision@5={metrics['precision']:.3f} "
        f"MRR@5={metrics['mrr']:.3f}"
    )

The following output was produced by executing the code:

reference              centroid_shift=0.000 query_top5_stability=1.000 corpus_neighbor_stability=1.000 HitRate@5=1.000 Precision@5=1.000 MRR@5=1.000
coordinated_migration  centroid_shift=0.398 query_top5_stability=1.000 corpus_neighbor_stability=1.000 HitRate@5=1.000 Precision@5=1.000 MRR@5=1.000
partial_migration      centroid_shift=0.398 query_top5_stability=0.000 corpus_neighbor_stability=1.000 HitRate@5=0.125 Precision@5=0.050 MRR@5=0.104

The coordinated migration moves the space without harming retrieval

The coordinated migration produces a centroid shift of 0.398.

If centroid movement were interpreted automatically as retrieval degradation, this scenario would generate a false alarm. Query top-5 stability remains 1.000, meaning every benchmark query receives exactly the same five documents as before.

Corpus-neighbor stability, HitRate at 5, Precision at 5, and MRR at 5 also remain 1.000.

The coordinate system changed, but the relationships required by retrieval did not.

The partial migration looks healthy under corpus-only diagnostics

The partial migration produces the same centroid shift of 0.398. Corpus-neighbor stability also remains 1.000 because every document received the same orthogonal transformation.

A monitoring system based only on document distributions and document-to-document neighbors would report the coordinated and partial migrations as equivalent.

The query benchmark reveals the failure. Query top-5 stability falls to 0.000, so none of the previous top-five result identifiers remain in the result lists.

HitRate at 5 falls to 0.125. Precision at 5 falls to 0.050, and MRR at 5 falls to 0.104.

The vector index still returns results, but the results no longer represent the intended synthetic topics reliably.

Neighbor stability must include the query side

Document-neighbor stability is useful for clustering, recommendations, duplicate detection, and corpus analysis. It can expose changes in local document geometry.

It does not verify that live queries use a compatible encoder.

For search and RAG, the monitoring path should compare fixed query results before and after any change to the encoder, preprocessing, chunking, index, filtering, or ranking logic.

For reference result set \(R_k^{old}(q)\) and current result set \(R_k^{new}(q)\), top-k overlap can be defined as:

$$ Stability@k(q) = \frac{ \left| R_k^{old}(q) \cap R_k^{new}(q) \right| }{ k } $$

Averaging this value across benchmark queries measures ranking continuity.

High stability is not automatically good. A new system can preserve the same poor results. Stability should be interpreted together with relevance metrics.

Task metrics decide whether movement matters

The most important evaluation question is whether expected evidence remains retrievable.

For a query with at least one relevant document, HitRate at \(k\) is one when any relevant result appears in the first \(k\) positions:

$$ HitRate@k = \frac{1}{|Q|} \sum_{q \in Q} \mathbf{1} \left\{ R_k(q) \cap G(q) \neq \varnothing \right\} $$

Precision at \(k\) measures the proportion of returned results that are relevant:

$$ Precision@k = \frac{1}{|Q|} \sum_{q \in Q} \frac{ \left| R_k(q) \cap G(q) \right| }{ k } $$

MRR focuses on the rank of the first relevant result.

The appropriate metric depends on the interface. A RAG system may need at least one authoritative passage. A research search interface may need several diverse relevant documents. A duplicate detector may require a reliable first match and a calibrated threshold.

The benchmark should represent production queries

A fixed query set is only useful when it represents the retrieval decisions that matter.

The benchmark should include common queries, high-risk queries, rare terminology, multilingual cases, recent documents, and older sources that must remain discoverable.

The MTEB benchmark evaluates text embeddings across multiple task families rather than assuming that one embedding quality measure transfers to every application.

The same principle applies in production. A model that performs well on semantic textual similarity is not automatically the best option for a specific retrieval corpus.

A benchmark should also preserve difficult negatives. Easy unrelated documents may make retrieval appear reliable while the system still confuses near-duplicate policies, similar product versions, or documents from the wrong jurisdiction.

Corpus composition should be monitored separately

New content can move the embedding distribution without indicating a defect.

A documentation corpus may add a new product family, another language, or a large archive of incident reports. Centroid and cluster statistics may change because the corpus now contains legitimate new information.

Composition monitoring should report source, language, document type, template, time range, and chunk-count changes.

The investigation should determine whether the new content is expected and whether it affects important retrieval queries. Distribution movement without task degradation may require no corrective action.

Preprocessing changes are representation changes

Embedding monitoring should not be limited to model upgrades.

Lowercasing, Unicode normalization, boilerplate removal, table conversion, OCR correction, chunk size, overlap, and metadata prefixes can all change the input passed to the encoder.

A chunking change can improve local context while reducing the ability to retrieve complete procedures. Removing boilerplate can improve discrimination while accidentally deleting policy qualifiers.

The preprocessing version should therefore be stored with every vector and included in migration diagnostics.

Model migrations require a compatibility boundary

A vector index should not silently mix embeddings from incompatible versions.

A safer record identifies the embedding model, preprocessing version, dimension, normalization rule, and index build.

embedding_record = {
    "asset_id": "policy-214-chunk-08",
    "embedding_model": "retrieval-encoder-v5",
    "preprocessing_version": "policy-cleaning-v3",
    "dimension": 1024,
    "normalized": True,
    "index_version": "policies-2026-07-30",
}

The query service should send the expected representation version with each request. The retrieval service should reject or route incompatible combinations rather than calculate a score because the dimensions happen to match.

A coordinated migration should update the corpus index, query encoder, retrieval benchmark, and rollback path as one release unit.

Dual indexing makes migrations testable

For important systems, a new embedding model can be deployed through parallel indexes.

The existing query is encoded with both model versions. Each index returns results independently, and the system compares relevance, latency, stability, and source coverage before promotion.

This approach increases temporary compute and storage cost, but it avoids an irreversible in-place migration.

Production shadow traffic can reveal query patterns missing from the offline benchmark. Sensitive content and user identifiers still require appropriate logging and retention controls.

Embedding thresholds must be recalibrated after migration

Similarity values are model-specific. A threshold such as 0.80 has no universal meaning across embedding models, preprocessing versions, or normalization choices.

A migration can preserve ranking quality while changing the numerical score distribution.

Systems using absolute thresholds for duplicate detection, abstention, routing, or filtering should therefore recalibrate those thresholds on reviewed positive and negative examples.

Copying the previous threshold into a new representation space can change acceptance rates even when nearest-neighbor ranking improves.

Embedding monitoring should use three control layers

1. Representation control: monitor model, preprocessing, norms, centroid movement, composition, and index version.
2. Retrieval control: measure fixed-query stability, relevance, source coverage, hard negatives, and score distributions.
3. Release control: test coordinated migration, dual indexes, compatibility checks, threshold calibration, and rollback.

The layers serve different purposes. Representation metrics help identify what changed. Retrieval metrics show whether the change matters. Release controls prevent incompatible components from reaching the same request path.

Alerts should distinguish movement from degradation

A representation alert should not trigger automatic rollback solely because the centroid moved.

A useful policy can classify an event as expected movement, investigation required, or confirmed retrieval regression.

The thresholds should be derived from benchmark variability and the cost of retrieval errors. They should not be copied from unrelated systems.

RAG evaluation should separate retrieval and generation

When a RAG answer becomes worse, the language model should not be assumed to be the failing component.

The evaluation trace should preserve the query, retrieved identifiers, passage versions, retrieval scores, context order, model version, and final answer.

If expected evidence is missing, the primary problem belongs to retrieval or corpus construction. If the evidence is present but the answer contradicts it, generation or answer evaluation requires investigation.

This separation prevents prompt changes from masking an index or embedding migration failure.

What embedding diagnostics cannot prove

Stable retrieval metrics on a fixed benchmark do not guarantee stable behavior for every production query.

Relevance labels can become outdated when the corpus or product changes. A fixed query suite can also miss new terminology and emerging user intents.

Centroid, neighborhood, and cluster metrics depend on the sample and representation choices. High-dimensional projections created for visualization can hide or exaggerate local changes.

Monitoring reduces the chance that a silent representation change reaches users unnoticed. It does not establish that the embedding space captures every semantic distinction required by the application.

Key takeaways

• Embedding drift monitoring must separate corpus changes, preprocessing changes, and representation migrations because they require different responses.
• Centroid shift and corpus-neighbor stability cannot verify query-document compatibility; fixed-query retrieval benchmarks are required.
• Embedding migrations should version both sides of retrieval, recalibrate thresholds, run parallel evaluation, and block mixed representation spaces.

Sources

1. Karpukhin, V., Oguz, B., Min, S., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of EMNLP 2020, 6769-6781.
2. Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474.
3. Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. (2023). MTEB: Massive Text Embedding Benchmark. Proceedings of EACL 2023, 2006-2029.