Image and video models are often presented through generation: a prompt becomes a picture, a clip, or a stylized scene. Generation is important, but many production systems create more value by understanding existing media.

Visual search, defect inspection, duplicate detection, anomaly monitoring, and event localization depend on representations rather than generated pixels. An embedding is a fixed-length numerical vector intended to preserve selected properties of an image, text description, frame, or video segment. Once media is represented as vectors, it can be indexed, compared, clustered, and connected to operational data.

The central difficulty is that mathematical similarity does not automatically match operational relevance. A model may group images by color, background, camera angle, or style when the user expects the same defect, object state, or action.

The thesis of this article is that visual representations should be evaluated against the decision they support. Cosine similarity is an intermediate score, not proof that a retrieved asset is useful. A production system also needs task-specific relevance labels, retrieval metrics, temporal evaluation, cost controls, and versioned evidence.

The system must retrieve evidence, not merely similar pixels

Consider a manufacturing team that stores photographs and short inspection videos from previous quality incidents.

An engineer uploads an image showing a crack near a component joint. The system should retrieve earlier examples of the same defect pattern so that the engineer can compare severity, affected product variants, and previous corrective actions.

A generic visual encoder may return photographs with the same product color, camera angle, or workshop background. Those results can be close in embedding space while remaining irrelevant to the defect investigation.

Video introduces another constraint. The relevant event may occupy only three seconds of a two-minute recording. A representation of the complete clip may capture the general scene but lose the moment when the defect appears.

The engineering objective is therefore not to maximize a generic visual score. It is to return evidence that supports the user's task.

Nearest-neighbor retrieval is a baseline, not a complete solution

A common visual search pipeline preprocesses an asset, encodes it into a vector, searches an index for nearby vectors, and returns the associated media.

Let \(x_q\) be a query image and \(f_\theta\) a visual encoder. The query embedding is:

$$\mathbf{z}_q = f_\theta(x_q)$$

For indexed asset \(x_i\), the stored embedding is:

$$\mathbf{z}_i = f_\theta(x_i)$$

A common comparison function is cosine similarity:

$$sim(\mathbf{z}_q, \mathbf{z}_i) = \frac{\mathbf{z}_q^\top \mathbf{z}_i}{\lVert \mathbf{z}_q \rVert_2 \lVert \mathbf{z}_i \rVert_2}$$

The score measures the angle between two vectors. A larger value means that the vectors point in more similar directions. It does not identify which visual property caused that similarity.

CLIP learns image and text encoders in a shared representation space from image-text pairs. This design supports image-to-image retrieval, text-to-image retrieval, and zero-shot classification.

DINOv2 learns general-purpose visual features through self-supervised training. These features can support retrieval and other downstream visual tasks without requiring a class label for every training image.

Both model families can provide strong starting representations. Neither defines the correct relevance policy for a particular workflow.

Operational relevance must be specified before evaluation

An encoder can preserve background, lighting, viewpoint, style, object identity, local texture, and scene composition at the same time. The useful properties depend on the decision.

A defect retrieval system should rank the same failure pattern above the same product color. An action search system should prioritize the correct event and temporal phase over the same room or person. A product-matching system may require exact identity while ignoring photographic style.

A generic encoder may align with the task, but that alignment must be measured on reviewed queries rather than inferred from a few visually convincing examples.

Retrieval quality depends on relevance labels

Let \(y_i(q)\) indicate whether asset \(i\) is relevant to query \(q\) under the operational definition. The retrieval system returns an ordered list:

$$R_k(q) = \left(i_1, i_2, \ldots, i_k\right)$$

Similarity scores determine the ranking. Relevance labels determine whether the ranking is useful.

Precision at \(k\) measures the proportion of returned assets that are relevant:

$$Precision@k = \frac{\sum_{i \in R_k(q)} y_i(q)}{k}$$

If \(G(q)\) is the complete set of known relevant assets, recall at \(k\) measures how much of that set appears in the result list:

$$Recall@k = \frac{|R_k(q) \cap G(q)|}{|G(q)|}$$

Reciprocal rank focuses on the position of the first relevant result:

$$RR@k = \begin{cases} \frac{1}{rank_q}, & \text{if a relevant result occurs in the top } k \\ 0, & \text{otherwise} \end{cases}$$

The correct metric depends on the interface. A user browsing several alternatives may care about precision across the result list. An incident investigator may care that the first useful match appears immediately. A workflow that must recover every known case may prioritize recall.

Evaluation data must include difficult negatives

A retrieval benchmark needs query-result judgements. Without them, similarity scores can be inspected but not validated against the operational task.

For defect search, a review record should identify the query asset, relevant matches, and the reason each match is relevant. The labels may be binary for a narrow task or graded when exact matches, related mechanisms, and same-component examples provide different value.

Easy negatives can make retrieval appear stronger than it is. An unrelated landscape photograph does not test whether the representation distinguishes a cracked component from an intact component photographed under the same conditions.

The most informative negative cases usually share tempting but irrelevant properties with the query: the same background, product, color, camera, or viewpoint without the target defect or event.

A controlled retrieval example

The following Python example compares generic embedding similarity with a task-aligned ranking.

The example contains eight synthetic assets. Their vectors are constructed to produce predefined cosine similarities to the query. The defect_scores array represents an additional task-specific signal, such as the output of a validated defect classifier or reranker.

The labels and scores are synthetic. They illustrate retrieval evaluation and are not outputs from CLIP, DINOv2, or a real inspection dataset.

import numpy as np

query_embedding = np.array([1.0, 0.0])
asset_ids = np.array(["A17", "A04", "A09", "A31", "A12", "A22", "A28", "A35"])
target_cosines = np.array([0.96, 0.93, 0.90, 0.87, 0.84, 0.82, 0.79, 0.77])

asset_embeddings = np.column_stack([
    target_cosines,
    np.sqrt(1.0 - target_cosines**2),
])

defect_scores = np.array([0.20, 0.35, 0.88, 0.91, 0.12, 0.86, 0.15, 0.10])
relevant = np.array([False, False, True, True, False, True, False, False])


def cosine_similarity(query: np.ndarray, matrix: np.ndarray) -> np.ndarray:
    query_norm = query / np.linalg.norm(query)
    matrix_norm = matrix / np.linalg.norm(matrix, axis=1, keepdims=True)
    return matrix_norm @ query_norm


def evaluate(order: np.ndarray, k: int) -> tuple[float, float, float]:
    top_k = order[:k]
    hits = relevant[top_k]
    precision = hits.mean()
    recall = hits.sum() / relevant.sum()
    relevant_ranks = np.flatnonzero(hits)
    reciprocal_rank = 0.0 if relevant_ranks.size == 0 else 1.0 / (relevant_ranks[0] + 1)
    return precision, recall, reciprocal_rank


cosine_scores = cosine_similarity(query_embedding, asset_embeddings)
task_scores = 0.65 * cosine_scores + 0.35 * defect_scores

rankings = {
    "embedding": np.argsort(-cosine_scores),
    "task_aligned": np.argsort(-task_scores),
}

for name, order in rankings.items():
    precision, recall, reciprocal_rank = evaluate(order, k=4)
    print(
        f"{name:12s} top4={asset_ids[order[:4]].tolist()} "
        f"P@4={precision:.2f} R@4={recall:.2f} RR@4={reciprocal_rank:.2f}"
    )

The following output was produced by executing the code:

embedding    top4=['A17', 'A04', 'A09', 'A31'] P@4=0.50 R@4=0.67 RR@4=0.33
task_aligned top4=['A09', 'A31', 'A22', 'A04'] P@4=0.75 R@4=1.00 RR@4=1.00

The experiment separates geometric similarity from task value

The embedding-only ranking returns A17 and A04 first. They have the highest cosine similarities, but neither is relevant under the synthetic defect definition.

The first relevant asset appears at rank three. Precision at 4 is 0.50, recall at 4 is 0.67, and reciprocal rank is 0.33.

The task-aligned ranking combines the generic visual representation with a defect-specific signal. All three relevant assets appear within the first four positions, and the first result is relevant. Precision at 4 increases to 0.75, recall reaches 1.00, and reciprocal rank reaches 1.00.

These values demonstrate the mechanics of ranking evaluation. They do not establish that the weight combination of 0.65 and 0.35 is suitable for a real inspection system. The weights were selected only to create a clear synthetic comparison.

Task alignment can be added without replacing the encoder

A generic representation does not always need to be replaced. Task alignment can be introduced through candidate eligibility, final ranking, or representation training.

Filtering is appropriate when relevance includes hard eligibility rules. A photograph from the wrong product family may be invalid even if the visual match is strong.

Reranking is useful when a broad encoder provides good recall but weak ordering. The system retrieves a larger candidate set with the generic embedding and applies a more expensive task-specific comparison to that smaller set.

Representation adaptation becomes useful when the generic embedding consistently preserves the wrong properties. It also increases training, validation, and versioning requirements.

Video understanding requires temporal localization

An image represents one visual state. A video contains ordered observations and changes between them.

Frame-level embeddings can support scene search or near-duplicate detection, but they do not automatically represent motion, action order, or event boundaries.

Suppose an inspection video shows a machine operating normally, vibrating, stopping, and then being opened by a technician. A frame classifier may recognize the machine and technician correctly. The operational task may require the start and end of the abnormal vibration.

Temporal action localization represents a predicted event as an interval:

$$\hat{g} = \left[\hat{t}_{start}, \hat{t}_{end}\right]$$

For reference interval:

$$g = \left[t_{start}, t_{end}\right]$$

Temporal Intersection over Union measures their overlap:

$$tIoU(\hat{g}, g) = \frac{|\hat{g} \cap g|}{|\hat{g} \cup g|}$$

A prediction can identify the correct action class but receive a low temporal overlap because it starts too early or ends too late.

ActionFormer addresses temporal action localization with multiscale temporal features, local self-attention, and a decoder that classifies moments and estimates action boundaries.

The engineering implication is that video evaluation must distinguish event recognition from event localization.

Sampling policy determines which events remain observable

Processing every frame can be expensive and redundant. Sampling reduces encoding cost, storage, and search latency, but it can remove short events before the model sees them.

Let a video have duration \(T\) seconds and sampling frequency \(f_s\) frames per second. The approximate number of sampled frames is:

$$N = \left\lceil T f_s \right\rceil$$

A ten-minute video sampled at one frame per second produces approximately 600 frames. Sampling at five frames per second produces approximately 3,000.

The denser representation may preserve more temporal detail but increases encoding, indexing, and retrieval cost. A fixed sampling rate is also not always appropriate. Shot detection, motion-based selection, event triggers, or hierarchical processing can allocate more computation to informative segments.

The sampling policy must be evaluated together with the model. A strong encoder cannot recover an event that was never included in its input.

The index and model version form one retrieval system

The encoder does not determine production ranking alone. The index type, distance function, candidate count, filters, and approximate search parameters can also change the result.

Approximate nearest-neighbor search trades exactness for speed and memory efficiency. More aggressive search settings may reduce latency while lowering the probability that the best candidates are returned.

The retrieval record should therefore preserve the encoder version, preprocessing version, normalization rule, index build, distance function, candidate count, and reranking configuration.

Changing the encoder normally requires rebuilding or migrating the index. Vectors produced by different model versions should not be assumed to share compatible geometry. A query encoded with one version and compared against assets encoded with another can produce numerical scores without a valid operational interpretation.

Evaluation must match the consuming interface

A visual search page displaying twenty results needs different metrics from an automated duplicate detector returning one decision.

Offline relevance labels should be supplemented with workflow signals. A result may be judged relevant but still provide little value because it lacks metadata, appears too late in the ranking, or requires excessive inspection time.

Click and acceptance data can add evidence, but they are influenced by result position, interface design, and exposure. They should not be treated as unbiased ground truth.

Latency and cost should be measured by pipeline stage

Visual systems can move substantial amounts of data. End-to-end latency should be separated into media loading, preprocessing, encoding, index search, reranking, and result delivery.

This decomposition identifies the actual bottleneck. If image decoding dominates the request, replacing the embedding model may have little effect. If reranking dominates, reducing the candidate set may matter more than changing the index.

Batch encoding and caching are useful for static corpora. Video segment embeddings can be computed during ingestion rather than during every search. Cost analysis should also include re-indexing because a new encoder may require recomputing millions of vectors and maintaining two indexes during migration.

Feedback needs model, index, and task context

Retrieval feedback becomes useful evaluation or training data only when its context is preserved.

retrieval_record = {
    "query_asset": "frame_00842.jpg",
    "task": "surface_crack_match",
    "embedding_model": "vision-encoder-v3",
    "index_version": "inspection-2026-07",
    "result_asset": "asset_17.jpg",
    "rank": 1,
    "similarity": 0.94,
    "review_label": "same_background_wrong_defect",
}

The review label records more than irrelevance. It explains why the result failed.

Error categories can reveal that the system overweights background, camera position, color, or object identity. This information can guide hard-negative collection, reranking, or representation adaptation.

The record should also distinguish model output from human interpretation. Reviewers may disagree about whether two defects are operationally equivalent, so feedback should not be treated automatically as perfect ground truth.

Embeddings are appropriate for broad retrieval, not every visual task

Embeddings are useful when the system needs flexible similarity, cross-modal search, clustering, or candidate generation across a large media corpus.

They are less sufficient when the task requires precise object boundaries, exact counting, deterministic measurement, or strict temporal localization. Detection, segmentation, tracking, geometric methods, or specialized temporal models may be better primary components.

A common architecture uses embeddings for high-recall candidate generation and specialized models for final verification. This separates broad retrieval from task-specific precision.

Production design should preserve three control layers

1. Representation control: version the encoder, preprocessing, sampling policy, and embedding schema.
2. Retrieval control: version the index, filters, candidate count, reranker, and task definition.
3. Evaluation control: preserve relevance labels, difficult negatives, segment results, latency, and correction evidence.

An initial deployment should normally begin with a frozen encoder and a reviewed query set. This makes it easier to identify whether failures come from representation, indexing, filtering, or ranking.

Reviewed errors should become regression cases. A visually convincing wrong result is often more useful for system improvement than another easy positive example.

The evaluation remains conditional on the operating environment

Strong retrieval metrics on one dataset do not guarantee equivalent behavior on new cameras, products, regions, or video styles.

Relevance labels can be incomplete or subjective. Temporal boundaries may also be ambiguous when an action begins gradually rather than at one clear frame.

Similarity scores do not explain causality. A model may retrieve two defect cases together because they share a background artifact correlated with the defect rather than because it represents the defect itself.

The purpose of evaluation is not to prove that the representation is universally correct. It is to determine whether the complete retrieval system is reliable enough for a defined class of queries and known operating conditions.

Key takeaways

• Visual embeddings make images and video segments searchable, but similarity scores must be validated against operational relevance.
• Image retrieval requires task-specific labels and difficult negatives, while video systems must also measure sampling and temporal localization errors.
• Production quality depends on the complete chain of encoder, preprocessing, index, reranking, versioning, latency, and reviewed feedback.

Sources

1. Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, 8748-8763.
2. Oquab, M., Darcet, T., Moutakanni, T., et al. (2024). DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research.
3. Zhang, C.-L., Wu, J., and Li, Y. (2022). ActionFormer: Localizing Moments of Actions with Transformers. European Conference on Computer Vision, 492-510.