A machine learning service can return successful responses, remain within its latency target, and still become less reliable. No exception is required. The application code may be unchanged while the data reaching the model gradually moves away from the population represented in training and validation.

This creates a gap between software health and model health. Infrastructure monitoring can tell us whether an endpoint is available. It cannot tell us whether the assumptions behind the model still hold.

This article examines one practical question: how can a team detect a potentially important change in model inputs before ground-truth labels become available? I use the Population Stability Index, or PSI, as a simple baseline. The main argument is that PSI can provide an early warning, but it cannot prove that predictive performance has deteriorated and should not trigger automatic retraining on its own.

A Healthy Endpoint Can Serve an Unreliable Model

During development, the environment is relatively controlled. The dataset is fixed, experiments can be repeated, and model quality is measured on validation or test data using metrics such as ROC-AUC, precision, recall, F1 score, log loss, or mean absolute error.

These measurements are necessary, but they describe model behavior on historical data. Production introduces a different process. Users change their behavior, upstream systems are modified, feature definitions evolve, measurement devices are recalibrated, and new customer segments appear.

None of these changes has to break the serving application. The model can continue returning valid numbers while the statistical conditions under which it was evaluated no longer apply.

This is one source of the system-level technical debt described in Hidden Technical Debt in Machine Learning Systems. Machine learning behavior depends not only on source code, but also on data, feature transformations, configuration, and interactions with the surrounding system.

A production monitoring strategy needs more than one of these layers. A healthy endpoint is necessary, but it is not sufficient evidence that the ML system is healthy.

The Operational Problem: Decisions Arrive Before Labels

Consider a fraud detection model that scores transactions in real time. The team must decide whether to approve a transaction, decline it, or send it for manual review. The cost of a poor prediction is immediate, but the final fraud label may not arrive for days or weeks.

Suppose a marketing campaign attracts a new customer segment with larger transactions. Alternatively, an upstream system may start reporting amounts in different units or apply a new rounding rule. In both cases, the distribution of a feature changes. The first change may be a legitimate business development. The second may be a data defect.

The model cannot distinguish these causes merely by receiving a different value distribution. The monitoring system therefore has to support several possible decisions:

• continue serving without intervention,
• investigate the data pipeline or product change,
• apply additional controls to an affected segment,
• roll back a feature or model version,
• collect labels and reevaluate the model,
• retrain only after the cause and expected benefit are understood.

The relevant question is not simply "did drift occur?". The useful question is:

Has the production population changed enough to justify investigation or a change in the operating policy?

A monitoring metric is valuable only when the team has defined what happens after it moves.

Why Offline Metrics May Warn Too Late

Offline model metrics require labeled examples. In many systems, labels are delayed, incomplete, selectively observed, or affected by decisions made by the model itself.

Fraud may be confirmed after a chargeback. Churn may be observed after the end of a subscription period. Product defects may be recorded after inspection or a customer complaint. A model can therefore operate on a changed population for a substantial period before direct performance estimates become available.

Input monitoring reduces this blind period, but its interpretation requires a precise distinction between types of distribution change.

Let \(X\) denote the model inputs and \(Y\) the target. A change in the input distribution can be written as:

$$P_{ref}(X) \neq P_{cur}(X)$$

This is often described as input drift or covariate drift, depending on the assumptions and terminology being used. PSI can detect a change in a binned marginal distribution of a selected feature.

A different problem occurs when the relationship between inputs and the target changes:

$$P_{ref}(Y \mid X) \neq P_{cur}(Y \mid X)$$

This is commonly called concept shift or concept drift. Detecting it normally requires labels or another source of information about prediction correctness.

The distinction matters because input drift and model degradation are not equivalent:

• The distribution of \(X\) can change while the model remains accurate.
• The relationship \(P(Y \mid X)\) can change even when the marginal distribution of \(X\) appears stable.
• A univariate feature metric can remain stable while dependencies between several features change.

PSI therefore provides evidence about a distribution, not evidence about prediction quality. A broader treatment of these distinctions can be found in Dataset Shift in Machine Learning.

Monitoring Is Useful Only When the System Is Recoverable

When predictions stop making sense, the algorithm is rarely the first useful object to inspect. The operational questions are more concrete:

• Which model version produced the prediction?
• Which training dataset and code revision produced that model?
• Which feature transformations were used during training and serving?
• Which production window triggered the alert?
• Can the team compare the current population with the correct reference population?
• Can the previous model or feature pipeline be restored safely?
• Who owns the investigation and the final decision?

If these questions cannot be answered, a drift score will usually create more confusion than control. The monitoring system may signal that something changed without providing enough lineage to determine what changed.

The production-readiness rubric in The ML Test Score treats testing, monitoring, debuggability, and rollback as properties of the complete ML system rather than optional additions to a trained model.

A practical minimum includes:

What Population Stability Index Measures

PSI compares two distributions: a reference population and a current population. The reference is often a training, validation, or previously accepted production window. The current population is usually a recent batch of production observations.

For a numerical feature, the usual procedure is:

1. Use the reference population to define \(B\) bins.
2. Place both reference and current observations into the same bins.
3. Calculate the proportion of observations in each bin.
4. Aggregate the difference between the two probability vectors.

For bins \(b = 1, \dots, B\), PSI is:

$$PSI = \sum_{b=1}^{B} \left(p_b^{cur} - p_b^{ref}\right) \ln\left(\frac{p_b^{cur}}{p_b^{ref}}\right)$$

Here:

• \(p_b^{ref}\) is the proportion of reference observations in bin \(b\),
• \(p_b^{cur}\) is the proportion of current observations in the same bin,
• \(\ln\) is the natural logarithm.

For binned probability vectors, this expression corresponds to a symmetric form of relative-entropy divergence often associated with Jeffreys divergence. A value near zero indicates similar proportions across the bins. The value increases as the binned distributions separate.

The calculation has several implementation choices that affect the result:

• how the reference window is selected,
• how many bins are created,
• whether bins have equal width or approximately equal reference frequency,
• how missing values and new categories are represented,
• how empty bins are handled,
• how many observations are included in the current window.

The logarithm is undefined when one of the proportions is zero. Implementations typically apply smoothing or another explicit empty-bin policy. This is not a cosmetic detail: rare values and new categories may be exactly the changes that monitoring is intended to detect.

Why Universal PSI Thresholds Are Misleading

Values such as 0.10 and 0.25 are frequently repeated as generic boundaries between stable, moderate, and substantial change. They may be useful as provisional dashboard labels, but they are not universal statistical laws.

A PSI value depends on the feature distribution, binning strategy, reference period, current sample size, and smoothing method. The same numerical value can also have different operational significance for different features. A small change in a safety-critical feature may deserve more attention than a larger change in a weak or redundant feature.

Recent methodological analysis of PSI explicitly warns against applying a common cutoff without use-case-specific evaluation. The appropriate alert policy should be calibrated using historical windows, known incidents, simulated changes, and the downstream sensitivity of the model. See Measuring Data Drift with the Unstable Population Indicator for a discussion of PSI, binning, empty bins, sample size, and cutoff selection.

In this article, 0.10 and 0.25 are used only as illustrative review levels. Crossing either value does not establish that the model should be retrained.

A Synthetic Experiment

The following experiment isolates one simple type of shift. It does not train a fraud model and does not generate target labels. It only creates a synthetic transaction-amount feature whose distribution moves gradually over twelve production windows.

The reference sample contains 20,000 observations drawn from a log-normal distribution:

$$X_{ref} \sim LogNormal(3.0, 0.6)$$

For week \(w\), the current population contains 5,000 observations:

$$X_w \sim LogNormal(3.0 + 0.035w, 0.6)$$

The increasing location parameter creates a controlled rightward shift. The experiment demonstrates whether PSI responds to that shift. It does not demonstrate that any model loses accuracy.

import numpy as np


def population_stability_index(
    reference: np.ndarray,
    current: np.ndarray,
    n_bins: int = 10,
    epsilon: float = 1e-6,
) -> float:
    reference = np.asarray(reference, dtype=float)
    current = np.asarray(current, dtype=float)

    if reference.ndim != 1 or current.ndim != 1:
        raise ValueError("reference and current must be one-dimensional")

    if len(reference) == 0 or len(current) == 0:
        raise ValueError("reference and current must not be empty")

    if not np.isfinite(reference).all() or not np.isfinite(current).all():
        raise ValueError("reference and current must contain only finite values")

    if n_bins < 2:
        raise ValueError("n_bins must be at least 2")

    # Define quantile bins from the reference population only.
    edges = np.quantile(reference, np.linspace(0.0, 1.0, n_bins + 1))
    edges = np.unique(edges)

    if len(edges) < 3:
        raise ValueError(
            "reference does not contain enough distinct values to create bins"
        )

    # Include current observations outside the reference range.
    edges[0], edges[-1] = -np.inf, np.inf

    ref_counts, _ = np.histogram(reference, bins=edges)
    cur_counts, _ = np.histogram(current, bins=edges)

    # Additive smoothing prevents division by zero and log(0).
    n_effective_bins = len(ref_counts)
    ref_pct = (ref_counts + epsilon) / (
        ref_counts.sum() + epsilon * n_effective_bins
    )
    cur_pct = (cur_counts + epsilon) / (
        cur_counts.sum() + epsilon * n_effective_bins
    )

    contributions = (cur_pct - ref_pct) * np.log(cur_pct / ref_pct)
    return float(np.sum(contributions))


rng = np.random.default_rng(42)

reference = rng.lognormal(
    mean=3.0,
    sigma=0.6,
    size=20_000,
)

for week in range(1, 13):
    current = rng.lognormal(
        mean=3.0 + 0.035 * week,
        sigma=0.6,
        size=5_000,
    )

    psi = population_stability_index(reference, current)
    print(f"week {week:02d}: PSI={psi:.4f}")

I ran this code with Python 3.13.5 and NumPy 2.3.5. The following output is the actual result of that execution:

week 01: PSI=0.0075
week 02: PSI=0.0155
week 03: PSI=0.0328
week 04: PSI=0.0424
week 05: PSI=0.0779
week 06: PSI=0.0960
week 07: PSI=0.1523
week 08: PSI=0.2013
week 09: PSI=0.2567
week 10: PSI=0.3309
week 11: PSI=0.3632
week 12: PSI=0.4421

What the Experiment Shows

The experiment supports a narrow conclusion: with a fixed reference population and fixed quantile bins, PSI increases as the simulated current distribution moves progressively farther from the reference distribution.

By week 7, the value exceeds the illustrative 0.10 review level. By week 9, it exceeds 0.25. This would be enough to trigger investigation under a policy using those provisional levels.

The experiment does not show that:

• a deployed fraud model has become less accurate,
• larger transaction amounts are invalid or fraudulent,
• the shift was caused by a pipeline failure,
• retraining would improve business outcomes,
• 0.10 or 0.25 is an appropriate threshold for a real system.

There is no target variable and no trained model in the experiment, so model performance cannot be measured. The correct interpretation is limited to the monitored feature: its binned marginal distribution has changed relative to the selected reference sample.

How to Evaluate a PSI Alerting Policy

A monitoring metric should be evaluated against the decision it is intended to support. Calculating PSI correctly is not enough. The team also needs evidence that its reference windows, aggregation periods, and alert levels produce useful operational signals.

I would evaluate a PSI policy using at least four kinds of evidence.

Historical Backtesting

Compute PSI across previous production windows, including normal seasonal variation, product releases, campaigns, pipeline incidents, and periods with known performance degradation. This reveals the normal range of the metric and the frequency of false alarms.

Controlled Perturbations

Introduce synthetic shifts that represent plausible failures, such as scaling errors, missing-value changes, category substitutions, truncation, delayed data, or population-mixture changes. Check whether the alert responds before the perturbation becomes operationally unacceptable.

Connection to Delayed Performance

When labels arrive, compare earlier drift signals with later estimates of discrimination, calibration, segment-level error, and business loss. The objective is not to prove that every drift event causes degradation. It is to learn which signals are useful for this particular model and decision process.

Stability Across Window Sizes and Segments

Repeat the calculation for different sample sizes, time windows, geographic regions, customer groups, channels, and model versions. A global PSI value can conceal a severe shift in a small but important segment.

A Small but Serious Production Control Loop

A useful monitoring design does not require an oversized platform. It requires a control loop that the team can operate consistently.

1. Record the prediction context. Store the model version, timestamp, permitted input values or privacy-preserving summaries, prediction, decision, and relevant request context.
2. Validate incoming data. Check schemas, types, missingness, categories, ranges, freshness, and volume before calculating drift metrics.
3. Compare production with an explicit reference. Record which reference dataset and bin definitions were used for every monitoring run.
4. Monitor selected features and outputs. Use PSI or alternative measures for important inputs, and monitor prediction distributions when they provide useful additional evidence.
5. Route alerts to an owner. Define severity, response time, investigation steps, and the person or team responsible for the final decision.
6. Evaluate delayed labels. Update performance, calibration, and business metrics when ground truth becomes available.
7. Keep responses reversible. Maintain the ability to restore a previous model, feature pipeline, threshold, or routing policy.

This reflects the broader MLOps principle that automation and monitoring must cover the complete ML lifecycle, not only the serving container. Google Cloud's MLOps architecture guidance similarly treats data verification, metadata management, validation, deployment, and monitoring as parts of one operating process.

When PSI Is a Reasonable Baseline

PSI is a useful starting point when the system has tabular numerical or categorical features, an explicit reference population, enough observations to estimate stable bin proportions, and a team that needs an interpretable univariate signal.

It is particularly useful when labels are delayed and the first operational question is whether the input population has changed.

PSI is less sufficient when:

• the important shift is multivariate rather than marginal,
• the input consists of images, text, audio, or high-dimensional embeddings,
• feature semantics or units change without a clear schema signal,
• the model uses interactions that cannot be assessed feature by feature,
• the current window is too small to estimate stable proportions,
• the required decision depends directly on performance rather than distribution similarity.

Alternative or complementary approaches include statistical tests for individual features, Jensen-Shannon or Wasserstein distances, classifier-based two-sample tests, monitoring in a learned representation space, prediction-distribution monitoring, and delayed-label performance evaluation. The choice should follow the failure mode the system needs to detect, not the availability of a particular dashboard.

Key Takeaways

• A successful API response does not establish that an ML model remains reliable.
• PSI measures change in a binned feature distribution. It does not directly measure model accuracy, concept drift, or business impact.
• Reference windows, bins, smoothing, sample size, and segmentation are part of the metric definition and must be reproducible.
• Values such as 0.10 and 0.25 should be treated as provisional review levels, not universal retraining rules.
• A useful alert leads to an owned investigation that can distinguish population change, data defects, training-serving skew, and actual model degradation.
• Lineage, delayed-label evaluation, and rollback determine whether monitoring produces operational control rather than additional noise.

Sources and Further Reading

Production ML Systems

1. Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems.
2. Breck, E., Cai, S., Nielsen, E., Salib, M., and Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. IEEE International Conference on Big Data.
3. Google Cloud. MLOps: Continuous Delivery and Automation Pipelines in Machine Learning.
4. Google for Developers. Rules of Machine Learning: Best Practices for ML Engineering.

Dataset Shift and PSI

1. Quinonero-Candela, J., Sugiyama, M., Schwaighofer, A., and Lawrence, N. D., editors. (2009). Dataset Shift in Machine Learning. MIT Press.
2. Haas, M. R., and Sibbald, L. (2024). Measuring Data Drift with the Unstable Population Indicator. Data Science, 7, 1-12.
3. Evidently. Data Drift Documentation.