Causal inference can fail before an estimator is selected. The problem is not always the regression formula, matching algorithm, weighting method, or model family. It is often whether treated and untreated observations are sufficiently comparable for the intended causal question.

If one treatment group occupies regions of the covariate space that have almost no counterpart in the other group, the estimate starts depending on extrapolation. A model can still return a numerical effect and a narrow confidence interval, but the data may provide little direct support for the comparison.

Overlap is therefore not a cosmetic diagnostic added after estimation. It helps determine whether the target effect can be learned from the observed population and which observations contribute credible comparisons.

The main argument of this article is that causal estimates should be evaluated together with support, balance, weight stability, trimming sensitivity, and the population described by the final estimand.

The practical question is whether the groups contain credible comparisons

Consider a company evaluating a retention campaign from observational data.

Customers were not assigned to the campaign randomly. The retention team preferentially contacted customers who had recently reduced their activity, opened several support tickets, or generated high expected revenue.

The analysis is expected to estimate whether the campaign reduced churn.

A direct comparison between contacted and uncontacted customers is unlikely to answer that question. The two groups differ before treatment because campaign assignment depends on customer characteristics related to churn.

Adjustment methods can reduce this imbalance when comparable treated and untreated customers exist. They cannot reconstruct untreated outcomes for a customer profile that is always treated, or treated outcomes for a profile that is never treated.

The goal is not only to produce an average effect. The analysis must state for which population the comparison is supported.

Overlap is part of identification, not model performance

Let \(A \in \{0, 1\}\) denote treatment, \(X\) the observed pre-treatment covariates, and \(Y(1)\) and \(Y(0)\) the potential outcomes under treatment and no treatment.

The individual treatment effect is:

$$\tau_i = Y_i(1) - Y_i(0)$$

Only one potential outcome is observed for each unit. The observed outcome is:

$$Y_i = A_iY_i(1) + (1-A_i)Y_i(0)$$

To estimate an average causal effect from observational data, an analysis commonly relies on consistency, conditional exchangeability, and positivity.

Conditional exchangeability can be expressed as:

$$\left(Y(1), Y(0)\right) \perp A \mid X$$

This states that after conditioning on the measured covariates, treatment assignment contains no additional information about the potential outcomes. It is not directly testable from the observed data and can fail when relevant confounders were not measured.

Positivity requires both treatment conditions to have non-zero probability for the covariate patterns in the target population:

$$0 < P(A=1 \mid X=x) < 1$$

Overlap is the empirical counterpart of this requirement. Even if the theoretical treatment probability is not exactly zero or one, a finite dataset can contain too few observations from one group to support stable adjustment.

The propensity score summarizes treatment assignment

The propensity score is the conditional probability of treatment given observed pre-treatment covariates:

$$e(X) = P(A=1 \mid X)$$

Rosenbaum and Rubin established the role of the propensity score as a balancing score for observed covariates under the assumptions of their framework.

A score close to zero means that treatment is unusual for the observed covariate profile. A score close to one means that the untreated condition is unusual.

Extreme scores are not automatically errors. They may reflect real treatment policies or contraindications. They do indicate that effect estimation in those regions depends on weak comparisons and potentially large weights.

A propensity histogram is only the first diagnostic

Plotting estimated propensity distributions by treatment group can reveal obvious separation. It should not be treated as a complete overlap assessment.

Different covariate profiles can receive similar scalar propensity scores. A misspecified treatment model may also create apparently acceptable score distributions while important covariates remain imbalanced.

A practical diagnosis should combine three views:

• propensity support and the frequency of extreme scores,
• covariate balance after the selected adjustment method,
• weight concentration, effective sample size, and sensitivity to support restrictions.

The diagnostics describe different failure modes. Propensity support examines treatment assignment. Balance checks whether adjustment made the groups comparable on measured covariates. Weight diagnostics show whether the estimate is dominated by a small part of the sample.

Weighting makes limited overlap operationally visible

For inverse probability weighting, an observation receives a weight based on the probability of its observed treatment.

For the average treatment effect, the unstabilized weight is:

$$w_i = \frac{A_i}{e(X_i)} + \frac{1-A_i}{1-e(X_i)}$$

A treated observation with a propensity score of 0.02 receives a weight of 50. An untreated observation with a score of 0.98 receives the same weight.

These observations represent treatment conditions that are rare for their covariate profiles. A few such cases can dominate the weighted estimate.

The effective sample size of non-negative weights can be summarized as:

$$ESS = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}$$

The effective sample size is not the number of retained rows. It measures how much information remains after accounting for weight concentration. Twelve thousand observations can behave like a much smaller sample when a few weights dominate.

Trimming removes unsupported regions but changes the question

A common response to limited overlap is to restrict the analysis to observations whose propensity scores lie inside a selected interval:

$$\alpha \leq e(X) \leq 1-\alpha$$

For example, setting \(\alpha=0.10\) retains observations with estimated propensity scores between 0.10 and 0.90.

Crump, Hotz, Imbens, and Mitnik study limited overlap and derive population restrictions intended to improve the precision of average treatment-effect estimation. Their work also discusses the practical rule of restricting estimated propensity scores to a central interval under selected conditions.

The interval should not be treated as a universal default. Its suitability depends on the target population, treatment process, sample size, estimator, and cost of excluding observations.

Trimming changes the estimand. The resulting effect applies to the retained overlap population rather than automatically to the complete original population.

A synthetic experiment with weak overlap

The following example generates synthetic observational data with three pre-treatment covariates.

Treatment assignment depends strongly on the first covariate, creating regions in which treatment or control observations are uncommon. The treatment effect also varies with that covariate.

Because the data are synthetic, the true average treatment effect can be calculated for every retained population. This makes it possible to compare the estimator with the known data-generating process.

The experiment fits a logistic regression propensity model and estimates the effect with Hajek-normalized inverse probability weighting. It reports the retained sample size, true effect in the retained population, estimated effect, maximum absolute weighted standardized mean difference, and effective sample size.

import numpy as np
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(42)
n_samples = 12_000

x1 = rng.normal(0, 1, n_samples)
x2 = rng.binomial(1, 0.45, n_samples)
x3 = rng.normal(0, 1, n_samples)
X = np.column_stack([x1, x2, x3])

true_logit = -0.3 + 2.4 * x1 + 0.9 * x2 - 0.4 * x3
true_propensity = 1 / (1 + np.exp(-true_logit))
treatment = rng.binomial(1, true_propensity)

individual_effect = 0.8 + 0.35 * x1
baseline_outcome = (
    2.0 + 0.7 * x1 - 0.5 * x2 + 0.3 * x3
    + rng.normal(0, 0.8, n_samples)
)
outcome = baseline_outcome + treatment * individual_effect

propensity_model = LogisticRegression(
    max_iter=2000,
    random_state=42,
)
propensity_model.fit(X, treatment)

propensity = np.clip(
    propensity_model.predict_proba(X)[:, 1],
    0.01,
    0.99,
)


def hajek_ate(y, t, e):
    treated_weights = t / e
    control_weights = (1 - t) / (1 - e)

    treated_mean = (
        np.sum(treated_weights * y)
        / treated_weights.sum()
    )
    control_mean = (
        np.sum(control_weights * y)
        / control_weights.sum()
    )

    return treated_mean - control_mean


def effective_sample_size(weights):
    return (
        weights.sum() ** 2
        / np.square(weights).sum()
    )


def max_weighted_smd(features, t, weights):
    smd = []

    for column in features.T:
        treated = column[t == 1]
        control = column[t == 0]
        wt = weights[t == 1]
        wc = weights[t == 0]

        mt = np.average(treated, weights=wt)
        mc = np.average(control, weights=wc)
        vt = np.average(
            (treated - mt) ** 2,
            weights=wt,
        )
        vc = np.average(
            (control - mc) ** 2,
            weights=wc,
        )

        smd.append(
            (mt - mc)
            / np.sqrt((vt + vc) / 2)
        )

    return np.max(np.abs(smd))


for alpha in [0.00, 0.05, 0.10, 0.15]:
    keep = (
        (propensity >= alpha)
        & (propensity <= 1 - alpha)
    )

    t = treatment[keep]
    e = propensity[keep]
    weights = (
        t / e
        + (1 - t) / (1 - e)
    )

    estimate = hajek_ate(
        outcome[keep],
        t,
        e,
    )
    true_effect = (
        individual_effect[keep].mean()
    )
    max_smd = max_weighted_smd(
        X[keep],
        t,
        weights,
    )
    ess = effective_sample_size(weights)

    print(
        f"trim={alpha:.2f} "
        f"kept={keep.sum():5d} "
        f"true_effect={true_effect:.3f} "
        f"estimate={estimate:.3f} "
        f"max_abs_smd={max_smd:.3f} "
        f"ess={ess:.0f}"
    )

The following output was produced by executing the code:

trim=0.00 kept=12000 true_effect=0.796 estimate=0.849 max_abs_smd=0.124 ess=3080
trim=0.05 kept= 9160 true_effect=0.789 estimate=0.747 max_abs_smd=0.013 ess=4921
trim=0.10 kept= 7544 true_effect=0.785 estimate=0.739 max_abs_smd=0.016 ess=5193
trim=0.15 kept= 6257 true_effect=0.787 estimate=0.760 max_abs_smd=0.010 ess=4940

The untrimmed estimate uses all rows but little effective information

Without trimming, all 12,000 observations remain in the analysis. The effective sample size is approximately 3,080 because the inverse probability weights are highly concentrated.

The maximum absolute weighted standardized mean difference is 0.124. This indicates that weighting has not removed all observed imbalance under the fitted propensity model.

The estimated effect is 0.849, compared with a true synthetic average effect of 0.796 for the complete generated population.

The difference is not proof that every untrimmed estimator will be biased by the same amount. It demonstrates that retaining all observations can produce unstable weighting and incomplete balance when overlap is weak.

A small support restriction improves balance and effective sample size

With propensity scores restricted to 0.05 through 0.95, 9,160 observations remain.

The maximum absolute weighted standardized mean difference falls from 0.124 to 0.013. The effective sample size rises from approximately 3,080 to 4,921 despite the removal of 2,840 rows.

This apparently paradoxical result occurs because the retained weights are less concentrated. Removing observations with extreme treatment probabilities can leave fewer rows but more usable weighted information.

The estimated effect changes from 0.849 to 0.747. That change is part of the result. It reflects both reduced dependence on extreme observations and the fact that the retained population differs from the original population.

Stronger trimming does not guarantee monotonic improvement

Increasing the restriction from 0.05 to 0.10 removes another 1,616 observations. The maximum absolute standardized mean difference remains low, and the effective sample size increases slightly to approximately 5,193.

At a 0.15 restriction, only 6,257 observations remain. Balance is still strong, but the effective sample size falls to approximately 4,940 because the actual retained sample has become smaller.

The point estimates do not move monotonically toward the known synthetic effect. Trimming reduces one source of instability but does not eliminate sampling error, propensity-model error, outcome noise, or sensitivity to the selected population.

The correct interpretation is not that the strictest trimming rule is always best. The analysis should identify a region in which comparison quality is acceptable and the remaining population still matches the decision problem.

The estimand must be renamed when the population changes

The average treatment effect for the original population is:

$$ATE = E\left[Y(1)-Y(0)\right]$$

After restricting the analysis to an overlap region \(S_\alpha\), the estimand becomes:

$$ATE_{\alpha} = E\left[ Y(1)-Y(0) \mid \alpha \leq e(X) \leq 1-\alpha \right]$$

This is an average effect for the retained population.

In the synthetic experiment, the true effect changes only slightly across the selected trimming rules because the generated treatment heterogeneity is moderate and the central populations have similar mean effects.

That stability is not guaranteed in real data. If treatment effects are largest in the excluded regions, the overlap-population effect can differ materially from the full-population effect.

The report should therefore state the inclusion rule and describe the retained population. Calling every trimmed result "the ATE" hides an important change in the causal question.

Balance should be checked after adjustment

Overlap describes whether useful comparisons appear possible. It does not prove that the selected adjustment method achieved balance.

After matching, weighting, or subclassification, the distribution of measured pre-treatment covariates should be compared between treatment groups.

A standardized mean difference for covariate \(X_j\) can be written as:

$$SMD_j = \frac{ \bar{X}_{1j} - \bar{X}_{0j} }{ \sqrt{ \left(s_{1j}^2+s_{0j}^2\right)/2 } }$$

For weighting, the means and variances should be calculated with the analysis weights.

Austin discusses several balance diagnostics and shows why examining only means or propensity-model discrimination is insufficient for assessing whether treatment groups are comparable after propensity-score adjustment.

No universal SMD threshold proves absence of confounding. Small observed imbalance does not address unmeasured confounders, incorrect feature definitions, or misspecified interactions.

The treatment model should not be selected by predictive accuracy alone

A propensity model predicts treatment assignment, but its objective in causal adjustment is not ordinary treatment classification.

A high ROC-AUC can indicate that the treatment groups are easy to distinguish. That may correspond to weak overlap rather than a desirable analysis property.

The model should be evaluated through the balance and weight behavior it produces for the intended estimator.

Relevant checks include whether important covariates remain imbalanced, whether weights are concentrated, and whether the conclusions change under reasonable treatment-model specifications.

Adding a highly predictive variable can improve treatment prediction while creating more extreme scores. The variable may still be required for confounding adjustment, but the operational consequence for overlap must be examined.

Precision does not establish identification

A narrow confidence interval quantifies sampling uncertainty under the estimator and assumptions used to construct it.

It does not show that positivity holds, that all confounders were measured, or that the outcome model extrapolates correctly into unsupported regions.

Model-based standard errors can become especially misleading when the estimator relies on strong extrapolation that is not represented in the uncertainty calculation.

A causal report should distinguish statistical precision from support for the causal comparison. A precise estimate from weakly overlapping groups can be less credible than a wider estimate from a clearly defined overlap population.

Overlap should also be diagnosed in covariate space

The propensity score compresses measured treatment predictors into one value. This is useful, but it can conceal local gaps in the original covariate space.

Two observations can have similar estimated propensity scores while differing substantially in clinically or operationally important variables.

Additional diagnostics can include nearest-neighbor distances, multidimensional density checks, exact or coarsened matches on critical variables, and inspection of regions in which one treatment group is absent.

These checks are particularly important when selected variables define hard treatment rules. For example, a safety contraindication may make one treatment condition impossible for a subgroup. No adjustment method should pretend that both treatment choices were observed there.

Global overlap can hide unsupported business segments

An overall propensity distribution may look acceptable while a high-value region, age group, product line, or clinical subgroup has almost no comparison data.

Segment diagnostics should report treatment counts, propensity ranges, weight concentration, and retained population share.

The number of segments should remain controlled. Very small groups produce unstable diagnostics, while hundreds of exploratory segment tests can generate misleading patterns.

If a business decision depends primarily on one segment, overlap for that segment may matter more than global overlap.

Estimator disagreement is evidence, not an inconvenience

Matching, inverse probability weighting, outcome regression, and doubly robust estimators rely on different modelling choices.

Agreement across reasonable methods does not prove the causal result, but strong disagreement can reveal model dependence or weak support.

A useful sensitivity analysis can compare:

• the unadjusted difference as a non-causal baseline,
• a primary adjusted estimator under several support restrictions,
• an alternative estimator using a different adjustment strategy.

The comparison should preserve the estimand. An estimator for the treated population should not be compared casually with an estimator for a trimmed overlap population as though both answer the same question.

Overlap decisions should be stored with the estimate

A causal estimate should remain connected to the data restrictions and diagnostics that support it.

effect_record = {
    "estimand": "ate_for_trimmed_overlap_population",
    "propensity_model": "logistic-v4",
    "trim_rule": "0.05 <= propensity <= 0.95",
    "n_original": 12000,
    "n_retained": 9160,
    "effective_sample_size": 4921,
    "max_abs_weighted_smd": 0.013,
    "effect_estimate": 0.747,
}

The record should also identify the outcome definition, treatment definition, covariate set, reference period, estimator version, and uncertainty method.

Without this context, a result can be copied into a dashboard or presentation and applied to a population that was excluded from the analysis.

A practical validation process starts before outcome modelling

I would organize the analysis into three control stages:

1. Design control: define the treatment, outcome, target population, causal timing, confounders, and estimand before inspecting treatment effects.
2. Support control: diagnose propensity distributions, covariate support, extreme weights, effective sample size, and segment-level overlap.
3. Estimation control: evaluate balance, trimming sensitivity, estimator agreement, uncertainty, and the population represented by the final result.

This order reduces the temptation to select the design that produces the cleanest treatment effect.

Overlap restrictions should be justified by design and diagnostics rather than chosen after observing which threshold creates the preferred conclusion.

What overlap diagnostics cannot prove

Good overlap does not prove conditional exchangeability. Treated and untreated observations can look similar on measured covariates while differing on an unobserved confounder.

A correctly estimated propensity score does not repair measurement error, post-treatment adjustment, interference between units, or an incorrectly defined treatment.

Trimming can improve support while reducing external validity. The resulting estimate may be credible for the retained population but irrelevant to the users excluded by the rule.

Overlap diagnostics clarify where the observed data supports comparison. They do not convert an observational study into a randomized experiment.

Key takeaways

• Overlap should be diagnosed before interpreting a causal estimate because weak support forces the estimator to rely on extreme weights or extrapolation.
• Trimming can improve balance and effective sample size, but it changes the population and therefore changes the estimand.
• A causal result should be reported with support restrictions, covariate balance, weight diagnostics, segment coverage, and sensitivity across reasonable specifications.

Sources

1. Rosenbaum, P. R., and Rubin, D. B. (1983). The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70(1), 41-55.
2. Crump, R. K., Hotz, V. J., Imbens, G. W., and Mitnik, O. A. (2009). Dealing with Limited Overlap in Estimation of Average Treatment Effects. Biometrika, 96(1), 187-199.
3. Austin, P. C. (2009). Balance Diagnostics for Comparing the Distribution of Baseline Covariates Between Treatment Groups in Propensity-Score Matched Samples. Statistics in Medicine, 28(25), 3083-3107.