A churn model can estimate which customers are likely to leave. It cannot, by itself, determine which customers should receive a retention offer.

These tasks require different target quantities. Churn prediction estimates an outcome under the conditions represented in historical data. Retention targeting must estimate how that outcome would change if the company intervened.

The distinction becomes important when campaign capacity is limited. Customers with the highest churn probability are not necessarily the customers whose behavior can be changed. Some will remain without an offer. Some will leave regardless of it. A smaller group may respond positively, while another may react negatively to being contacted.

This article studies a deliberately narrow setting: a binary retention outcome, a randomized treatment and control experiment, and a T-learner used to rank customers by estimated treatment effect. The central argument is that uplift modeling should be evaluated as a decision-support method, not as an ordinary classifier.

The business decision is treatment allocation

Suppose a retention team can contact only 10 percent of eligible customers. The offer has a financial and operational cost, so treating everyone is not acceptable. The practical decision is therefore:

Which customers should receive the offer?

A standard churn model estimates a probability such as:

$$P(Y = 1 \mid X = x)$$

where \(Y = 1\) denotes churn and \(X\) contains customer characteristics available before the campaign.

This probability is useful for forecasting risk. Ranking customers by it identifies people who are likely to churn under the historical conditions represented in the data. It does not identify people for whom an offer changes the outcome.

For intervention targeting, the relevant quantity is a difference between two potential outcomes:

$$\tau(x) = E[Y(1) \mid X = x] - E[Y(0) \mid X = x]$$

Here, \(Y(1)\) is the outcome under treatment, \(Y(0)\) is the outcome without treatment, and \(\tau(x)\) is the conditional average treatment effect, commonly abbreviated as CATE.

In this article, \(Y = 1\) represents retention rather than churn. A positive CATE means that the offer is expected to increase retention for customers with characteristics \(x\). A negative CATE means that the offer is expected to reduce retention.

A model can perform well on the first two questions and still produce a poor answer to the third.

Why churn risk and uplift can produce opposite rankings

Consider two customers.

Customer A has an 80 percent probability of churning without an offer and a 78 percent probability of churning with an offer. This customer has high baseline risk, but treatment reduces that risk by only 2 percentage points.

Customer B has a 40 percent probability of churning without an offer and a 25 percent probability with an offer. This customer has lower baseline risk, but treatment reduces it by 15 percentage points.

A churn-risk model ranks Customer A higher. An uplift model ranks Customer B higher.

Neither ranking is universally correct. The appropriate ranking depends on the decision. Churn probability is relevant when the objective is forecasting total churn. Treatment effect is more relevant when the objective is allocating a limited intervention.

This distinction also explains why targeting the customers with the highest risk can waste campaign capacity. High-risk customers may be difficult or impossible to influence, while customers with moderate risk may be substantially more responsive.

The required outcome is counterfactual

For an individual customer \(i\), the treatment effect can be written as:

$$\tau_i = Y_i(1) - Y_i(0)$$

The difficulty is that only one potential outcome can be observed. A customer either receives the offer or does not. The factual outcome becomes visible, but the outcome under the alternative decision remains counterfactual.

There is therefore no customer-level training column containing the true individual treatment effect. Uplift modeling is not ordinary supervised learning with an observed causal label.

A randomized experiment makes average effects identifiable because treatment assignment is independent of the potential outcomes. Conditional on the eligible population and the assignment design, the central independence condition can be expressed as:

$$\left(Y(1), Y(0)\right) \perp T \mid X$$

In a properly implemented randomized experiment, this condition follows from the assignment mechanism rather than from a predictive model. Randomization removes systematic treatment selection, although finite samples can still contain accidental imbalances.

Conditions required for a causal interpretation

Random assignment is not the only requirement. The analysis also relies on several conditions:

• Consistency: the observed outcome equals the potential outcome corresponding to the treatment that was actually received.
• Positivity or overlap: relevant customer profiles have a non-zero probability of appearing in both the treatment and control groups.
• No interference: treating one customer does not change another customer's outcome, unless such interactions are modeled explicitly.
• A well-defined treatment: the treatment label does not combine materially different offers, channels, or delivery procedures into one ambiguous intervention.
• Pre-treatment covariates: variables used for targeting are measured before the treatment decision and are not consequences of the campaign.

Randomization addresses confounding caused by treatment selection. It does not automatically solve treatment non-compliance, measurement error, interference, small samples, missing outcomes, or changes between the experimental and production populations.

Conditions required for useful targeting

Even an internally valid experiment may be insufficient for production targeting. The data must contain enough observations in both treatment arms, enough variation in customer characteristics, and enough treatment-effect heterogeneity to estimate a stable ranking.

The future campaign must also be sufficiently similar to the experiment. Changes in offer value, communication channel, eligibility rules, economic conditions, or customer composition can invalidate the assumption that an estimated effect will transport to the next campaign.

The four uplift groups are a useful abstraction

A common explanation of uplift modeling divides customers into four behavioral groups:

This taxonomy clarifies why response probability and treatment effect differ. A sure thing may have a high probability of retention after treatment, but treatment produces no incremental benefit. A do-not-disturb may also appear likely to respond in one arm while being harmed by the intervention.

The groups must not be interpreted as observed customer classes. Membership depends on both potential outcomes, and only one is observed. A standard experiment cannot reveal whether a specific customer is a persuadable or a sure thing.

In practice, the estimand is usually the conditional average effect:

$$\tau(x) = \mu_1(x) - \mu_0(x)$$

where:

$$\mu_1(x) = E[Y \mid T = 1, X = x]$$

and:

$$\mu_0(x) = E[Y \mid T = 0, X = x]$$

The result describes an estimated average difference among customers with similar observed characteristics. It is not a known deterministic effect for one individual.

A T-learner provides a transparent baseline

A T-learner estimates the two response functions separately:

1. Fit a model on treated observations to estimate \(\mu_1(x)\).
2. Fit a second model on control observations to estimate \(\mu_0(x)\).
3. Subtract the two predictions.

The estimated CATE is:

$$\hat{\tau}(x) = \hat{\mu}_1(x) - \hat{\mu}_0(x)$$

The letter T refers to the use of two independent learners. Each learner solves a conventional outcome-prediction problem, but the difference between their predictions is interpreted as an estimated treatment effect.

The method is a useful baseline because it is easy to implement and can use many probabilistic classifiers as base learners. It also exposes the main difficulty of uplift estimation: a potentially small treatment effect must be recovered as the difference between two larger response probabilities.

The same structure creates several limitations.

• Each response model uses only one treatment arm, reducing the effective training sample.
• Prediction errors from both models contribute to the error of their difference.
• The two models may learn incompatible response surfaces.
• Strong outcome prediction does not guarantee accurate treatment-effect ranking.
• An imbalanced treatment allocation can make one response model much less stable than the other.

No meta-learner is uniformly best. The T-learner should be treated as a reference method that must be compared with alternatives under an evaluation design aligned with the intended policy.

A retention experiment

The following experiment creates a synthetic randomized retention campaign. The data do not represent real customers and provide no evidence about the effects of actual retention programs.

The simulation contains 100,000 observations and four pre-treatment features. Treatment is independently assigned with probability 0.5. Seventy percent of the observations are used for training, and the remaining 30 percent form an untouched randomized test set.

The untreated retention probability is generated from a nonlinear logistic model. A heterogeneous treatment effect is then added on the probability scale. The effect depends on the same features that influence baseline retention.

The data-generating process is intentionally constructed so that customers with high estimated churn risk tend to have small or negative treatment effects. This creates a controlled counterexample in which a churn-risk ranking and an uplift ranking support different decisions.

Because the treated probability is clipped to the valid range from 0.01 to 0.99, the code defines the true CATE after clipping:

$$\tau(x) = p_1(x) - p_0(x)$$

This detail matters. Using the requested pre-clipping effect as ground truth would make the reported PEHE and subgroup effects inconsistent with the probabilities that actually generated the outcomes.

The code below was executed with Python 3.13.5, NumPy 2.3.5, scikit-learn 1.8.0, and Matplotlib 3.10.8.

import sys

import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import sklearn
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split


def sigmoid(values: np.ndarray) -> np.ndarray:
    return 1.0 / (1.0 + np.exp(-values))


def make_model() -> HistGradientBoostingClassifier:
    return HistGradientBoostingClassifier(
        max_iter=200,
        learning_rate=0.05,
        max_leaf_nodes=15,
        min_samples_leaf=100,
        l2_regularization=1.0,
        random_state=42,
    )


def difference_in_means(
    outcome: np.ndarray,
    treatment: np.ndarray,
    mask: np.ndarray,
) -> tuple[float, float, float]:
    treated_outcome = outcome[mask & (treatment == 1)]
    control_outcome = outcome[mask & (treatment == 0)]

    treated_mean = treated_outcome.mean()
    control_mean = control_outcome.mean()
    effect = treated_mean - control_mean

    standard_error = np.sqrt(
        treated_mean * (1.0 - treated_mean) / len(treated_outcome)
        + control_mean * (1.0 - control_mean) / len(control_outcome)
    )

    return (
        float(effect),
        float(effect - 1.96 * standard_error),
        float(effect + 1.96 * standard_error),
    )


rng = np.random.default_rng(42)
n_samples = 100_000

x1 = rng.normal(size=n_samples)
x2 = rng.normal(size=n_samples)
x3 = rng.binomial(1, 0.4, size=n_samples)
x4 = rng.uniform(-1.0, 1.0, size=n_samples)
features = np.column_stack([x1, x2, x3, x4])

baseline_logit = (
    -0.5
    + 0.8 * x1
    + 0.8 * x2
    + 0.35 * x3
    + 0.25 * x4
)
p_control = sigmoid(baseline_logit)

requested_cate = (
    0.18 * sigmoid(1.2 * x1 + 1.0 * x2 + 0.6 * x4)
    - 0.06
)
p_treated = np.clip(p_control + requested_cate, 0.01, 0.99)

true_cate = p_treated - p_control

treatment = rng.binomial(1, 0.5, size=n_samples)
observed_probability = np.where(
    treatment == 1,
    p_treated,
    p_control,
)
outcome = rng.binomial(1, observed_probability)

(
    x_train,
    x_test,
    t_train,
    t_test,
    y_train,
    y_test,
    cate_train,
    cate_test,
) = train_test_split(
    features,
    treatment,
    outcome,
    true_cate,
    test_size=0.30,
    random_state=42,
    stratify=treatment,
)

treated_model = make_model()
control_model = make_model()

treated_model.fit(
    x_train[t_train == 1],
    y_train[t_train == 1],
)
control_model.fit(
    x_train[t_train == 0],
    y_train[t_train == 0],
)

mu1_hat = treated_model.predict_proba(x_test)[:, 1]
mu0_hat = control_model.predict_proba(x_test)[:, 1]

uplift_score = mu1_hat - mu0_hat
churn_risk_score = 1.0 - mu0_hat

n_test = len(x_test)
n_top = n_test // 10

uplift_order = np.argsort(-uplift_score)
risk_order = np.argsort(-churn_risk_score)

all_mask = np.ones(n_test, dtype=bool)
uplift_mask = np.zeros(n_test, dtype=bool)
risk_mask = np.zeros(n_test, dtype=bool)

uplift_mask[uplift_order[:n_top]] = True
risk_mask[risk_order[:n_top]] = True

observed_ate = difference_in_means(
    y_test,
    t_test,
    all_mask,
)
top_uplift_effect = difference_in_means(
    y_test,
    t_test,
    uplift_mask,
)
top_risk_effect = difference_in_means(
    y_test,
    t_test,
    risk_mask,
)

true_ate = float(cate_test.mean())
true_top_uplift = float(cate_test[uplift_mask].mean())
true_top_risk = float(cate_test[risk_mask].mean())

pehe_rmse = float(
    mean_squared_error(cate_test, uplift_score) ** 0.5
)

fractions = np.linspace(0.0, 1.0, 11)
qini_values = [0.0]

for fraction in fractions[1:]:
    prefix_size = int(n_test * fraction)
    indices = uplift_order[:prefix_size]

    prefix_treatment = t_test[indices]
    prefix_outcome = y_test[indices]

    treated_count = int((prefix_treatment == 1).sum())
    control_count = int((prefix_treatment == 0).sum())

    treated_responders = int(
        prefix_outcome[prefix_treatment == 1].sum()
    )
    control_responders = int(
        prefix_outcome[prefix_treatment == 0].sum()
    )

    qini_value = (
        treated_responders
        - control_responders
        * treated_count
        / control_count
    )
    qini_values.append(float(qini_value))

random_targeting = fractions * qini_values[-1]

qini_area = float(
    np.trapezoid(
        np.asarray(qini_values) - random_targeting,
        x=fractions,
    )
)

print(f"python={sys.version.split()[0]}")
print(f"numpy={np.__version__}")
print(f"scikit_learn={sklearn.__version__}")
print(f"matplotlib={matplotlib.__version__}")
print(f"test_size={n_test}")
print(f"true_ate={true_ate:.4f}")

print(
    "observed_ate="
    f"{observed_ate[0]:.4f} "
    f"95%_CI=[{observed_ate[1]:.4f}, {observed_ate[2]:.4f}]"
)

print(f"synthetic_pehe_rmse={pehe_rmse:.4f}")

print(
    "top_decile_uplift_effect="
    f"{top_uplift_effect[0]:.4f} "
    f"95%_CI=[{top_uplift_effect[1]:.4f}, "
    f"{top_uplift_effect[2]:.4f}]"
)
print(f"top_decile_uplift_true_effect={true_top_uplift:.4f}")

print(
    "top_decile_churn_risk_effect="
    f"{top_risk_effect[0]:.4f} "
    f"95%_CI=[{top_risk_effect[1]:.4f}, "
    f"{top_risk_effect[2]:.4f}]"
)
print(f"top_decile_churn_risk_true_effect={true_top_risk:.4f}")

print(
    "incremental_retained_per_10000_uplift_targeted="
    f"{top_uplift_effect[0] * 10_000:.0f}"
)
print(
    "incremental_retained_per_10000_risk_targeted="
    f"{top_risk_effect[0] * 10_000:.0f}"
)

print(f"qini_area_decile_grid={qini_area:.1f}")

fig, ax = plt.subplots(figsize=(12, 4.8))

ax.plot(
    fractions,
    qini_values,
    marker="o",
    label="T-learner ranking",
)
ax.plot(
    fractions,
    random_targeting,
    linestyle="--",
    label="Random targeting",
)

ax.set_title("Qini curve on the held-out randomized sample")
ax.set_xlabel("Fraction of customers targeted")
ax.set_ylabel(
    "Incremental retained customers in treated test observations"
)
ax.legend()

fig.tight_layout()
fig.savefig(
    "uplift-modeling-treatment-effects-output-revised.png",
    dpi=160,
)
plt.close(fig)

The following values are the actual output from that execution:

python=3.13.5
numpy=2.3.5
scikit_learn=1.8.0
matplotlib=3.10.8
test_size=30000
true_ate=0.0296
observed_ate=0.0328 95%_CI=[0.0215, 0.0440]
synthetic_pehe_rmse=0.0461
top_decile_uplift_effect=0.0526 95%_CI=[0.0206, 0.0846]
top_decile_uplift_true_effect=0.0844
top_decile_churn_risk_effect=-0.0356 95%_CI=[-0.0535, -0.0177]
top_decile_churn_risk_true_effect=-0.0421
incremental_retained_per_10000_uplift_targeted=526
incremental_retained_per_10000_risk_targeted=-356
qini_area_decile_grid=108.0

The ranking changes the campaign decision

The true average treatment effect in the synthetic test population is 0.0296. On average, the offer increases retention probability by approximately 3 percentage points.

The observed difference between treatment and control retention rates is 0.0328. The normal-approximation 95 percent confidence interval extends from 0.0215 to 0.0440. The known simulated effect lies inside this interval.

The average effect is useful for deciding whether the campaign works in the population as a whole. It does not answer which customers should be selected when only 10 percent can be contacted.

Targeting by estimated uplift

The top decile selected by the T-learner has an observed treatment effect of 0.0526. Under a direct scaling of this estimate, treating 10,000 customers from the same population would produce approximately 526 additional retained customers compared with not treating them.

The known simulated effect in the selected segment is 0.0844. The difference between the observed and true values illustrates sampling uncertainty. Even with 3,000 observations in the selected test segment, the estimated effect remains noisy.

The extrapolation to 10,000 future customers is conditional on assumptions that are not tested by the calculation. It assumes that:

• the selected future population resembles the experimental population,
• the offer is delivered in the same way,
• the treatment effect remains stable,
• there are no capacity or interference effects,
• the estimated subgroup effect transports to the future campaign.

The number 526 should therefore be interpreted as an estimate under a stated policy and population, not as a guaranteed campaign result.

Targeting by churn risk

The top decile selected by predicted churn risk has an observed treatment effect of -0.0356. In this simulation, treating the highest-risk customers reduces retention. Scaled to 10,000 targeted customers, the estimate corresponds to 356 fewer retained customers.

The known simulated effect in this segment is -0.0421, which is consistent with the observed negative effect.

This is not a general claim about retention campaigns. The result follows from the synthetic data-generating process. The experiment demonstrates a logical possibility: a model can correctly identify high-risk customers while producing a harmful intervention policy.

PEHE is available only because the data are synthetic

The experiment reports a root PEHE value of 0.0461. PEHE, the precision in estimation of heterogeneous effects, measures the root mean squared difference between estimated and true conditional effects:

$$PEHE = \sqrt{E\left[\left(\hat{\tau}(X) - \tau(X)\right)^2\right]}$$

A lower value indicates that the estimated CATE is closer to the known effect function used by the simulation.

PEHE cannot usually be calculated on real campaign data. The true customer-level or conditional treatment effect is not observed. Even in a randomized experiment, each customer reveals only one potential outcome.

For real data, model selection must rely on quantities that can be estimated from the assignment mechanism and observed outcomes. These include uplift curves, Qini-style measures, policy value estimates, subgroup treatment effects, and prospective experimental comparisons.

Evaluate uplift out of sample

An uplift model should be evaluated on observations that were not used to fit the treatment-effect estimator. The evaluation sample should retain a known treatment assignment mechanism, preferably from a randomized experiment.

Reusing training data can produce an optimistic uplift curve. A flexible learner may discover random differences between treatment arms and then rank the same observations according to those differences. Evaluating that ranking in sample rewards noise that may disappear in a new campaign.

A practical evaluation design separates three functions:

• Training data are used to estimate the response surfaces.
• Validation data are used to choose features, learners, hyperparameters, and targeting rules.
• A final randomized test set is used once to evaluate the frozen policy.

For smaller experiments, a three-way split may leave too few observations in each treatment arm. Cross-fitting can use data more efficiently by ensuring that every observation receives a score from a model that was trained on other observations.

Cross-fitting does not remove the need for an explicit final estimand and valid inference. It only reduces the amount of data lost to sample splitting.

Qini curves measure ranking, not response classification

To build a Qini curve, customers are sorted from the highest to the lowest estimated uplift. For successive prefixes of that ranking, outcomes in randomized treatment and control observations are compared.

The construction used in the experiment is:

$$Q(k) = Y_1(k) - Y_0(k)\frac{N_1(k)}{N_0(k)}$$

where:

• \(Y_1(k)\) is the number of positive outcomes among treated observations in the first \(k\) ranked cases,
• \(Y_0(k)\) is the corresponding number among control observations,
• \(N_1(k)\) and \(N_0(k)\) are the treatment and control counts in the prefix.

A useful ranking places customers with larger treatment effects near the beginning. Its curve should therefore rise faster than the random-targeting baseline.

The Qini area reported by the code is 108.0 on a decile grid. This value is specific to the exact formula, sample size, treatment allocation, grid, and normalization used in the implementation.

Qini coefficients are not fully standardized across libraries. Implementations may differ in weighting, normalization, interpolation, random baselines, and integration conventions. Values from two tools should not be compared until their definitions have been checked.

A global area metric can also hide the part of the curve that matters operationally. If the campaign can contact exactly 10 percent of customers, the effect and expected value within the first decile may be more relevant than performance across all targeting depths.

Classification metrics answer another question

Accuracy, ROC-AUC, F1 score, and log loss evaluate outcome prediction. They can diagnose the treated and control response models, but they do not directly evaluate treatment-effect ranking.

Suppose the true response probabilities are:

$$\mu_1(x) = 0.52$$

and:

$$\mu_0(x) = 0.49$$

The treatment effect is only:

$$\tau(x) = 0.03$$

A prediction error of a few percentage points in either response surface can reverse the sign of the estimated effect. Both classifiers may still have acceptable global predictive metrics.

Outcome metrics are therefore diagnostic rather than decisive. Final evaluation should examine:

• the treatment effect at operational targeting depths,
• uncertainty in selected segments,
• stability across periods and customer groups,
• comparison with random, risk-based, treat-all, and treat-none baselines,
• the expected economic value of the resulting policy.

The final objective is policy value

Even a correctly ranked CATE is not a complete targeting policy. An intervention has a cost, retained customers have different values, and campaign capacity may be limited.

A simplified expected incremental value for a customer with characteristics \(x\) is:

$$NetValue(x) = V(x)\tau(x) - C(x)$$

where \(V(x)\) is the expected value of producing the desired outcome and \(C(x)\) is the cost of treatment.

Without capacity constraints, treatment is economically justified when:

$$V(x)\tau(x) > C(x)$$

The corresponding policy can be written as:

$$\pi(x) = \mathbf{1}\left\{V(x)\tau(x) - C(x) > 0\right\}$$

If only 10 percent of customers can be treated, the policy should generally rank customers by expected incremental value and select the best feasible subset. Ranking by CATE alone is optimal only under additional assumptions, such as equal customer value and equal treatment cost.

A production policy may also need to account for:

• contact and channel limits,
• customer fatigue and previous interventions,
• legal and consent requirements,
• fairness or eligibility constraints,
• uncertainty penalties for unstable estimates,
• possible adverse outcomes such as complaints or opt-outs.

The useful output is therefore not merely an uplift score. It is a decision rule evaluated under the actual constraints of the campaign.

Failure modes that change the meaning of the result

Ranking by response and calling it uplift

A response model predicts an outcome under treatment or under historical exposure. It does not estimate the difference between treatment and control outcomes.

The error can be detected by checking whether the method explicitly uses both treatment arms to estimate a causal contrast. A model trained only on treated responders is not an uplift model, even when its scores are used for campaign targeting.

Using post-treatment features

Offer opens, link clicks, call outcomes, and post-contact activity may strongly predict retention. They cannot be used to decide who should receive the offer because they occur after the treatment decision.

A timestamped feature inventory should record when every variable becomes available. Targeting features must be available before assignment and must not be consequences of treatment.

Treating historical campaign data as randomized

Historical treatment is often selected by previous business rules. Treated and untreated customers may therefore differ before the campaign.

A naive T-learner fitted to such data may learn the previous targeting policy rather than the causal effect of the offer. Randomization should be preferred. When only observational data are available, conditional exchangeability, overlap, and treatment-assignment assumptions must be stated and defended.

Propensity-based and doubly robust methods can reduce some forms of bias under their assumptions. They do not convert unmeasured confounding into an identified causal effect.

Evaluating the ranking on training observations

An in-sample uplift curve can reward treatment-control noise found during training. Each evaluated observation should receive a score from a model that did not use that observation during fitting.

An independent randomized test set provides the clearest final evaluation. Cross-fitting is a possible alternative when data are limited.

Ignoring uncertainty in selected segments

Uplift rankings naturally surface extreme estimates. In small samples, these extremes may reflect variance rather than stable heterogeneity.

For each operational segment, the analysis should report treatment and control counts, effect estimates, uncertainty intervals, and stability across resamples, periods, or repeated experiments.

Ignoring negative treatment effects

A high probability of a positive outcome under treatment does not imply that treatment is beneficial. The control probability may be even higher.

A targeting policy must allow the no-treatment decision. Adverse outcomes such as complaints, opt-outs, or reduced engagement should be included when they affect the business objective.

Optimizing a global metric instead of the campaign

The model with the largest Qini area is not automatically the model with the greatest business value. It may perform well over the full ranking while performing poorly at the actual budget limit.

Model selection should reflect the targeting depth, customer value, treatment cost, and constraints that will be used in production.

When a T-learner is a reasonable starting point

A T-learner is a useful baseline when treatment and outcome are clearly defined, both treatment arms contain enough data, overlap is adequate, and the base learners can estimate outcome probabilities without severe instability.

It is particularly useful when a team needs a transparent first comparison before adopting more specialized estimators.

The method becomes less attractive when one treatment arm is much smaller, treatment effects are weak relative to outcome variation, or subtracting two independently estimated response surfaces produces unstable scores.

Relevant alternatives include S-learners, X-learners, R-learners, transformed-outcome methods, causal trees, causal forests, and doubly robust learners. Their performance depends on the sample size, treatment allocation, response surfaces, effect structure, regularization, and evaluation design.

The correct practical question is not which method is universally best. It is which frozen policy produces the greatest reliable value under the intended deployment conditions.

From an offline score to a production policy

I would not deploy the raw T-learner score directly. I would first define and evaluate the complete decision process:

1. Define the eligible population, treatment, control condition, outcome, and evaluation horizon.
2. Specify treatment cost, customer value, capacity, and adverse outcomes.
3. Verify randomization, treatment delivery, overlap, and feature timestamps.
4. Fit candidate estimators using pre-treatment variables only.
5. Select the model and targeting depth on validation data using policy-aligned metrics.
6. Evaluate the frozen policy on an untouched randomized sample.
7. Report uncertainty for the selected segment and for the estimated policy value.
8. Run a prospective randomized comparison against the current targeting rule.
9. Monitor assignment, treatment delivery, population drift, incremental outcomes, and unintended effects.

The prospective comparison remains important. Offline evaluation estimates how the policy would have performed under a particular experiment and a set of assumptions. A new randomized campaign tests whether the complete operational system produces the expected result.

Key takeaways

• A churn model estimates baseline risk. An uplift model estimates how treatment changes an outcome.
• The individual treatment effect is not observed because each customer reveals only one potential outcome.
• A T-learner estimates CATE by subtracting separate treated and control response models. It does not recover known individual counterfactuals.
• Uplift rankings should be evaluated out of sample using data with a known treatment assignment mechanism.
• Qini and uplift curves evaluate ranking quality, but operational decisions should focus on treatment effect and net value at realistic targeting depths.
• A production system requires a complete policy, uncertainty analysis, and prospective experimental validation, not only an uplift score.

Sources and further reading

Potential outcomes and causal identification

1. Rubin, D. B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology, 66(5), 688-701.
2. Imbens, G. W., and Rubin, D. B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge University Press.
3. Hernan, M. A., and Robins, J. M. (2024). Causal Inference: What If.

Heterogeneous treatment effects and meta-learners

1. Athey, S., and Imbens, G. (2016). Recursive Partitioning for Heterogeneous Causal Effects. Proceedings of the National Academy of Sciences, 113(27), 7353-7360.
2. Gutierrez, P., and Gerardy, J. Y. (2017). Causal Inference and Uplift Modelling: A Review of the Literature. Proceedings of Machine Learning Research, 67, 1-13.
3. Kunzel, S. R., Sekhon, J. S., Bickel, P. J., and Yu, B. (2019). Metalearners for Estimating Heterogeneous Treatment Effects Using Machine Learning. Proceedings of the National Academy of Sciences, 116(10), 4156-4165.

Ranking and policy evaluation

1. Radcliffe, N. J., and Surry, P. D. (2011). Quality Measures for Uplift Models.
2. Hitsch, G. J., Misra, S., and Zhang, W. W. (2024). Heterogeneous Treatment Effects and Optimal Targeting Policy Evaluation. Quantitative Marketing and Economics, 22, 115-168.