Explainability is useful when it helps someone make a better decision. It does not need to expose every mathematical operation inside a model. It needs to provide enough evidence for debugging, review, communication, governance, or corrective action.

The required evidence depends on the audience. A data scientist may need feature contributions and segment behavior. An engineer may need model versions, feature snapshots, and execution traces. A domain expert may need a reason that can be compared with operational knowledge. A person affected by a decision may need a clear explanation and a practical way to challenge or correct it.

This variation makes explainability an engineering problem rather than a single visualization method. The system must decide what is being explained, for whom, at which level, and with which limitations.

The main argument of this article is that explanation quality should be evaluated through its decision value, faithfulness, stability, and traceability. A polished chart is not enough if it cannot help a reviewer identify an error or understand the scope of the claim.

The practical problem is deciding what evidence a reviewer needs

Consider a subscription service that uses a churn model to prioritize retention outreach.

The model assigns one customer a churn probability of 0.78. A retention specialist must decide whether to contact the customer, which offer to use, and whether the prediction is credible enough to influence the workflow.

A probability alone does not answer those questions. The reviewer may need to know which current feature values influenced the score, whether similar customers were represented in the training data, whether the model version is approved, and whether the explanation is local to this prediction.

The explanation should support a specific next action. It should not merely make the model appear transparent.

A single explanation interface is unlikely to satisfy all four audiences. The underlying evidence can be shared, but its presentation and level of detail should match the decision being made.

Feature attribution is only one explanation layer

A common approach is to assign an importance or contribution value to each feature for one prediction.

LIME explains an individual prediction by fitting an interpretable surrogate model around the selected observation. The explanation describes the local behavior of the original model under the sampling and representation choices used by LIME.

SHAP defines a family of additive feature-attribution methods based on Shapley values. It assigns feature contributions relative to an expected model output under a selected background distribution.

Both approaches can help inspect a prediction. Neither automatically proves that the prediction is correct, fair, causal, or operationally justified.

A serious production system usually needs all three layers, although they do not need to appear in the same interface.

A local explanation has a defined scope

Let a model produce prediction \(f(\mathbf{x})\) for feature vector \(\mathbf{x}\).

An additive local explanation represents the output as a baseline plus feature contributions:

$$f(\mathbf{x}) \approx \phi_0 + \sum_{j=1}^{p} \phi_j$$

The value \(\phi_0\) represents a baseline output, while \(\phi_j\) represents the attributed contribution of feature \(j\) under the selected explanation method.

The exact meaning of the contribution depends on the model and method. It may refer to probability, log-odds, model score, or a local surrogate approximation. These scales should not be mixed.

A contribution also does not establish causality. A positive contribution means that the observed feature value moved the model output upward relative to the explanation baseline. It does not prove that changing the real-world feature would cause the outcome to change.

The explanation can be correct and still be unhelpful

Faithfulness describes whether the explanation accurately reflects the behavior of the model being explained.

Usefulness describes whether the explanation helps the intended audience complete a task. A faithful list of twenty technical features may be unusable for a domain reviewer. A readable reason may be unhelpful to an engineer if it omits the original feature values and model version.

Doshi-Velez and Kim argue that interpretability evaluation should reflect the application and user rather than relying only on informal visual inspection. Their taxonomy distinguishes application-grounded, human-grounded, and functionally grounded evaluation.

This distinction matters because an explanation method can perform well under one evaluation and poorly under another. A mathematically consistent attribution is not automatically understandable to a user. A user-preferred explanation is not automatically faithful to the model.

Polished explanations can create false confidence

Explanation interfaces often convert numerical outputs into fluent statements such as "low activity increased the risk" or "recent support contacts drove the decision."

Such statements can sound more conclusive than the underlying method justifies.

Several limitations may be hidden. Correlated features can divide or redistribute attribution in unintuitive ways. A local surrogate can be sensitive to its neighborhood definition. A contribution can change when the background dataset changes. Small input changes can also produce large ranking changes among the displayed factors.

The explanation should therefore identify its scope, method, output scale, and reference population. It should not present approximate attribution as a complete account of the model or the real-world process.

An exact contribution example for logistic regression

The following Python example creates a synthetic churn dataset and trains a logistic regression model. The data are generated only for demonstration and do not represent results from a real subscription system.

Because logistic regression is additive on the log-odds scale, the contribution of each standardized feature can be calculated exactly as the feature value multiplied by its fitted coefficient.

This is not a SHAP or LIME implementation. It is a direct decomposition of one linear model prediction.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.default_rng(42)
n_samples = 3000

feature_names = np.array([
    "recent_usage",
    "days_since_login",
    "support_tickets",
    "contract_months",
    "discount_rate",
])

X = np.column_stack([
    rng.normal(12, 4, n_samples),
    rng.gamma(2.0, 6.0, n_samples),
    rng.poisson(1.5, n_samples),
    rng.integers(1, 37, n_samples),
    rng.uniform(0.0, 0.35, n_samples),
])

true_weights = np.array([-0.35, 0.28, 0.45, -0.08, 1.20])
standardized_X = (X - X.mean(axis=0)) / X.std(axis=0)
logit = -0.8 + standardized_X @ true_weights
churn_probability = 1.0 / (1.0 + np.exp(-logit))
y = rng.binomial(1, churn_probability)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    random_state=42,
    stratify=y,
)

model = Pipeline([
    ("scale", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=2000, random_state=42)),
])

model.fit(X_train, y_train)

test_auc = roc_auc_score(
    y_test,
    model.predict_proba(X_test)[:, 1],
)

row = X_test[0:1]
scaled_row = model.named_steps["scale"].transform(row)[0]
classifier = model.named_steps["classifier"]

contributions = scaled_row * classifier.coef_[0]
prediction_logit = classifier.intercept_[0] + contributions.sum()
prediction = 1.0 / (1.0 + np.exp(-prediction_logit))

order = np.argsort(-np.abs(contributions))[:3]
top_factors = [
    (str(feature_names[index]), round(float(contributions[index]), 3))
    for index in order
]

print(f"test_auc={test_auc:.3f}")
print(f"prediction={prediction:.3f}")
print(f"intercept={classifier.intercept_[0]:.3f}")
print(f"contribution_sum={contributions.sum():.3f}")
print(f"top_factors={top_factors}")

The following output was produced by executing the code:

test_auc=0.799
prediction=0.206
intercept=-0.854
contribution_sum=-0.497
top_factors=[('discount_rate', -0.607), ('support_tickets', -0.163), ('recent_usage', 0.135)]

The contribution values are measured on the log-odds scale

The fitted model achieves a test ROC-AUC of 0.799 on the synthetic dataset. This value describes ranking performance for the generated evaluation sample. It does not validate the explanation interface or represent expected performance on real churn data.

For the selected observation, the model returns a churn probability of 0.206. The fitted intercept is -0.854, and the sum of the feature contributions is -0.497.

The three largest absolute contributions are associated with discount_rate, support_tickets, and recent_usage. The first two reduce the log-odds for this observation, while recent_usage increases them.

The contribution signs should be interpreted relative to the standardized feature values and fitted coefficients. They do not mean that discounts or support tickets universally reduce churn. They describe one prediction made by one fitted model.

Local contributions should remain connected to source values

A feature name and contribution value are not sufficient for review.

The reviewer should also be able to inspect the observed feature value, transformation, reference distribution, and data timestamp. Otherwise, an unexpected contribution cannot be traced back to the original evidence.

For example, a large contribution from days_since_login may result from a legitimate inactive period, an incorrect timestamp, or a feature pipeline using a different time zone from training. The attribution alone cannot distinguish these cases.

Local and global explanations should not be substituted

A local explanation describes one prediction or a small neighborhood around it. It cannot establish the dominant behavior of the complete model.

A feature may have a large contribution for one observation while having limited importance across the population. Another feature may have moderate contributions across many observations and therefore matter more globally.

Global analysis can include permutation importance, accumulated local effects, partial dependence, model coefficients, aggregated attributions, or segment-specific behavior. Each method has its own assumptions and limitations.

The explanation interface should state whether a result is local, global, or segment-specific. Mixing these levels can produce incorrect conclusions about model behavior.

Correlated features complicate attribution

Business datasets often contain correlated or redundant variables. Recent usage, session count, active days, and time since last login may describe overlapping behavior.

When correlated features provide similar information, an attribution method must decide how to distribute credit among them. Different background datasets, perturbation strategies, or conditional assumptions can produce different allocations.

This does not always mean that the explanation method is defective. The data may not contain enough information to assign one unique causal responsibility to each correlated feature.

Reviewers should therefore inspect feature groups and source relationships rather than treating the top-ranked variable as the only real driver.

Stability should be tested rather than assumed

An explanation is operationally fragile when small, plausible input changes produce large changes in the displayed reasoning while the prediction remains similar.

Stability can be evaluated by perturbing non-critical inputs within realistic ranges, recomputing the explanation, and comparing the selected factors, signs, and rankings.

The expected stability depends on the model and the case. A prediction close to a decision boundary may legitimately be sensitive. A stable prediction with constantly changing explanations may require closer inspection.

Explanation stability should not be optimized in isolation. An explanation can remain stable because it is too coarse to reveal meaningful local differences.

Counterfactual tests require feasible changes

A counterfactual explanation describes how an input would need to change for the model output or decision to change.

Such explanations can be useful when they identify an actionable path. They can be misleading when they suggest impossible or inappropriate modifications.

A model may indicate that a decision would change if a customer's age decreased, a historical event disappeared, or two dependent financial variables changed independently. These counterfactuals may be mathematically valid inputs but invalid real-world scenarios.

Counterfactual generation should therefore respect immutable attributes, feature dependencies, allowed ranges, and the cost of the proposed change.

Explanation validation should use three complementary tests

1. Model fidelity: verify that the explanation reflects the behavior of the model under the stated scope and assumptions.
2. Human decision value: test whether the intended audience detects errors, reviews cases, or makes decisions more effectively with the explanation.
3. Operational traceability: confirm that the prediction, feature values, model version, method, and review outcome can be reconstructed.

A method can pass one test and fail another. A faithful explanation can be too technical for a user. A readable explanation can omit important uncertainty. A detailed explanation can become useless if it cannot be connected to the original prediction.

Explanation usefulness should be measured through tasks

Asking reviewers whether an explanation looks convincing is a weak evaluation.

A stronger test assigns realistic tasks. Engineers may need to identify a broken feature transformation. Domain experts may need to detect a prediction that conflicts with policy. Reviewers may need to choose whether to accept, override, or escalate a decision.

Useful measurements include error-detection rate, review time, correction quality, agreement, escalation accuracy, and inappropriate automation acceptance.

The goal is not to maximize reviewer trust. The goal is to support appropriate reliance: acceptance when the evidence is sufficient and challenge when it is not.

Explanations should be evaluated across segments

An explanation method may work well for common cases and poorly for rare but important ones.

Sparse categories, missing values, unusual feature combinations, and observations far from the training distribution can produce unstable or difficult-to-interpret explanations.

Evaluation should therefore compare explanation fidelity, stability, review usefulness, and missing evidence across important segments.

A high-risk segment may require more detailed evidence or automatic escalation even when the same explanation format is sufficient elsewhere.

Traceability is a separate production requirement

Interpretability concerns how model behavior can be understood. Traceability concerns whether the system can reconstruct the prediction and its evidence.

A simple model can be interpretable but poorly traceable if the feature snapshot or model version was not preserved. A complex model can have strong traceability even when its internal behavior requires post-hoc explanation methods.

A production explanation record should therefore include identifiers and context beyond the displayed factors.

explanation_record = {
    "prediction_id": "score-2026-07-21-044",
    "model_version": "risk-model-v12",
    "feature_snapshot": "features-2026-07-21T10:30:00Z",
    "explanation_method": "linear-log-odds-contribution",
    "explanation_scope": "local-prediction",
    "output_scale": "log-odds",
    "review_status": "pending",
}

The explanation_scope field prevents a local result from being presented as a global model description. The output_scale field prevents log-odds contributions from being interpreted directly as probability changes.

User-facing reasons should not imitate causal explanations

User-facing explanations often need to be shorter than internal engineering records.

A useful reason can describe the observed inputs and the policy consequence. It should not claim that one feature caused the outcome unless the system has evidence supporting a causal interpretation.

For example, "The application was routed to review because the submitted income could not be verified" describes an observed validation condition. "Low income caused the rejection" is a stronger claim and may be inaccurate if several checks or thresholds were involved.

The explanation should also identify the next available action, such as correcting data, providing documentation, requesting review, or appealing the decision.

LLM explainability often depends on evidence provenance

Feature attribution is not always the most useful explanation mechanism for language-model systems.

For a retrieval-augmented answer, the most useful evidence may include the retrieved sources, passages used to support the response, retrieval scores, model version, and unsupported claims detected during evaluation.

For a tool-using agent, explanation may require a trace of observations, tool calls, policy decisions, approvals, and resulting external state.

These records do not expose every internal computation. They make the system behavior inspectable at the level required for debugging and review.

Explainability requirements should follow the risk

A low-risk content recommendation may need a short user-facing reason and basic internal logging.

A decision affecting money, access, employment, compliance, or safety may require stronger lineage, validation, review controls, and an appeal mechanism.

The explanation method should not be selected solely because it produces an attractive visualization. It should be selected because it supports the decisions, controls, and failure analysis required by the workflow.

Production implementation needs three explanation surfaces

1. Review surface: present the reason, source values, uncertainty, and next action to the intended reviewer.
2. Engineering surface: preserve model inputs, versions, traces, explanation parameters, and reproducibility evidence.
3. Evaluation surface: measure fidelity, stability, segment behavior, review quality, and correction outcomes.

The surfaces can share the same underlying records while presenting different levels of detail.

Explanations should also be versioned. Changing the model, background distribution, feature pipeline, or explanation library can change the displayed contributions even when the user interface remains unchanged.

What an explanation cannot establish

An explanation cannot prove that the model target is appropriate, the training data are unbiased, or the resulting decision is beneficial.

It cannot convert correlation into causation. It may not identify interactions or dependencies in a form that a user can interpret correctly. A local explanation also cannot establish global model safety.

The correct interpretation is narrower: an explanation provides evidence about selected aspects of model behavior under a defined method and scope.

That evidence becomes useful when it is connected to review, traceability, and a real decision process.

Key takeaways

• Explanation quality depends on the audience, decision, risk, and scope rather than on the amount of detail displayed.
• Local feature contributions can support review, but they do not establish causality, global model behavior, or decision correctness.
• Production explainability requires traceability, stability testing, task-based evaluation, and evidence that reviewers can challenge.

Sources

1. Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). Why Should I Trust You? Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144.
2. Lundberg, S. M., and Lee, S.-I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, 30.
3. Doshi-Velez, F., and Kim, B. (2017). Towards a Rigorous Science of Interpretable Machine Learning. arXiv:1702.08608.