Deep learning made feature extraction feel almost automatic for images, audio, and text. Tabular machine learning did not follow the same path. In structured business data, the strongest model is often not the one with the most impressive architecture. It is the one fed with variables that actually describe the problem.

This is why I still treat feature engineering as serious modelling work. A raw table rarely contains features in the form a model needs. It contains dates, identifiers, categories, missing values, operational shortcuts, and historical artifacts from systems that were never designed for machine learning.

In this note I focus on one common example: regularized target encoding. It looks simple, but it teaches an important lesson. A useful transformation can also become a very efficient way to leak the target if it is implemented carelessly.

Why Tabular Data Is Different

Images have spatial locality. Text has sequence. Audio has temporal structure. Tabular data is less polite. A single dataset may contain prices, product categories, timestamps, user identifiers, regional codes, binary flags, and fields created for internal reporting rather than prediction.

That heterogeneity is exactly why feature engineering still matters. The model does not see business context. It sees columns. If the representation is poor, even a strong algorithm spends most of its capacity compensating for weak inputs.

I do not mean that every project needs hundreds of hand-crafted variables. That usually becomes noise. I mean that a few well-designed transformations can make the learning problem simpler, more stable, and easier to debug.

A Practical Model

For tabular problems, I usually ask one question before adding a feature: does this variable make the target easier to separate without using information that would not exist at prediction time?

That second part is the trap. Many features look predictive because they accidentally contain the answer. Leakage is rarely dramatic. It often hides inside dates, status fields, post-event aggregates, or encodings computed on the full dataset.

Target Encoding Without Fooling Yourself

Target encoding replaces a category with a statistic of the target for that category. If customers from a specific region historically have a higher probability of churn, the region can be encoded using that probability.

The idea is useful, especially for high-cardinality categories. But the naive implementation is dangerous:

encoded = df.groupby("category")["target"].mean()
df["category_encoded"] = df["category"].map(encoded)

baseline_auc=0.781
with_engineered_features_auc=0.842
lift=+0.061

This leaks information because each row helps compute the statistic used to encode itself. The model receives a feature that is partially built from the answer. Validation scores can look excellent while production performance disappoints.

Regularization and Out-of-Fold Encoding

A safer version needs two ideas. First, smooth category estimates toward the global mean, especially for rare categories. Second, compute encodings out of fold so that a training row never sees its own target while being encoded.

The smoothing formula can be written as:

$$\hat{\mu}_c = \frac{n_c \mu_c + \alpha \mu}{n_c + \alpha}$$

where \(\mu_c\) is the category target mean, \(n_c\) is the number of observations in the category, \(\mu\) is the global target mean, and \(\alpha\) controls how strongly rare categories are pulled toward the global estimate.

import numpy as np
import pandas as pd
from sklearn.model_selection import KFold


def out_of_fold_target_encode(frame, column, target, n_splits=5, alpha=20):
    result = pd.Series(index=frame.index, dtype=float)
    global_mean = frame[target].mean()
    folds = KFold(n_splits=n_splits, shuffle=True, random_state=42)

    for train_idx, valid_idx in folds.split(frame):
        train = frame.iloc[train_idx]
        stats = train.groupby(column)[target].agg(["mean", "count"])
        smooth = (stats["count"] * stats["mean"] + alpha * global_mean) / (stats["count"] + alpha)
        result.iloc[valid_idx] = frame.iloc[valid_idx][column].map(smooth)

    return result.fillna(global_mean)

This is still not the whole story. In time-dependent problems, ordinary random folds may be wrong. If the model will predict the future, the encoding must respect time. Otherwise the validation setup again becomes more optimistic than production.

What I Usually Watch For

Leakage Before Performance

If a feature gives a suspiciously large lift, I first assume leakage until proven otherwise. This sounds pessimistic, but it saves time. The best-looking feature in a notebook is often the one that knows too much.

Prediction-Time Availability

Every feature should pass a simple test: would this value exist at the exact moment of prediction? If not, it does not belong in the training data, no matter how predictive it is.

Stability Across Segments

A feature can work globally and fail for a specific region, product, customer type, or time window. I prefer checking stability early, because production failures often appear first in a segment, not in the aggregate metric.

Feature Engineering Is Model-Dependent

Different models need different help. Linear models often benefit from scaling, monotonic transformations, interactions, and careful handling of non-linearity. Tree-based models are more tolerant of skewed numeric variables, but still benefit from clean categorical encodings and meaningful aggregates.

This is why blindly applying a generic feature recipe is risky. A feature is not good in isolation. It is good when it makes the chosen model solve the actual problem more reliably.

What I Would Remember

• Feature engineering is still central for tabular machine learning.
• High-cardinality categorical variables need careful encoding, not blind one-hot expansion.
• Target encoding must be regularized and computed out of fold.
• Leakage usually looks like excellent validation performance before it looks like a bug.
• Every feature must be checked against prediction-time availability.
• The best features are not the cleverest ones. They are the ones that survive production assumptions.

References

1. Kuhn, M., and Johnson, K. (2019). Feature Engineering and Selection: A Practical Approach for Predictive Models. CRC Press.
2. Micci-Barreca, D. (2001). A Preprocessing Scheme for High-Cardinality Categorical Attributes in Classification and Prediction Problems. ACM SIGKDD Explorations Newsletter, 3(1), 27-32.
3. Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., and Gulin, A. (2018). CatBoost: Unbiased Boosting with Categorical Features. Advances in Neural Information Processing Systems, 31.
4. Chen, T., and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794.