A single decision tree is easy to inspect, but it can be unstable. A small change in the training sample may change an early split, which changes the branches below it and can produce a different prediction.

Random Forest addresses this weakness by training many randomized trees and combining their outputs. Each tree sees a different bootstrap sample, and each split considers only a random subset of features. The trees are therefore related, but they are not identical.

The method is often a strong baseline for tabular classification and regression. It can represent nonlinear relationships, interactions, and mixed regions of feature space without requiring feature scaling or a predefined functional form.

This article explains how a forest is constructed, why randomization matters, how out-of-bag evaluation works, which hyperparameters deserve attention, and how to interpret the fitted model without treating feature importance as causal evidence.

What you should know first

• Basic binary classification concepts, including train and test data.
• Metrics such as ROC-AUC, accuracy, and balanced accuracy.
• Basic Python, NumPy, and the estimator interface used by scikit-learn.

The article focuses on classification, but most of the construction principles also apply to random-forest regression.

Why one decision tree is often not enough

A decision tree repeatedly partitions the feature space. At each node, it chooses a feature and threshold intended to improve the purity of the resulting child nodes.

This produces an interpretable sequence of rules. It also creates a dependency between early and later decisions. If a slightly different training sample produces another first split, much of the tree can change.

This behavior is commonly described as high variance. The tree can fit complex patterns, but its learned structure may depend strongly on the particular sample used for training.

Pruning, limiting tree depth, or requiring larger leaves can reduce this instability. Random Forest takes another route: it builds many trees with deliberately different training conditions and aggregates their predictions.

How the forest reduces instability

Assume that the forest contains \(B\) trees. For regression, each tree produces a numerical prediction \(T_b(x)\), and the forest averages them:

$$ \hat{y}(x) = \frac{1}{B} \sum_{b=1}^{B} T_b(x) $$

For classification, the result is commonly introduced as a vote. In scikit-learn, RandomForestClassifier averages the class probabilities produced by the trees and selects the class with the highest mean probability.

The benefit of averaging depends on both the quality and the correlation of the trees.

If every tree has variance \(\sigma^2\) and the average pairwise correlation between trees is \(\rho\), a simplified variance expression for their mean is:

$$ Var(\bar{T}) = \sigma^2 \left( \rho + \frac{1-\rho}{B} \right) $$

This expression assumes comparable tree variances and correlations, so it is an explanatory approximation rather than a complete description of a fitted forest.

Increasing \(B\) reduces the second term. The first term remains when the trees make highly correlated errors. Random Forest therefore uses two forms of randomization: bootstrap samples make the training sets different, while random feature subsets reduce similarity between tree structures.

Bootstrap sampling gives each tree a different dataset

For every tree, the algorithm draws a training sample with replacement from the original training set. Because sampling uses replacement, one observation can appear several times while another may not appear at all.

For a dataset containing \(n\) rows, the probability that a specific row is not selected in one bootstrap sample is:

$$ \left(1-\frac{1}{n}\right)^n $$

As \(n\) grows, this value approaches:

$$ e^{-1} \approx 0.368 $$

Approximately 36.8 percent of the training observations are therefore omitted from an individual tree's bootstrap sample. These omitted rows are called out-of-bag observations for that tree.

The exact omitted proportion varies from tree to tree. Across a sufficiently large forest, each training observation is normally out-of-bag for multiple trees.

Random feature subsets prevent a few predictors from dominating every tree

Bootstrap samples alone may not create enough diversity when one or two strong predictors dominate the data.

If every tree examines every feature at every split, many trees may choose the same strong predictor near the root. Their structures and errors then remain correlated.

Random Forest limits the features considered at each split. A tree may contain all available features across its complete structure, but an individual split sees only a randomly selected subset.

The max_features parameter controls the size of this subset. Smaller subsets usually create more diverse trees, but they may also prevent an individual tree from using an important predictor when it would be useful.

The parameter therefore controls a trade-off between tree strength and tree diversity.

Out-of-bag predictions provide an internal estimate

An out-of-bag prediction for one training observation is calculated using only trees whose bootstrap samples did not contain that observation.

For classification, the forest aggregates the probabilities or class decisions from those trees. The result can be compared with the known training label without asking the same tree to predict an observation it used for fitting.

OOB evaluation is computationally convenient because it uses the resampling already performed during forest construction. It does not require a separate validation split for the internal estimate.

It is not a universal replacement for a holdout set or task-specific cross-validation. A final evaluation may still need to preserve time order, customer groups, locations, devices, or another dependency that ordinary bootstrap sampling does not represent.

The main hyperparameters control different behaviors

n_estimators is usually not the main control for overfitting. Adding more trees commonly stabilizes the ensemble because predictions are aggregated. The cost is additional computation and memory.

Tree complexity is controlled more directly through parameters such as max_depth, min_samples_leaf, and min_samples_split.

A complete experiment on a real classification dataset

The experiment uses the Wisconsin Diagnostic Breast Cancer dataset distributed through scikit-learn. It contains 569 observations and 30 numerical features derived from digitized images of fine-needle aspirates.

The original dataset is available from the UCI Machine Learning Repository under the CC BY 4.0 license. The example changes the target encoding so that malignant cases are represented by one.

This dataset is used only to demonstrate model evaluation. The experiment is not evidence that the resulting model is suitable for clinical decisions.

The evaluation has three parts. Repeated stratified cross-validation compares one decision tree with a forest. A stratified holdout set provides final test metrics. OOB error and permutation importance provide additional diagnostic views.

import warnings

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.metrics import balanced_accuracy_score, roc_auc_score
from sklearn.model_selection import (
    RepeatedStratifiedKFold,
    cross_val_score,
    train_test_split,
)
from sklearn.tree import DecisionTreeClassifier

SEED = 42

data = load_breast_cancer()
X = data.data
y = (data.target == 0).astype(int)
feature_names = np.array(data.feature_names)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=SEED,
)

cv = RepeatedStratifiedKFold(
    n_splits=5,
    n_repeats=5,
    random_state=SEED,
)

single_tree = DecisionTreeClassifier(
    random_state=SEED,
)

forest = RandomForestClassifier(
    n_estimators=400,
    max_features="sqrt",
    oob_score=True,
    random_state=SEED,
    n_jobs=-1,
)

tree_cv_auc = cross_val_score(
    single_tree,
    X,
    y,
    scoring="roc_auc",
    cv=cv,
    n_jobs=-1,
)

forest_cv_auc = cross_val_score(
    forest,
    X,
    y,
    scoring="roc_auc",
    cv=cv,
    n_jobs=-1,
)

forest.fit(X_train, y_train)

test_probability = forest.predict_proba(X_test)[:, 1]
test_prediction = forest.predict(X_test)

tree_counts = [2, 3, 5, 8, 10, 15, 20, 30, 50, 100, 200, 400]
oob_error = []
test_error = []

for n_estimators in tree_counts:
    model = RandomForestClassifier(
        n_estimators=n_estimators,
        max_features="sqrt",
        oob_score=True,
        random_state=SEED,
        n_jobs=-1,
    )

    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        model.fit(X_train, y_train)

    oob_error.append(1.0 - model.oob_score_)
    test_error.append(1.0 - model.score(X_test, y_test))

importance = permutation_importance(
    forest,
    X_test,
    y_test,
    scoring="roc_auc",
    n_repeats=30,
    random_state=SEED,
    n_jobs=-1,
)

top_features = np.argsort(
    -importance.importances_mean
)[:5]

print(
    f"dataset_samples={X.shape[0]} "
    f"features={X.shape[1]} "
    f"malignant_rate={y.mean():.3f}"
)

print(
    "single_tree_cv_auc="
    f"{tree_cv_auc.mean():.3f}"
    f"+/-{tree_cv_auc.std():.3f}"
)

print(
    "random_forest_cv_auc="
    f"{forest_cv_auc.mean():.3f}"
    f"+/-{forest_cv_auc.std():.3f}"
)

print(
    f"forest_oob_accuracy="
    f"{forest.oob_score_:.3f}"
)

print(
    f"forest_test_auc="
    f"{roc_auc_score(y_test, test_probability):.3f}"
)

print(
    f"forest_test_balanced_accuracy="
    f"{balanced_accuracy_score(y_test, test_prediction):.3f}"
)

print(
    "oob_error="
    + str({
        count: round(error, 3)
        for count, error
        in zip(tree_counts, oob_error)
    })
)

print(
    "test_error="
    + str({
        count: round(error, 3)
        for count, error
        in zip(tree_counts, test_error)
    })
)

print(
    "top_permutation_features="
    + str([
        (
            str(feature_names[index]),
            round(
                float(
                    importance.importances_mean[index]
                ),
                4,
            ),
            round(
                float(
                    importance.importances_std[index]
                ),
                4,
            ),
        )
        for index in top_features
    ])
)

The following output was produced by executing the code:

dataset_samples=569 features=30 malignant_rate=0.373
single_tree_cv_auc=0.925+/-0.031
random_forest_cv_auc=0.990+/-0.008
forest_oob_accuracy=0.953
forest_test_auc=0.996
forest_test_balanced_accuracy=0.943
oob_error={2: 0.195, 3: 0.138, 5: 0.08, 8: 0.066, 10: 0.045, 15: 0.045, 20: 0.045, 30: 0.045, 50: 0.047, 100: 0.045, 200: 0.047, 400: 0.047}
test_error={2: 0.077, 3: 0.063, 5: 0.035, 8: 0.049, 10: 0.049, 15: 0.042, 20: 0.042, 30: 0.042, 50: 0.042, 100: 0.035, 200: 0.042, 400: 0.042}
top_permutation_features=[('worst area', 0.01, 0.004), ('worst concave points', 0.0082, 0.0039), ('worst perimeter', 0.0071, 0.0033), ('worst radius', 0.0056, 0.0027), ('mean concave points', 0.0032, 0.0026)]

The forest is more stable across validation folds

The single decision tree reaches a mean repeated cross-validation ROC-AUC of 0.925 with a standard deviation of 0.031.

The Random Forest reaches 0.990 with a standard deviation of 0.008 under the same validation splits.

In this experiment, the forest therefore improves both the mean score and the stability of the score between folds. This is consistent with the intended effect of aggregating multiple randomized trees.

The result does not establish that Random Forest will dominate a decision tree on every dataset. It shows what happened for this dataset, target encoding, metric, and validation design.

The holdout metrics answer different questions

The fitted forest obtains a holdout ROC-AUC of 0.996. ROC-AUC measures how effectively the scores rank malignant observations above benign observations across possible thresholds.

The balanced accuracy is 0.943. This metric averages recall across the two classes and therefore describes the selected classification threshold rather than ranking alone.

A model can have a high ROC-AUC and still use an unsuitable operating threshold. In a real system, the threshold should be selected from the relative consequences of false negatives and false positives.

The OOB accuracy is 0.953. It is close to the ordinary holdout accuracy represented by the test-error curve, but the two values are not expected to match exactly. They use different observations, prediction ensembles, and sampling mechanisms.

The OOB curve shows when more trees stop changing much

With only two or three trees, many observations have too few out-of-bag predictions. The corresponding OOB error is unstable and should not be treated as a reliable estimate.

The measured OOB error falls from 0.195 with two trees to 0.080 with five trees. It reaches approximately 0.045 by ten trees and then changes little in this experiment.

The holdout error also stabilizes, although it moves between 0.035 and 0.049 because the test set is small.

The plateau does not mean that ten trees are universally sufficient. Another random seed, larger dataset, more difficult target, or different hyperparameters may require a larger forest.

How to tune a Random Forest without searching everything

I would begin with a forest large enough for validation metrics to stabilize. After that, the most useful search usually concerns tree regularization and feature randomization rather than continuing to increase the number of trees indefinitely.

min_samples_leaf is a practical regularization control. Increasing it prevents leaves supported by very few observations and often produces smoother probabilities.

max_features controls tree correlation. Reducing it can improve ensemble diversity, but an excessively small value may weaken the trees because useful predictors are often unavailable at a split.

max_depth can limit complexity directly, although fully grown trees with bootstrap aggregation and appropriate leaf constraints can also work well. The correct choice depends on validation results and the required inference cost.

Hyperparameter selection should use the validation design appropriate for the data. Random cross-validation is unsuitable when observations are connected through time, customer identity, location, or another grouping structure.

Permutation importance measures reliance on a fitted feature

Permutation importance starts with the score of a fitted model on an evaluation dataset.

One feature column is shuffled, breaking its relationship with the target and with the remaining rows. The model is scored again. The importance is the decrease in the selected metric:

$$ I_j = s_{\mathrm{baseline}} - s_{\mathrm{permuted},j} $$

A larger decrease means that the fitted model relied more strongly on that feature for the measured dataset and metric.

The experiment calculates importance on the holdout set using ROC-AUC and repeats each permutation 30 times.

The largest measured mean decrease is 0.010 for worst area. Several related geometric features follow with similar or smaller values.

Small importance values do not mean the features contain no information

The dataset contains several correlated measurements of related cell-nucleus properties, including radius, perimeter, area, and concavity summaries.

When predictors are redundant, shuffling one of them may cause only a small score decrease because the forest can still use correlated alternatives.

Permutation importance therefore measures the reliance of this fitted model on one feature while the others remain available. It does not measure the feature's intrinsic value in isolation.

It also does not establish causality. A high importance value shows predictive reliance under the evaluation distribution, not that changing the measured property would cause the diagnosis to change.

Why impurity-based importance needs caution

Tree ensembles expose an impurity-based importance through feature_importances_. It accumulates the weighted impurity reductions attributed to each feature across the trees.

This calculation is fast and can help inspect model internals. It is based on the training process, not on a held-out performance decrease.

The scikit-learn documentation warns that impurity-based importance can favor high-cardinality features and may not describe which variables support performance on unseen data.

Permutation importance on a validation or test set avoids those specific issues, although it remains sensitive to correlated predictors, dataset size, metric choice, and random permutations.

OOB evaluation and cross-validation are complementary

OOB evaluation is useful during model development because it comes from the fitted bootstrap ensemble. It can help identify whether the forest is large enough and whether a parameter change improves the internal estimate.

Cross-validation measures performance across several explicit training and validation partitions. It is more flexible because the split logic can preserve class ratios, groups, or time order.

A final holdout set protects the last evaluation from repeated tuning decisions. It is most useful when it remains untouched until the model and evaluation policy are selected.

Common mistakes change the meaning of a good score

Leakage before the forest is fitted

Random Forest does not protect against target leakage. A feature derived after the outcome, a preprocessing step fitted on the complete dataset, or duplicated entities across train and test partitions can produce optimistic validation results.

The split must represent what information will be available when a future prediction is made.

Evaluating only accuracy on imbalanced data

Accuracy can hide poor performance for a minority class. The example reports balanced accuracy and ROC-AUC because they expose different properties.

For a real workflow, evaluation may also require class recall, precision, precision-recall curves, calibration, threshold cost, and segment-level stability.

Reading importance as a complete explanation

A feature ranking does not show whether the feature is causal, whether its effect is monotonic, or whether the model uses it consistently across observations.

Importance should be combined with data-quality inspection, segment analysis, partial dependence or other response analysis, and domain review where appropriate.

When Random Forest is a strong starting point

Random Forest is often effective for medium-sized tabular datasets with nonlinear relationships and interactions. It requires relatively little preprocessing and can provide a strong benchmark before more specialized modeling.

It is less suitable when the system requires smooth extrapolation outside the training range. Tree predictions are constructed from observed regions and do not naturally continue a numerical trend beyond those regions.

It may also be an unsuitable final choice when strict global transparency, very low latency, or a compact mathematical relationship is more important than flexible predictive performance.

Key takeaways

• Random Forest improves stability by aggregating trees trained on bootstrap samples and randomized feature subsets.
• OOB estimates are useful internal diagnostics, but final validation must reproduce the deployment split and decision metric.
• Permutation importance describes model reliance on an evaluation dataset; it does not establish causality or independent feature value.

Sources

1. Breiman, L. (2001). Random Forests. Machine Learning, 45, 5-32.
2. scikit-learn. Ensembles: Random Forests, RandomForestClassifier, and OOB Errors for Random Forests. Official documentation.
3. UCI Machine Learning Repository and scikit-learn. Breast Cancer Wisconsin Diagnostic Dataset, Dataset Loader, and Permutation Feature Importance.