Random Forests

Jeremy Howard's case for random forests, from the fast.ai course:

• They handle both classification and regression.
• They work with columns of any kind: pixels, postcodes, revenue.
• They are hard to overfit badly, and when they do overfit it is easy to fix.
• They come with a built-in estimate of generalisation (the out-of-bag score), so a single dataset goes a long way.
• They make few statistical assumptions: no normality, no linearity.
• They need little feature engineering.

He also argues the curse of dimensionality is largely irrelevant in machine learning, and that you should add any column that might carry information. That is broadly right for random forests: each split chooses the most useful column, so irrelevant ones are mostly ignored. It is overstated as a general claim. In high dimensions distances between points do concentrate, and distance-based methods such as k-nearest neighbours degrade noticeably when many irrelevant features are added.

The running example is the Blue Book for Bulldozers dataset, predicting the log of the sale price, with a time-based split: training on earlier sales and validating on later ones.

A single tree

A decision tree is a sequence of binary splits. At the root there are 28,588 rows with a mean log price of 10.098; predicting that mean for every row gives a mean squared error of 0.479.

The best first split is Coupler_System <= -0.5. It leaves 3,140 rows with an MSE of 0.113 and 25,448 rows with an MSE of 0.412. The tree scores a split by the average MSE of the two groups, weighted by their sizes:

To find that split, the tree tries every column and, for each, every threshold between consecutive distinct values. A boolean column has only one candidate, col <= 0.5.

Left alone, a tree keeps splitting until every leaf holds one row. It then predicts the training data perfectly (here, a training RMSE of 0 against a validation RMSE of 0.379) and generalises poorly. min_samples_leaf and max_depth stop it earlier.

From trees to a forest

We'd like trees deep enough to capture interactions without overfitting. A random forest gets that by averaging many deep trees, each trained on a different bootstrap sample of the rows. Each tree overfits its own sample, but in different ways, so their errors are only weakly correlated and largely cancel when averaged. Averaging reduces variance; it does not remove bias, so the forest is only as good as what the trees can represent.

Out-of-bag score

Each tree's bootstrap sample leaves out roughly a third of the rows. For each row, we can average the predictions of only the trees that never saw it, and score those predictions. That out-of-bag (OOB) score estimates generalisation without a separate validation set.

Here the OOB $R^2$ is 0.913: below the training score of 0.988, and above the 0.883 on the later validation period. That is what we'd expect, since the OOB rows come from the same period as the training data, while the validation set also has to cope with change over time.

Hyperparameters

Number of trees

More trees never make the forest worse in expectation; they just take longer. The improvement flattens out, so plot validation error against the number of trees and stop where the curve does:

Minimum samples per leaf

Setting min_samples_leaf above 1 makes shallower trees that train faster. Each tree is weaker on its own but generalises better. Here min_samples_leaf=3 nudged validation $R^2$ from 0.883 to 0.886.

Maximum features

We want the trees to be as uncorrelated as possible. If one feature is especially predictive, every tree will split on it first, they'll look alike, and other interactions get missed. So as well as sampling rows per tree, we sample columns per split: max_features sets how many columns each split may choose from, as a count or a fraction.

For RandomForestClassifier the default is "sqrt", the square root of the number of columns. For RandomForestRegressor, used here, the default since scikit-learn 1.1 is 1.0: all columns, which is plain bagging. Setting max_features=0.5 (with min_samples_leaf=3) raised validation $R^2$ to 0.894.

Lower max_features makes individual trees weaker but less correlated, so it tends to pay off more as the number of trees grows. scikit-learn's OOB error example shows this.

Feature importance

Which columns matter, and how to measure that without being misled, is in Feature Importance. Dropping the less important columns here barely changed the validation score, which makes the model easier to interpret at little cost.

Confidence from tree variance

The forest's prediction is the mean of its trees' predictions. Their standard deviation is a cheap measure of confidence: rows where the trees disagree are ones to treat with more caution.

preds = np.stack([t.predict(X_valid) for t in m.estimators_])
pred_mean, pred_std = preds.mean(axis=0), preds.std(axis=0)

Adapted from the fast.ai book and Introduction to Machine Learning for Coders course.

Original notebook

Tags: Machine Learning