Python sklearn Decision Tree and Random Forest Feature Importance

When you need to explain why a model makes a certain prediction, decision trees and random forests in scikit-learn offer a direct path: feature importance. This article covers python sklearn decision tree random forest and feature importance in practice, from training the models to interpreting the importance scores they produce.

Training a Decision Tree with scikit-learn

A decision tree is a supervised learning model that splits data recursively based on feature thresholds. In scikit-learn, the DecisionTreeClassifier and DecisionTreeRegressor classes implement this algorithm. The classifier is used for categorical targets, while the regressor handles continuous targets.

Here is a minimal example using the built-in Iris dataset:

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris

X, y = load_iris(return_X_y=True)
clf = DecisionTreeClassifier(random_state=42)
clf.fit(X, y)

The random_state parameter ensures reproducibility. Without it, the tree construction can vary between runs because the algorithm may use random tie-breaking when choosing splits. The fitted clf object now contains the learned tree structure and can be used to make predictions.

Decision trees are easy to inspect. You can export the tree as a Graphviz DOT file or use sklearn.tree.plot_tree to visualize it. However, a single tree is often sensitive to small changes in the training data, which is why random forests are frequently preferred.

Training a Random Forest with scikit-learn

A random forest is an ensemble of decision trees, typically trained with bootstrap sampling and random feature selection at each split. In scikit-learn, RandomForestClassifier and RandomForestRegressor provide this functionality.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, y)

The n_estimators parameter controls the number of trees in the forest. More trees generally improve stability but increase training time and memory usage. The random_state parameter again ensures reproducibility.

Random forests reduce overfitting compared to a single tree by averaging the predictions of many trees. This averaging also makes the model more robust to noise in the training data. The tradeoff is that a random forest is more expensive to train and evaluate than a single decision tree, especially with large datasets and many estimators.

Extracting Feature Importance from Tree Models

Both decision trees and random forests expose a feature_importances_ attribute after fitting. This attribute is an array of floating-point values, one per feature, that sum to 1.0. Higher values indicate that the feature contributes more to the model's decisions.

importances = clf.feature_importances_
print(importances)

For the random forest, the attribute is accessed the same way:

rf_importances = rf.feature_importances_
print(rf_importances)

The importance values are computed from the mean decrease in impurity (MDI). For each split in a tree, the decrease in impurity (Gini impurity for classification, variance for regression) is weighted by the number of samples reaching that node. These weighted decreases are summed for each feature and normalized across all features. For a random forest, the importance is averaged over all trees in the ensemble.

To map importance values to feature names, you can use a simple loop or a pandas Series:

import pandas as pd

feature_names = load_iris().feature_names
importance_series = pd.Series(rf_importances, index=feature_names).sort_values(ascending=False)
print(importance_series)

This gives you a ranked list of features, which is useful for feature selection or for communicating model behavior to stakeholders.

Interpreting Feature Importance Values

The numerical values themselves are only meaningful in a relative sense. A feature with an importance of 0.3 is considered twice as important as a feature with 0.15, but the absolute scale depends on the dataset and the model configuration. What matters is the ranking and the magnitude differences.

A common way to visualize importance is a horizontal bar chart:

import matplotlib.pyplot as plt

importance_series.plot.barh()
plt.xlabel('Importance')
plt.title('Random Forest Feature Importance')
plt.show()

When interpreting these values, keep in mind that they reflect how much each feature reduces impurity in the tree splits. They do not indicate the direction of the effect (positive or negative) or whether the relationship is linear. A feature that is important for the model may still have a complex, nonlinear relationship with the target.

For a single decision tree, importance can be noisy because the tree tends to overfit. Random forest importance is more stable because it averages over many trees, but it is still subject to the biases described in the next section.

Limitations and Pitfalls of Feature Importance

Feature importance from tree-based models is a useful diagnostic, but it has known limitations that you should account for when drawing conclusions.

High-cardinality features can be overvalued. Features with many unique values, such as a user ID or a timestamp, can artificially inflate importance because the tree can easily split on them to isolate small groups of samples. This does not necessarily mean the feature is genuinely predictive in a generalizable way.

Correlated features split importance. If two features are highly correlated, the importance may be distributed between them, making each appear less important than they would be if only one were present. This can mislead you into thinking neither is relevant when in fact one or both are.

Importance is not causal. A feature can be important because it is a proxy for another variable, not because it has a direct effect on the target. For example, in a housing price model, the number of bedrooms might be important, but the underlying cause could be the overall house size, which is correlated with bedroom count.

Impurity-based importance is biased towards features with many categories. This is a well-known issue, especially for categorical features that are one-hot encoded. The bias arises because the impurity reduction can be larger when a feature allows many splits.

For more reliable importance estimates, consider using permutation importance, which measures the drop in model performance when a feature's values are randomly shuffled. Scikit-learn provides sklearn.inspection.permutation_importance for this purpose. Permutation importance is more computationally expensive but gives a more direct measure of predictive contribution.

Practical Considerations for Production

When you use feature importance in a production setting, the way you interpret and act on it matters.

Feature selection: You can use importance scores to reduce the number of features before training a final model. However, always validate the reduced feature set with cross-validation. A feature that ranks low in one model might be useful in combination with others, and dropping it could hurt performance.

Model monitoring: Feature importance can change when the data distribution shifts. If you retrain models periodically, tracking importance over time can help you detect when the model's behavior is drifting. This is especially relevant for random forests, where the ensemble structure can mask individual tree instability.

Memory and latency: Random forests store all trees in memory, which can be large for big datasets or many estimators. For online prediction, a single decision tree is faster and lighter, but at the cost of accuracy and stability. If you need both speed and robustness, consider using a smaller forest or a distilled model.

Explainability: Feature importance is a global explanation technique; it tells you which features matter on average, not why a specific prediction was made. For local explanations, you would need methods like SHAP or LIME. Still, feature importance is a quick and interpretable first step in understanding what your model is doing.

When you work with python sklearn decision tree random forest and feature importance, always treat the importance scores as a heuristic, not as a ground truth. Combine them with domain knowledge and additional validation to make sound engineering decisions.