Decision Trees & Random Forests
A decision tree is like a tree of questions — each node asks one question, and you follow the answer down to a leaf, which is your prediction.
1. What You'll Learn
- Decision tree fundamentals: information gain / gain ratio / Gini index, and the ID3 / C4.5 / CART algorithm family
- Tree pruning: pre-pruning (max_depth / min_samples_leaf) and post-pruning
- Random Forest: the Bagging idea, random feature selection, and OOB estimation
- Feature importance analysis: mean decrease impurity vs. permutation importance
- Bob's product category grading: using a Random Forest to predict whether a product will become a hit
2. A Real Story from an E-commerce Product Manager
(1) The Pain Point: Hit-product prediction relied entirely on gut feeling, with a hit rate under 30%
Bob needed to pick potential hit products out of 1 thousand new items. In the past this was pure judgment, and his hit rate was only 30%. Each hit product brings in an average of 500 thousand USD in monthly revenue, so missing one is a huge loss. Gut feeling can't be quantified, reused, or iterated on.
(2) The Random Forest Solution
A Random Forest can automatically discover hit-product patterns from historical product data — price range, category characteristics, first-week sales trends — and it naturally provides a feature importance ranking.
PYTHON
from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
rf.fit(X_train, y_train)
print(f"OOB Accuracy: {rf.oob_score_:.3f}")
print(f"Top feature: {feature_names[rf.feature_importances_.argmax()]}")
(3) The Payoff: Hit-product prediction accuracy rose from 30% to 65%
Bob replaced gut feeling with a Random Forest, raising his hit-product prediction accuracy from 30% to 65%. He now identifies 5 extra hit products each month, adding roughly 3 million USD in annual revenue.
3. How Decision Trees Work
(1) Splitting Criteria
At each node, a decision tree picks the best feature and threshold to split on. There are three common criteria:
graph TB
ROOT[Root Node<br/>All Data] --> SPLIT1{Feature: price<br/>Threshold: 50 USD}
SPLIT1 -->|≤ 50| LEFT[Left Child<br/>60% hit products]
SPLIT1 -->|> 50| RIGHT[Right Child<br/>10% hit products]
LEFT --> SPLIT2{Feature: first_week_sales}
SPLIT2 -->|> 500| LEAF1[Leaf: HIT<br/>90% confidence]
SPLIT2 -->|≤ 500| LEAF2[Leaf: NOT HIT<br/>40% confidence]
RIGHT --> LEAF3[Leaf: NOT HIT<br/>5% confidence]
| Criterion | Core Formula | Preference | Algorithm Family |
|---|---|---|---|
| Information Gain | $H(D) - H(D | A)$ | Favors multi-valued features |
| Gain Ratio | Information Gain / Intrinsic Value | Corrects multi-value bias | C4.5 |
| Gini Impurity | $1 - \sum p_i^2$ | Favors large-class purity | CART |
▶ Example: Decision Tree Classification on Iris
PYTHON
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report
import matplotlib.pyplot as plt
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
iris.data, iris.target, test_size=0.3, random_state=42, stratify=iris.target
)
# Train with pre-pruning
tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=5, random_state=42)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Tree depth: {tree.get_depth()}")
print(f"Number of leaves: {tree.get_n_leaves()}")
# Visualize tree
fig, ax = plt.subplots(figsize=(14, 8))
plot_tree(tree, feature_names=iris.feature_names,
class_names=iris.target_names, filled=True, rounded=True, ax=ax)
plt.title("Decision Tree (max_depth=3)")
plt.tight_layout()
plt.savefig("decision_tree.png", dpi=150)
Output:
TEXT
# Runs successfully
(2) Pruning Strategies
▶ Example: Comparing Pre-pruning Parameters
PYTHON
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
configs = {
"No pruning": DecisionTreeClassifier(random_state=42),
"max_depth=2": DecisionTreeClassifier(max_depth=2, random_state=42),
"max_depth=3": DecisionTreeClassifier(max_depth=3, random_state=42),
"min_samples_leaf=5": DecisionTreeClassifier(min_samples_leaf=5, random_state=42),
"max_depth=3 + min_samples_leaf=5": DecisionTreeClassifier(
max_depth=3, min_samples_leaf=5, random_state=42),
}
for name, model in configs.items():
scores = cross_val_score(model, X, y, cv=5, scoring="accuracy")
model.fit(X, y)
print(f"{name:35s}: Accuracy={scores.mean():.3f}, Leaves={model.get_n_leaves()}")
Output:
TEXT
# Runs successfully
| Pruning Type | Parameter | Effect | Recommended Value |
|---|---|---|---|
| Pre-pruning | max_depth | Limits tree depth | 3-10 |
| Pre-pruning | min_samples_leaf | Minimum samples per leaf | 5-20 |
| Pre-pruning | min_samples_split | Minimum samples to split | 10-40 |
| Pre-pruning | max_features | Features considered per split | sqrt(n_features) |
| Post-pruning | ccp_alpha | Cost-complexity pruning | Choose via GridSearch |
4. Random Forest
(1) The Bagging Idea
Random Forest = Bagging (Bootstrap Aggregating) + random feature selection.
graph TB
DATA[Original Data] --> B1[Bootstrap Sample 1]
DATA --> B2[Bootstrap Sample 2]
DATA --> B3[Bootstrap Sample 3]
DATA --> BN[Bootstrap Sample N]
B1 --> T1[Tree 1<br/>Random Feature Subset]
B2 --> T2[Tree 2<br/>Random Feature Subset]
B3 --> T3[Tree 3<br/>Random Feature Subset]
BN --> TN[Tree N<br/>Random Feature Subset]
T1 --> VOTE[Majority Vote<br/>/ Average]
T2 --> VOTE
T3 --> VOTE
TN --> VOTE
▶ Example: Random Forest Classification + OOB Evaluation
PYTHON
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
rf = RandomForestClassifier(
n_estimators=100,
max_depth=None,
oob_score=True,
random_state=42,
)
rf.fit(X_train, y_train)
print(f"OOB Score: {rf.oob_score_:.4f}")
print(f"Test Accuracy: {accuracy_score(y_test, rf.predict(X_test)):.4f}")
print(f"Number of trees: {rf.n_estimators}")
Output:
TEXT
# Runs successfully
(2) Hyperparameter Tuning
▶ Example: Random Forest Regression — SalesPredict Sales Forecasting
PYTHON
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np
rng = np.random.default_rng(42)
n = 500
X = rng.uniform(0, 100, (n, 6))
y = 50 + 0.8 * X[:, 0] + 1.2 * X[:, 1] - 0.5 * X[:, 2] + rng.normal(0, 5, n)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Compare different n_estimators
for n_est in [10, 50, 100, 200]:
rf = RandomForestRegressor(n_estimators=n_est, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
score = rf.score(X_test, y_test)
mae = mean_absolute_error(y_test, rf.predict(X_test))
print(f"n_estimators={n_est:3d}: R²={score:.4f}, MAE={mae:.2f}")
Output:
TEXT
# Runs successfully
5. Feature Importance Analysis
(1) Mean Decrease Impurity vs. Permutation Importance
▶ Example: Comparing the Two Feature Importance Methods
PYTHON
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
import numpy as np
X, y = load_iris(return_X_y=True)
feature_names = load_iris().feature_names
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, y)
# Method 1: MDI (built-in, fast but biased)
mdi_importance = rf.feature_importances_
# Method 2: Permutation importance (slower but more reliable)
perm_result = permutation_importance(rf, X, y, n_repeats=30, random_state=42, n_jobs=-1)
perm_importance = perm_result.importances_mean
# Compare
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
y_pos = np.arange(len(feature_names))
axes[0].barh(y_pos, mdi_importance, color="#2196F3")
axes[0].set_yticks(y_pos)
axes[0].set_yticklabels(feature_names)
axes[0].set_title("MDI Feature Importance")
axes[1].barh(y_pos, perm_importance, color="#4CAF50")
axes[1].set_yticks(y_pos)
axes[1].set_yticklabels(feature_names)
axes[1].set_title("Permutation Feature Importance")
plt.tight_layout()
plt.savefig("feature_importance.png", dpi=150)
Output:
TEXT
# Runs successfully
| Dimension | MDI Importance | Permutation Importance |
|---|---|---|
| Computation speed | Fast (computed during training) | Slow (requires repeated predictions) |
| Bias | Biased toward high-cardinality features | Unbiased |
| Applicability | Built into tree models | Works with any model |
| Reliability | Moderate | More reliable |
▶ Example: Bob's Hit-Product Feature Ranking
PYTHON
from sklearn.ensemble import RandomForestClassifier
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 1000
df = pd.DataFrame({
"price_usd": rng.uniform(10, 200, n),
"first_week_sales": rng.integers(10, 2000, n),
"category_trend_score": rng.uniform(0, 1, n),
"ad_budget_k": rng.uniform(1, 50, n),
"review_score": rng.uniform(1, 5, n),
"return_rate": rng.uniform(0, 0.3, n),
"stock_depth": rng.integers(10, 500, n),
})
df["is_hit"] = (
(df["first_week_sales"] > 500)
& (df["category_trend_score"] > 0.6)
& (df["price_usd"] < 100)
).astype(int) | (rng.random(n) < 0.05) # Add 5% noise
X = df.drop(columns=["is_hit"])
y = df["is_hit"]
rf = RandomForestClassifier(n_estimators=200, random_state=42)
rf.fit(X, y)
importance = pd.DataFrame({
"feature": X.columns,
"importance": rf.feature_importances_,
}).sort_values("importance", ascending=False)
print("Feature Importance for Hit Product Prediction:")
print(importance.to_string(index=False))
Output:
TEXT
Feature Importance for Hit Product Prediction:
❓ FAQ
Q Why is a Random Forest better than a single decision tree?
A For two reasons — 1) Bagging reduces variance (multiple trees vote and average); 2) random feature selection lowers the correlation between trees, making the ensemble stronger. A single tree tends to overfit.
Q Is a larger n_estimators always better?
A Not necessarily. Past a certain point the gains diminish and you only add computation time. Usually 100-500 is enough. Watch the OOB score to see when it stabilizes.
Q What is the OOB Score?
A Out-of-Bag evaluation. Each tree is trained on about 63% of the data, and the remaining 37% naturally forms a validation set. The OOB Score is the average prediction across all trees on their own OOB samples — equivalent to cross-validation, but free.
Q Do decision trees need standardization?
A No. Decision trees split on feature thresholds, so they're unaffected by scale. But if a Random Forest is combined with other models (such as a Scaler in a Pipeline), then you may need it.
Q What if all feature importances are close to 0?
A It suggests the features may genuinely have little relationship with the target. Try feature interactions or nonlinear transformations, or use permutation importance to check whether MDI is biased.
Q Can a Random Forest handle class imbalance?
A Yes. Set class_weight="balanced" or class_weight={0:1, 1:10}. You can also combine it with oversampling (SMOTE) or undersampling.
📖 Summary
- A decision tree builds recursively using a splitting criterion (information gain / Gini index) until a stopping condition is met
- Pruning prevents overfitting: pre-pruning (limit depth / leaf sample count) + post-pruning (ccp_alpha)
- Random Forest = Bagging + random feature selection, reducing variance and resisting overfitting
- The OOB Score is the Random Forest's "free" cross-validation, replacing train/test split evaluation
- Two feature importance methods: MDI (fast but biased) vs. Permutation (slow but reliable)
- The Random Forest is the "Swiss army knife" of the ML toolbox — almost always a solid baseline choice
📝 Exercises
- Basic (Difficulty ⭐): Use DecisionTreeClassifier to classify Iris, varying max_depth from 1 to 5, and plot the accuracy vs. depth curve. Hint: loop over training with
cross_val_score. - Intermediate (Difficulty ⭐⭐): Use RandomForestRegressor on California Housing, comparing R² and OOB Score for n_estimators=[10,50,100,200]. Hint: set
oob_score=True. - Challenge (Difficulty ⭐⭐⭐): Implement Bob's full hit-product prediction pipeline — generate simulated data, train a RandomForestClassifier, analyze features with permutation_importance, handle imbalance with class_weight (hit products < 10%), and output a classification report. Hint: refer to the hit-product example in Section 5.
← Previous: Logistic Regression | Next: Support Vector Machines →
