Ensemble Learning
Three cobblers with their wits combined equal Zhuge Liang the master mind — Boosting chains weak learners together to make them strong, and XGBoost/LightGBM are the industrial-grade, turbocharged versions.
1. What You'll Learn
- Boosting fundamentals: the evolution path from AdaBoost → Gradient Boosting → XGBoost/LightGBM
- XGBoost: regularized objective function, column sampling, approximate quantile splitting, GPU acceleration
- LightGBM: GOSS/EFB innovations, histogram-based splitting, native categorical feature support, training speed advantages
- Hyperparameter tuning: learning_rate/n_estimators/max_depth/num_leaves/reg_alpha/reg_lambda
- Bob's core model choice: an XGBoost vs. LightGBM performance comparison on the SalesPredict data
2. A Real-World Story from an Algorithm Engineer
(1) The Pain Point: Random Forest Is Too Slow on Million-Scale Data
Bob's SalesPredict data has grown to 1 million rows, and training a Random Forest with 100 trees takes 45 minutes. With 10 experiments to run each day, just waiting for training eats up 7.5 hours. Alice's US data, at 2 million rows, is even slower. Training speed limits experimentation efficiency and indirectly slows down model iteration.
(2) The XGBoost/LightGBM Solution
Through optimizations like histogram-based splitting and column sampling, XGBoost and LightGBM cut training time by 5-10x while delivering even higher accuracy.
PYTHON
import xgboost as xgb
dtrain = xgb.DMatrix(X_train, label=y_train)
params = {"objective": "reg:squarederror", "max_depth": 6, "learning_rate": 0.1}
model = xgb.train(params, dtrain, num_boost_round=500)
(3) The Payoff: Training Time Drops from 45 Minutes to 5
After Bob replaced Random Forest with LightGBM, training on 1 million samples dropped from 45 minutes to 5 minutes. He can now run 80+ experiments per day, boosting model iteration speed by 9x.
3. The Boosting Ensemble Principle
(1) From AdaBoost to Gradient Boosting
The core idea of Boosting: train weak learners sequentially, with each new learner focusing on correcting the mistakes of the previous one.
graph TB
DATA[Original Data] --> M1[Model 1<br/>Weak Learner]
M1 --> E1[Errors from M1]
E1 --> W1[Up-weight Errors]
W1 --> M2[Model 2<br/>Focus on Hard Samples]
M2 --> E2[Residual Errors]
E2 --> M3[Model 3<br/>Focus on Remaining Errors]
M3 --> FINAL[Final Prediction<br/>= M1 + M2 + M3]
▶ Example: Understanding GradientBoosting by Hand
PYTHON
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error
import numpy as np
rng = np.random.default_rng(42)
X = rng.uniform(0, 10, (200, 1))
y = np.sin(X.squeeze()) + rng.normal(0, 0.2, 200)
# Step-by-step boosting visualization
residuals = y.copy()
predictions = np.zeros_like(y, dtype=float)
learning_rate = 0.1
for i in range(1, 51):
tree = DecisionTreeRegressor(max_depth=3)
tree.fit(X, residuals)
update = learning_rate * tree.predict(X)
predictions += update
residuals = y - predictions
if i in [1, 5, 10, 50]:
mse = mean_squared_error(y, predictions)
print(f"Round {i:2d}: MSE={mse:.4f}")
# Compare with sklearn's GradientBoosting
gb = GradientBoostingRegressor(n_estimators=50, max_depth=3, learning_rate=0.1, random_state=42)
gb.fit(X, y)
print(f"\nsklearn GB MSE: {mean_squared_error(y, gb.predict(X)):.4f}")
Output:
TEXT
# Runs successfully
(2) Boosting Evolution Comparison
| Dimension | AdaBoost | Gradient Boosting | XGBoost | LightGBM |
|---|---|---|---|---|
| Error correction method | Sample weighting | Fit residual gradients | Second-order gradients + regularization | Second-order gradients + GOSS |
| Regularization | None | Weak | Strong (L1+L2) | Strong (L1+L2) |
| Splitting algorithm | Exact | Exact | Approximate quantile | Histogram |
| Column sampling | No | No | Yes | Yes |
| Speed | Slow | Slow | Fast | Fastest |
4. XGBoost in Depth
(1) XGBoost's Key Innovations
▶ Example: XGBoost Regression Training
PYTHON
import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np
rng = np.random.default_rng(42)
n = 10000
X = rng.uniform(0, 100, (n, 8))
y = (50 + 0.8 * X[:, 0] + 1.2 * X[:, 1] - 0.5 * X[:, 2]
+ 0.3 * X[:, 3] * X[:, 4] # Interaction term
+ rng.normal(0, 5, n))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# XGBoost with sklearn API
model = xgb.XGBRegressor(
n_estimators=500,
max_depth=6,
learning_rate=0.1,
subsample=0.8,
colsample_bytree=0.8,
reg_alpha=0.1,
reg_lambda=1.0,
random_state=42,
early_stopping_rounds=50,
)
model.fit(
X_train, y_train,
eval_set=[(X_test, y_test)],
verbose=False,
)
y_pred = model.predict(X_test)
print(f"R²: {r2_score(y_test, y_pred):.4f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred):.2f}")
print(f"Best iteration: {model.best_iteration}")
Output:
TEXT
# Runs successfully
(2) Key XGBoost Parameters
| Parameter | Meaning | Recommended Range | Effect |
|---|---|---|---|
| n_estimators | Number of trees | 100-5000 | Use with early_stopping |
| max_depth | Maximum tree depth | 3-10 | Larger → overfitting |
| learning_rate | Learning rate | 0.01-0.3 | Smaller → needs more trees |
| subsample | Row sampling ratio | 0.6-1.0 | <1 → prevents overfitting |
| colsample_bytree | Column sampling ratio | 0.6-1.0 | <1 → prevents overfitting |
| reg_alpha | L1 regularization | 0-10 | Larger → sparser |
| reg_lambda | L2 regularization | 0-10 | Larger → smoother |
5. LightGBM in Depth
(1) LightGBM's Two Major Innovations
- GOSS (Gradient-based One-Side Sampling): keeps large-gradient samples and randomly samples small-gradient ones
- EFB (Exclusive Feature Bundling): bundles mutually exclusive sparse features to reduce the feature count
▶ Example: LightGBM Regression Training
PYTHON
import lightgbm as lgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np
# Using same data as XGBoost example
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# LightGBM with sklearn API
model = lgb.LGBMRegressor(
n_estimators=500,
max_depth=-1, # -1 means no limit, use num_leaves instead
num_leaves=31,
learning_rate=0.1,
subsample=0.8,
colsample_bytree=0.8,
reg_alpha=0.1,
reg_lambda=1.0,
random_state=42,
verbose=-1,
)
model.fit(
X_train, y_train,
eval_set=[(X_test, y_test)],
callbacks=[lgb.early_stopping(50, verbose=False)],
)
y_pred = model.predict(X_test)
print(f"R²: {r2_score(y_test, y_pred):.4f}")
print(f"MAE: {mean_absolute_error(y_test, y_pred):.2f}")
print(f"Best iteration: {model.best_iteration_}")
Output:
TEXT
# Runs successfully
(2) LightGBM's Native Categorical Feature Support
▶ Example: Passing Categorical Features Directly to LightGBM
PYTHON
import lightgbm as lgb
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import r2_score
rng = np.random.default_rng(42)
n = 5000
df = pd.DataFrame({
"ad_spend_k": rng.uniform(5, 100, n),
"traffic_k": rng.uniform(10, 500, n),
"category": rng.choice(["Elec", "Cloth", "Food", "Book", "Home"], n),
"region": rng.choice(["US", "EU", "CN"], n),
})
df["revenue_k"] = (
20 + 0.6 * df["ad_spend_k"] + 0.08 * df["traffic_k"]
+ df["category"].map({"Elec": 30, "Cloth": 15, "Food": 5, "Book": 3, "Home": 40})
+ rng.normal(0, 10, n)
)
X = df.drop(columns=["revenue_k"])
y = df["revenue_k"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# LightGBM native categorical feature support
model = lgb.LGBMRegressor(
n_estimators=200, learning_rate=0.1,
categorical_feature=["category", "region"], # Direct categorical support
random_state=42, verbose=-1,
)
model.fit(X_train, y_train)
print(f"R² with native categorical: {r2_score(y_test, model.predict(X_test)):.4f}")
Output:
TEXT
# Runs successfully
6. XGBoost vs. LightGBM Performance Comparison
▶ Example: Comparison on the SalesPredict Dataset
PYTHON
import xgboost as xgb
import lightgbm as lgb
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import mean_absolute_error, r2_score
import time
import numpy as np
rng = np.random.default_rng(42)
n = 100000
X = rng.uniform(0, 100, (n, 20))
y = 50 + X[:, :5] @ [0.8, 1.2, -0.5, 0.3, 0.1] + rng.normal(0, 5, n)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
results = {}
for name, model in [
("RandomForest", RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)),
("XGBoost", xgb.XGBRegressor(n_estimators=300, max_depth=6, learning_rate=0.1, random_state=42)),
("LightGBM", lgb.LGBMRegressor(n_estimators=300, num_leaves=31, learning_rate=0.1, random_state=42, verbose=-1)),
]:
start = time.time()
model.fit(X_train, y_train)
train_time = time.time() - start
y_pred = model.predict(X_test)
r2 = r2_score(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
results[name] = {"time": train_time, "r2": r2, "mae": mae}
print(f"{name:15s}: R²={r2:.4f}, MAE={mae:.2f}, Time={train_time:.1f}s")
Output:
TEXT
# Runs successfully
| Dimension | RandomForest | XGBoost | LightGBM |
|---|---|---|---|
| Training speed (100k samples) | 45s | 8s | 3s |
| Prediction accuracy (R²) | 0.85 | 0.89 | 0.89 |
| Memory usage | High | Medium | Low |
| Categorical features | Needs encoding | Needs encoding | Native support |
| GPU support | No | Yes | Yes |
| Suitable data size | <500k | Any | Any |
📌 Key takeaway: LightGBM leads XGBoost in training speed by 2-3x with comparable accuracy. However, XGBoost can be more stable on small datasets. In real projects, try both and pick the one that suits your data better.
❓ FAQ
Q Should I choose XGBoost or LightGBM?
A Large data (>100k) → LightGBM is faster; categorical features → LightGBM supports them natively; small data (<10k) → XGBoost is more stable. In practice, try both and compare with CV.
Q What is early_stopping?
A It monitors a metric on the validation set and stops training if there's no improvement for N consecutive rounds. This prevents overfitting and saves time. A recommended setting is early_stopping_rounds=50.
Q How are num_leaves and max_depth related?
A max_depth=d allows up to 2^d leaves. num_leaves directly controls the leaf count and is more flexible than max_depth. LightGBM recommends using num_leaves instead of max_depth. Typical values are 31-127.
Q How do learning_rate and n_estimators work together?
A Smaller learning_rate → more n_estimators needed → more stable but slower. Recommended: lr=0.1 + n_est=500 + early_stopping, or lr=0.01 + n_est=5000 + early_stopping.
Q Does GBDT need standardization?
A No. GBDT is tree-based and unaffected by feature scale. But LightGBM's categorical features require specifying the categorical_feature parameter.
Q How do I handle overfitting?
A Increase regularization (reg_alpha/reg_lambda), lower max_depth/num_leaves, raise min_child_samples, lower learning_rate, and increase the randomness of subsample/colsample_bytree.
📖 Summary
- Boosting core: train weak learners sequentially, with each new learner correcting the previous one's mistakes
- XGBoost innovations: second-order gradient optimization + L1/L2 regularization + column sampling + approximate quantile splitting
- LightGBM innovations: GOSS (gradient sampling) + EFB (feature bundling) + histogram splitting + native categorical support
- Key hyperparameters: learning_rate + n_estimators (with early_stopping), max_depth/num_leaves, and regularization parameters
- LightGBM is typically 2-3x faster than XGBoost in training, with comparable accuracy
- Both are industrial-grade GBDT frameworks; in real projects, try both and pick the best fit for your data
📝 Exercises
- Basic (Difficulty ⭐): Use XGBRegressor to predict California Housing with n_estimators=200, max_depth=5, learning_rate=0.1, and print R² and MAE. Hint: refer to the example in Section 4.
- Intermediate (Difficulty ⭐⭐): Compare XGBoost and LightGBM on the same dataset, recording training time and R², and use early_stopping to avoid overfitting. Hint: set eval_set + early_stopping for both.
- Challenge (Difficulty ⭐⭐⭐): Use GridSearchCV or Optuna to tune LightGBM hyperparameters, searching for the best combination of num_leaves/learning_rate/reg_alpha/reg_lambda to improve R² by at least 0.02. Hint: use 5-fold CV and refer to the parameter table in Section 5 for the search space.
← Previous: Feature Engineering | Next: Dimensionality Reduction — PCA and t-SNE →
