Model Evaluation & Tuning
If your evaluation methodology is wrong, all optimization is built on sand — measure accurately first, then tune.
1. What You'll Learn
- Evaluation methodology: Hold-out / K-Fold / Stratified K-Fold / TimeSeriesSplit
- Classification evaluation: confusion matrix, Precision-Recall curves, F1-β, multi-class metrics
- Regression evaluation: MAE / MSE / RMSE / MAPE / R², aligning metrics with business goals
- Hyperparameter tuning: GridSearchCV / RandomizedSearchCV / Optuna Bayesian optimization
- Overfitting diagnosis: learning curves, validation curves
2. A Real Story from an ML Engineer
(1) The Problem: 95% Accuracy on the Test Set, 75% After Deployment
Bob evaluated his model with train_test_split and got a test accuracy of 95%. But one week after deployment, real-world accuracy was only 75% — because with time series data, the training and test distributions overlapped, and the model had "peeked" at future information. A flawed evaluation method led to inflated metrics and a disastrous launch.
(2) The Fix: Proper Evaluation Methodology
Time series data must use TimeSeriesSplit — the training set contains only past data, and the test set is future data, simulating real-world conditions.
PYTHON
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, test_idx in tscv.split(X):
# train_idx always < test_idx (no future leakage)
model.fit(X[train_idx], y[train_idx])
score = model.score(X[test_idx], y[test_idx])
(3) The Payoff: Metrics That Reflect True Performance
Bob re-evaluated with TimeSeriesSplit, and accuracy dropped from an "inflated" 95% to a "realistic" 82%. The number went down, but post-deployment accuracy was also 82% — evaluation and reality finally matched, with no more nasty surprises.
3. Evaluation Methodology
(1) Four Cross-Validation Strategies
graph TB
DATA[Dataset] --> HO[Hold-out<br/>Single Train/Test Split]
DATA --> KF[K-Fold CV<br/>Random Splits]
DATA --> SKF[Stratified K-Fold<br/>Class-Balanced Splits]
DATA --> TSS[TimeSeriesSplit<br/>Chronological Splits]
HO --> HO_USE[Quick baseline<br/>Large dataset]
KF --> KF_USE[General purpose<br/>Stable estimate]
SKF --> SKF_USE[Classification<br/>Imbalanced data]
TSS --> TSS_USE[Time series<br/>No future leakage]
▶ Example: Comparing Four CV Strategies
PYTHON
from sklearn.model_selection import (train_test_split, KFold, StratifiedKFold,
TimeSeriesSplit, cross_val_score)
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_iris
import numpy as np
X, y = load_iris(return_X_y=True)
model = RandomForestClassifier(n_estimators=50, random_state=42)
# 1. Hold-out
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
model.fit(X_tr, y_tr)
holdout_score = model.score(X_te, y_te)
# 2. K-Fold
kf_scores = cross_val_score(model, X, y, cv=KFold(n_splits=5, shuffle=True, random_state=42))
# 3. Stratified K-Fold
skf_scores = cross_val_score(model, X, y, cv=StratifiedKFold(n_splits=5, shuffle=True, random_state=42))
# 4. TimeSeriesSplit (for time-ordered data)
tss_scores = cross_val_score(model, X, y, cv=TimeSeriesSplit(n_splits=5))
print(f"Hold-out: {holdout_score:.4f}")
print(f"K-Fold: {kf_scores.mean():.4f} +/- {kf_scores.std():.4f}")
print(f"Stratified K-Fold: {skf_scores.mean():.4f} +/- {skf_scores.std():.4f}")
print(f"TimeSeriesSplit: {tss_scores.mean():.4f} +/- {tss_scores.std():.4f}")
Output:
TEXT
# Execution successful
| Method | Data Type | Pros | Cons |
|---|---|---|---|
| Hold-out | Any | Fast | Result depends on the split |
| K-Fold | Any | Stable | Random partitioning |
| Stratified K-Fold | Classification | Preserves class proportions | Classification only |
| TimeSeriesSplit | Time series | No future leakage | Growing training size |
4. Classification and Regression Metrics
(1) Full Classification Metrics
▶ Example: Comprehensive Classification Evaluation
PYTHON
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, roc_auc_score, classification_report,
confusion_matrix)
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
pipe = Pipeline([("scaler", StandardScaler()), ("model", LogisticRegression(max_iter=500))])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
y_prob = pipe.predict_proba(X_test)[:, 1]
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall: {recall_score(y_test, y_pred):.4f}")
print(f"F1: {f1_score(y_test, y_pred):.4f}")
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, y_pred)}")
print(f"\nDetailed Report:\n{classification_report(y_test, y_pred)}")
Output:
TEXT
# Execution successful
(2) Regression Metrics Aligned with Business Goals
▶ Example: Regression Metrics and Their Business Meaning
PYTHON
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
y_true = np.array([100, 150, 200, 250, 300]) # thousand USD
y_pred = np.array([95, 160, 190, 260, 310])
mae = mean_absolute_error(y_true, y_pred)
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
r2 = r2_score(y_true, y_pred)
mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
print(f"MAE: {mae:.2f} k USD")
print(f"RMSE: {rmse:.2f} k USD")
print(f"R²: {r2:.4f}")
print(f"MAPE: {mape:.1f}%")
# Business impact calculation
avg_monthly_revenue = 1000 # thousand USD
mape_pct = mape / 100
inventory_cost_pct = 0.3 # 30% of overstock is waste
annual_loss = avg_monthly_revenue * mape_pct * inventory_cost_pct * 12
print(f"\nAnnual inventory loss from prediction error: {annual_loss:.0f} k USD")
Output:
TEXT
# Execution successful
| Metric | Formula | Business Meaning |
|---|---|---|
| MAE | mean|y-ŷ| | Average deviation (k USD) |
| RMSE | √(mean(y-ŷ)²) | Penalizes large errors more |
| R² | 1 - SS_res/SS_tot | Proportion of variance explained |
| MAPE | mean|y-ŷ|/y × 100% | Relative error (%) |
5. Hyperparameter Tuning
(1) Three Search Strategies
▶ Example: GridSearch vs RandomSearch
PYTHON
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
import time
X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
param_grid = {
"n_estimators": [50, 100, 200],
"max_depth": [5, 10, 15, None],
"min_samples_leaf": [1, 5, 10],
}
# GridSearch: exhaustive search
start = time.time()
grid = GridSearchCV(RandomForestRegressor(random_state=42), param_grid, cv=3, scoring="r2", n_jobs=-1)
grid.fit(X_train, y_train)
grid_time = time.time() - start
# RandomSearch: random sampling
start = time.time()
random = RandomizedSearchCV(RandomForestRegressor(random_state=42), param_grid,
n_iter=15, cv=3, scoring="r2", n_jobs=-1, random_state=42)
random.fit(X_train, y_train)
random_time = time.time() - start
print(f"GridSearch: R²={grid.best_score_:.4f}, Time={grid_time:.1f}s, Trials={len(grid.cv_results_['mean_test_score'])}")
print(f"RandomSearch: R²={random.best_score_:.4f}, Time={random_time:.1f}s, Trials=15")
Output:
TEXT
# Execution successful
| Method | Search Strategy | Coverage | Speed | Best For |
|---|---|---|---|---|
| GridSearch | Exhaustive over all combinations | 100% | Slow | Few parameters |
| RandomSearch | Random sampling | Partial | Fast | Many parameters |
| Optuna | Bayesian optimization | Guided search | Efficient | Complex search spaces |
(2) Optuna Bayesian Optimization
▶ Example: Hyperparameter Tuning with Optuna
PYTHON
# pip install optuna
import optuna
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
def objective(trial):
n_estimators = trial.suggest_int("n_estimators", 50, 500)
max_depth = trial.suggest_int("max_depth", 3, 20)
min_samples_leaf = trial.suggest_int("min_samples_leaf", 1, 20)
max_features = trial.suggest_float("max_features", 0.3, 1.0)
model = RandomForestRegressor(
n_estimators=n_estimators, max_depth=max_depth,
min_samples_leaf=min_samples_leaf, max_features=max_features,
random_state=42, n_jobs=-1,
)
scores = cross_val_score(model, X_train, y_train, cv=3, scoring="r2")
return scores.mean()
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=30, show_progress_bar=False)
print(f"Best R²: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")
Output:
TEXT
# Function defined successfully
6. Overfitting Diagnosis
(1) Learning Curves and Validation Curves
▶ Example: Plotting Learning Curves
PYTHON
from sklearn.model_selection import learning_curve, validation_curve
from sklearn.ensemble import RandomForestRegressor
from sklearn.datasets import fetch_california_housing
import matplotlib.pyplot as plt
import numpy as np
X, y = fetch_california_housing(return_X_y=True)
# Learning curve: performance vs training set size
train_sizes, train_scores, val_scores = learning_curve(
RandomForestRegressor(n_estimators=50, random_state=42),
X, y, train_sizes=np.linspace(0.1, 1.0, 10),
cv=3, scoring="r2", n_jobs=-1,
)
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Learning curve
axes[0].plot(train_sizes, train_scores.mean(axis=1), "o-", label="Training")
axes[0].plot(train_sizes, val_scores.mean(axis=1), "o-", label="Validation")
axes[0].fill_between(train_sizes, train_scores.mean(axis=1) - train_scores.std(axis=1),
train_scores.mean(axis=1) + train_scores.std(axis=1), alpha=0.1)
axes[0].fill_between(train_sizes, val_scores.mean(axis=1) - val_scores.std(axis=1),
val_scores.mean(axis=1) + val_scores.std(axis=1), alpha=0.1)
axes[0].set_xlabel("Training Size")
axes[0].set_ylabel("R² Score")
axes[0].set_title("Learning Curve")
axes[0].legend()
axes[0].grid(True, alpha=0.3)
# Validation curve: performance vs hyperparameter
param_range = [3, 5, 7, 10, 15, 20, None]
train_scores2, val_scores2 = validation_curve(
RandomForestRegressor(n_estimators=50, random_state=42),
X, y, param_name="max_depth", param_range=param_range,
cv=3, scoring="r2", n_jobs=-1,
)
axes[1].plot(range(len(param_range)), train_scores2.mean(axis=1), "o-", label="Training")
axes[1].plot(range(len(param_range)), val_scores2.mean(axis=1), "o-", label="Validation")
axes[1].set_xticks(range(len(param_range)))
axes[1].set_xticklabels([str(p) for p in param_range])
axes[1].set_xlabel("max_depth")
axes[1].set_ylabel("R² Score")
axes[1].set_title("Validation Curve")
axes[1].legend()
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("learning_validation_curves.png", dpi=150)
Output:
TEXT
# Execution successful
| Diagnosis | Curve Pattern | Meaning | Solution |
|---|---|---|---|
| Underfitting | Both train and validation scores are low | Model too simple | Increase complexity / add features |
| Overfitting | High training, low validation | Model too complex | Regularize / simplify / add data |
| Good fit | Both high and close together | Well balanced | No changes needed |
| Needs more data | Validation curve still rising | Insufficient data | Collect more data |
❓ FAQ
Q How do I choose the number of folds for cross-validation?
A 5 or 10 are the most common choices. 5-fold offers a good balance between speed and stability; 10-fold is more stable but twice as slow. For large datasets (>50k samples), 3 folds is sufficient; for small datasets (<1k samples), use 10 folds.
Q Why can't I use K-Fold for time series data?
A K-Fold shuffles data randomly, which can result in training on "future" data to predict the "past," causing data leakage. TimeSeriesSplit guarantees that the training set always precedes the test set, simulating real forecasting scenarios.
Q What if GridSearchCV is too slow?
A Three strategies — 1) shrink the search space (coarse search first, then fine-grained); 2) use RandomizedSearchCV (n_iter=20); 3) use Optuna Bayesian optimization (intelligent search).
Q What are the drawbacks of MAPE?
A MAPE explodes when y_true is close to zero (division by a near-zero value). Alternatives — use SMAPE (symmetric MAPE) or MASE (mean absolute scaled error).
Q What if the gap between training and validation scores on the learning curve is large?
A A large gap = overfitting. Solutions — 1) increase regularization; 2) reduce model complexity; 3) add more training data; 4) use Dropout / early stopping.
Q How much better is Optuna than GridSearch?
A Optuna uses Bayesian optimization to intelligently select the next set of parameters. It typically finds in 30–50 trials what GridSearch needs 500+ trials to find, saving over 80% of compute resources.
📖 Summary
- Evaluation methodology is the foundation of any ML project: use StratifiedKFold for classification, TimeSeriesSplit for time series
- Classification metric choice depends on the business: churn → Recall, spam detection → Precision, recommendations → F1
- Regression metrics must align with business costs: a 10% MAPE on million-dollar forecasts means $100k in error
- Hyperparameter tuning progression: GridSearch (exhaustive) → RandomSearch (sampling) → Optuna (intelligent)
- Overfitting diagnosis: learning curves reveal whether you have enough data; validation curves reveal whether parameters are optimal
- Correct evaluation matters more than high scores — inflated metrics collapse after deployment
📝 Exercises
- Basic (Difficulty ⭐): Compare the mean accuracy and standard deviation of RandomForestClassifier on Iris using 5-fold and 10-fold cross-validation. Hint: cross_val_score(cv=5/10).
- Intermediate (Difficulty ⭐⭐): Use GridSearchCV to find the optimal parameters for XGBoost (max_depth / learning_rate / n_estimators), and output the best parameters and CV R². Hint: Pipeline + param_grid.
- Challenge (Difficulty ⭐⭐⭐): Plot a learning curve and a validation curve to diagnose underfitting/overfitting in a model, propose a concrete improvement plan, and verify the improvement. Hint: learning_curve + validation_curve + before/after comparison.
← Previous: Convolutional Neural Networks | Next: Introduction to MLOps →
