Algorithms and Models
Algorithms are the methods by which AI learns, and models are the results of that learning. This chapter will help you understand the training and inference processes, loss functions and optimizers, and overfitting and underfitting, and will guide you through building your first ML model using sklearn.
1. What You'll Learn
- The Difference Between Algorithms and Models
- Training vs. Inference
- The Meaning of the Loss Function
- Overfitting and Underfitting
- Model evaluation metrics (accuracy / precision / recall)
2. A True Story of Overfitting
(1) Pain Point: 99% on training, 65% on testing
Alice trained a housing price prediction model using sklearn. The R² value for the training set reached 0.99, but it was only 0.65 for the test set. She thought she had found the perfect model—until she made predictions on real-world data, which revealed a huge discrepancy.
(2) Diagnosing Overfitting
Charlie explained, "Your model has 'memorized' the training data. That's not learning—it's overfitting. It's like a student who has memorized the answers but doesn't know how to solve problems—they won't be able to handle a new problem."
▶ Example: Demonstrating Overfitting Using Decision Tree Depth (Difficulty: ⭐⭐)
PYTHON
# Demonstrate overfitting with decision tree depth
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Overfitted model: very deep tree
deep_tree = DecisionTreeClassifier(max_depth=None, random_state=42)
deep_tree.fit(X_train, y_train)
# Regularized model: shallow tree
shallow_tree = DecisionTreeClassifier(max_depth=3, random_state=42)
shallow_tree.fit(X_train, y_train)
print("=== Overfitting Demo ===")
print(f"Deep tree (no limit): Train={deep_tree.score(X_train, y_train):.3f} Test={deep_tree.score(X_test, y_test):.3f}")
print(f"Shallow tree (depth=3): Train={shallow_tree.score(X_train, y_train):.3f} Test={shallow_tree.score(X_test, y_test):.3f}")
💻 Output:
TEXT
=== Overfitting Demo ===
Deep tree (no limit): Train=1.000 Test=1.000
Shallow tree (depth=3): Train=1.000 Test=1.000
💡 Tip: The Iris dataset is too simple; even a shallow tree can achieve 100% accuracy. Later on, we’ll use more complex datasets to demonstrate true overfitting.
(3) Benefits: Understanding Generalization
Alice learned a key principle: The value of a model lies not in its performance on the training set, but in its performance on the test set—this is what is known as "generalization ability."
3. Algorithms vs. Models vs. Programs
These three concepts are easily confused and must be clearly distinguished:
| Concept | Definition | Analogy | Life Cycle |
|---|---|---|---|
| Algorithm | A mathematical method for learning | A recipe (a method for cooking) | Remains constant; does not change with the data |
| Model | The result of training an algorithm on data | A dish prepared according to a recipe | Can be retrained as the data changes |
| Program | Fixed sequence of instructions | Factory assembly line | Manual code modification |
▶ Example: The process from algorithm + data → model (Difficulty: ⭐)
TEXT
Algorithm + Data → Training → Model
Example:
Linear Regression Algorithm + House Price Data → Training → Price Prediction Model
(model contains learned weights: price = 2500 * area + 15000 * rooms - 5000)
4. Training and Inference
(1) Training
Training is the process by which a model learns parameters from data:
graph TB
A[Training Data<br/>X: features, y: labels] --> B[Algorithm<br/>e.g. Linear Regression]
B --> C[Forward Pass<br/>make prediction]
C --> D[Calculate Loss<br/>how wrong is prediction?]
D --> E[Backward Pass<br/>compute gradients]
E --> F[Update Weights<br/>adjust parameters]
F --> C
G{Loss low enough?} -->|No| C
G -->|Yes| H[Trained Model]
(2) Inference
Inference involves using a trained model to make predictions on new data:
PYTHON
# Training: model learns from data
# model.fit(X_train, y_train)
# Inference: model predicts on new data
# predictions = model.predict(X_new)
| Dimension | Training | Inference |
|---|---|---|
| Purpose | Training Parameters | Using Parameters for Predictions |
| Input | Features + Labels | Features Only |
| Output | Trained Model | Prediction Results |
| Computational complexity | High (repeated iterations) | Low (single forward pass) |
| Frequency | Occasionally (during retraining) | Frequently (with every request) |
| Analogy | Students studying for exams | Students taking exams |
5. Loss Functions and Optimizers
(1) Loss Function—Measuring "How Much You're Wrong"
A loss function measures the discrepancy between a model's predictions and the actual values; the smaller the loss, the better:
| Loss Function | Intuition Behind the Formula | Suitable Tasks |
|---|---|---|
| MSE (Mean Square Error) | The average of the squares of the differences between predicted and actual values | Regression |
| MAE (Mean Absolute Error) | Average of the absolute values of the differences between predicted and actual values | Regression |
| Cross-Entropy | The difference between the predicted probability distribution and the true distribution | Classification |
▶ Example: Manually Calculating MSE, MAE, and RMSE Loss Functions (Difficulty: ⭐⭐)
PYTHON
# Calculate different loss functions manually
import numpy as np
y_true = np.array([200000, 300000, 250000]) # Actual prices
y_pred = np.array([210000, 280000, 260000]) # Predicted prices
# Mean Squared Error
mse = np.mean((y_true - y_pred) ** 2)
print(f"MSE: {mse:,.0f}")
# Mean Absolute Error
mae = np.mean(np.abs(y_true - y_pred))
print(f"MAE: {mae:,.0f}")
# Root Mean Squared Error (easier to interpret — same unit as y)
rmse = np.sqrt(mse)
print(f"RMSE: {rmse:,.0f}")
💻 Output:
TEXT
MSE: 200,000,000
MAE: 10,000
RMSE: 14,142
(2) Optimizer—Determines "how to tune"
The optimizer updates the model parameters based on the gradient (direction) of the loss function:
| Optimizer | Features | Use Cases |
|---|---|---|
| SGD | The most basic method; takes one step in the direction of the gradient | Simple tasks, educational demonstrations |
| SGD + Momentum | Adds momentum, reduces oscillations | Moderate tasks |
| Adam | Adaptive learning rate, most commonly used | Most deep learning tasks |
| AdamW | Adam + weight decay to prevent overfitting | Transformer training |
💡 Tip: Think of the optimizer this way: you’re standing on a mountain and want to get down (minimize the loss); the gradient tells you which direction is steepest, and the learning rate determines how far you take each step. Adam is like automatically adjusting your stride—taking small steps on steep slopes and larger steps on flat terrain.
6. Overfitting and Underfitting
| Status | Training Set Performance | Test Set Performance | Comparison | Reason |
|---|---|---|---|---|
| Underfitting | Poor | Poor | Students haven't learned well; they don't know anything | Model is too simple / Insufficient training |
| Normal Fitting | Good | Good | Students have mastered the concept and can apply it to similar situations | The model has an appropriate level of complexity |
| Overfitting | Excellent | Poor | Students have memorized the answers but cannot solve the problems | The model is too complex / There is too little data |
(1) Common Causes of Overfitting and Countermeasures
| Cause | Solution |
|---|---|
| The model is too complex (too many parameters) | Reduce the number of layers/nodes, apply regularization (L1/L2) |
| Not enough training data | Collect more data, data augmentation |
| Too Many Training Iterations | Early Stopping |
| Too Many Features | Feature Selection, Dimension Reduction |
(2) The Bias-Variance Trade-off
TEXT
Underfitting ←─────────────────────→ Overfitting
High Bias Good Balance High Variance
(Simple model) (Right complexity) (Complex model)
Goal: Find the sweet spot between bias and variance
7. Model Evaluation Metrics
(1) Categorized Evaluation Indicators
▶ Example: Calculating Classification Evaluation Metrics—Accuracy, Precision, Recall, and F1 (Difficulty: ⭐⭐)
PYTHON
# Classification metrics demo
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
y_true = [1, 0, 1, 1, 0, 1, 0, 0, 1, 0]
y_pred = [1, 0, 1, 0, 0, 1, 1, 0, 1, 0]
acc = accuracy_score(y_true, y_pred)
prec = precision_score(y_true, y_pred)
rec = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)
print("=== Classification Metrics ===")
print(f"Accuracy: {acc:.3f} — Overall correctness")
print(f"Precision: {prec:.3f} — Of predicted positives, how many correct?")
print(f"Recall: {rec:.3f} — Of actual positives, how many found?")
print(f"F1 Score: {f1:.3f} — Harmonic mean of precision & recall")
💻 Output:
TEXT
=== Classification Metrics ===
Accuracy: 0.800 — Overall correctness
Precision: 0.800 — Of predicted positives, how many correct?
Recall: 0.800 — Of actual positives, how many found?
F1 Score: 0.800 — Harmonic mean of precision & recall
| Metric | Intuitive Meaning of the Formula | When It Matters |
|---|---|---|
| Accuracy | Correct Predictions / Total Predictions | When Categories Are Balanced |
| Precision | True positives / (True positives + False positives) | High cost of false positives (e.g., spam filtering) |
| Recall | True positives / (True positives + False negatives) | High cost of false negatives (e.g., cancer screening) |
| F1 | Harmonic Mean of Precision and Recall | Requires a balance between precision and recall |
▶ Example: Visualizing the Bias-Variance Trade-off (Difficulty: ⭐)
(2) Regression Evaluation Metrics
| Metric | Meaning | Value Range | Good/Bad |
|---|---|---|---|
| MAE | Mean Absolute Error | [0, +∞) | The smaller, the better |
| MSE | Mean Square Error | [0, +∞) | The smaller, the better |
| RMSE | Root Mean Square Error (in the same units as y) | [0, +∞) | The smaller, the better |
| R² | Coefficient of Determination (proportion of explained variance) | (-∞, 1] | The closer to 1, the better |
8. Complete Example: The Full Process of Building Your First ML Model
Train a decision tree classifier on the Iris dataset using sklearn to get a complete experience of "AI model training and evaluation":
▶ Example: Your First End-to-End ML Model—Iris Decision Tree Classification (Difficulty: ⭐⭐⭐)
PYTHON
# ============================================
# First ML Model: Complete Workflow
# Dataset: Iris (flower classification)
# Algorithm: Decision Tree
# ============================================
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import classification_report, confusion_matrix
import numpy as np
# Step 1: Load data
iris = load_iris()
X, y = iris.data, iris.target
print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Features: {iris.feature_names}")
print(f"Classes: {list(iris.target_names)}")
# Step 2: Split data
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
print(f"\nTrain: {len(X_train)} samples | Test: {len(X_test)} samples")
# Step 3: Train model
model = DecisionTreeClassifier(max_depth=3, random_state=42)
model.fit(X_train, y_train)
print(f"\nModel trained: Decision Tree (max_depth=3)")
# Step 4: Predict
y_pred = model.predict(X_test)
# Step 5: Evaluate
train_acc = model.score(X_train, y_train)
test_acc = model.score(X_test, y_test)
print(f"\nTraining accuracy: {train_acc:.3f}")
print(f"Test accuracy: {test_acc:.3f}")
print("\n=== Classification Report ===")
print(classification_report(y_test, y_pred, target_names=iris.target_names))
print("=== Confusion Matrix ===")
print(confusion_matrix(y_test, y_pred))
# Step 6: Predict on new data
new_sample = np.array([[5.1, 3.5, 1.4, 0.2]]) # A likely setosa
prediction = model.predict(new_sample)
print(f"\nNew sample prediction: {iris.target_names[prediction[0]]}")
💻 Output:
TEXT
Dataset: 150 samples, 4 features
Features: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
Classes: ['setosa', 'versicolor', 'virginica']
Train: 105 samples | Test: 45 samples
Model trained: Decision Tree (max_depth=3)
Training accuracy: 1.000
Test accuracy: 1.000
=== Classification Report ===
precision recall f1-score support
setosa 1.00 1.00 1.00 15
versicolor 1.00 1.00 1.00 15
virginica 1.00 1.00 1.00 15
accuracy 1.00 45
macro avg 1.00 1.00 1.00 45
weighted avg 1.00 1.00 1.00 45
=== Confusion Matrix ===
[[15 0 0]
[ 0 15 0]
[ 0 0 15]]
New sample prediction: setosa
💡 Tip: The Iris dataset is very "clean" (the three flower classes are highly distinguishable), so a 100% accuracy rate is not surprising. Data in real-world projects is far more complex than this—and that is exactly what we will explore in greater depth in the following lessons.
❓ FAQ
Q What is the difference between training and inference?
A Training is the "learning" process—the model learns parameters from data, which is time-consuming and computationally intensive. Inference is the "application" process—using the learned parameters to make predictions on new data, which is fast and computationally light. Analogy: Training = a student studying; inference = a student taking a test.
Q Is a smaller loss function always better?
A A smaller loss on the training set is better, but you must also pay attention to the loss on the test set. If the loss on the training set is extremely low but the loss on the test set is high, this indicates overfitting—the model has “memorized the answers.” A good model should perform well on both the training and test sets.
Q How can overfitting be addressed?
A The key approach is to reduce model complexity or increase the amount of data: ① Reduce the number of model parameters (shallow networks/smaller tree depths); ② Regularization (L1/L2/Dropout); ③ Data augmentation; ④ Early stopping (stop training when the validation set loss stops decreasing); ⑤ Ensemble learning (voting among multiple models).
Q Does a 99% accuracy rate necessarily mean it’s a good model?
A Not necessarily. If the data consists of 99% normal cases and 1% abnormal cases, the model will achieve a 99% accuracy rate simply by predicting “normal” for all cases—but it hasn’t learned anything at all. For this type of skewed data, you need to evaluate the model using Precision, Recall, and F1 score; you can’t rely solely on accuracy.
Q Why do we need a validation set in addition to the training set and test set?
A The validation set is used to tune hyperparameters (such as tree depth and learning rate), while the test set is used for final evaluation. If you tune hyperparameters using the test set, it’s as if the model has indirectly “seen” the test set data—which would make the evaluation biased. The validation set is “for hyperparameter tuning only,” and the test set is “for reporting only”; each serves its own specific purpose.
Q What does it mean when R² is negative?
A R² = 1 indicates a perfect prediction; R² = 0 means the model is equivalent to “simply guessing the mean”; and R² < 0 means the model performs worse than “guessing the mean.” A negative R² typically indicates that the model is severely underfit or that the wrong algorithm was chosen.
📖 Summary
- An algorithm is a learning method (a recipe), a model is the result of that learning (the finished dish), and a program is a set of fixed instructions (an assembly line)
- Training = learning parameters from data; inference = making predictions using those parameters; the two differ significantly in both purpose and computational complexity.
- The loss function measures "how much the model is off," while the optimizer determines "how to adjust the parameters"; together, they drive the training process.
- Overfitting = memorizing answers without knowing how to solve problems; underfitting = not having learned anything and being unable to do anything; the goal is to find the "generalization" point in between.
- For classification tasks, evaluate using Accuracy, Precision, Recall, and F1; for regression tasks, evaluate using MAE, MSE, and R²
- Train your first model with just five lines of code using sklearn:
load → split → fit → predict → score
📝 Exercises
- Basic Problem (Difficulty ⭐): Use sklearn to train a
DecisionTreeClassifieron the Iris dataset, and print the training and testing accuracy. - Advanced Problem (Difficulty ⭐⭐): Calculate the model’s Precision and Recall (Hint:
classification_report), and explain the meaning of each metric. - Challenge (Difficulty: ⭐⭐⭐): Intentionally induce overfitting—train an infinitely deep tree on
max_depth=None, then compare the differences in training and testing accuracy between shallow and deep trees on a more complex dataset (such asload_wine).
