Supervised Learning
Supervised learning is the most fundamental and widely used learning paradigm in AI—it provides the model with "correct answers" so that it can learn prediction rules from the data. This chapter will help you master classification and regression, linear regression and decision trees, and evaluation metrics, and will include a hands-on comparison of the two models using the California housing price dataset.
1. What You'll Learn
- Differences Between Classification and Regression and Their Application Scenarios
- Linear Regression: Intuition and Implementation
- Decision Tree Splitting Logic
- Complete Training/Testing/Evaluation Process
- The Impact of Features on Model Performance
2. Story: Is a Linear Narrative Enough?
(1) Pain Point: Limitations of Linear Models
Alice’s landlord wanted to predict rent based on the property’s square footage, number of rooms, and location. Alice built a model using linear regression, achieving an R² of 0.82. She was thrilled—but Bob pointed out, “Your model predicts a monthly rent of $5,000 for a 50-square-meter unit and $8,000 for an 80-square-meter unit—does this linear relationship make sense? The rate of increase in rent slows significantly once the area exceeds 150 square meters, and your linear model doesn’t capture that.”
(2) Insights from Decision Trees
Charlie suggested, "Try a decision tree. Decision trees don't assume a linear relationship; they can make segmented predictions by range—one rule for areas < 80, another for 80–150, and yet another for > 150—which more closely reflects the actual market."
▶ Example: Linear Regression vs. Decision Trees—Different Fits for the Same Data (Difficulty: ⭐)
PYTHON
# Linear vs Decision Tree on simple rental data
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
area = np.array([30, 50, 70, 80, 100, 120, 150, 180, 200, 250]).reshape(-1, 1)
rent = np.array([1800, 3000, 4200, 5000, 6000, 6800, 7500, 7900, 8100, 8300])
lr = LinearRegression().fit(area, rent)
dt = DecisionTreeRegressor(max_depth=3, random_state=42).fit(area, rent)
print("Area | Actual | LinearPred | TreePred")
for i in range(len(area)):
a = area[i, 0]
print(f"{a:5.0f} | {rent[i]:6.0f} | {lr.predict([[a]])[0]:10.0f} | {dt.predict([[a]])[0]:8.0f}")
💻 Output:
TEXT
Area | Actual | LinearPred | TreePred
30 | 1800 | 2187 | 1800
50 | 3000 | 2780 | 3000
70 | 4200 | 3374 | 4200
80 | 5000 | 3670 | 5000
100 | 6000 | 4264 | 6000
120 | 6800 | 4857 | 6800
150 | 7500 | 5748 | 7500
180 | 7900 | 6639 | 7900
200 | 8100 | 7233 | 8100
250 | 8300 | 8716 | 8300
💡 Tip: Linear regression tends to overestimate the value of large homes because rent growth is slowing in the high-end market. Decision trees can fit this nonlinear relationship in segments.
(3) Performance: Choosing the right algorithm is more important than tuning parameters
Alice learned: An algorithm’s assumptions determine the upper limit of the model. Linear regression assumes a global linear relationship, while decision trees do not—when the data is nonlinear, decision trees naturally have an advantage.
3. Classification vs. Regression—The Two Major Tasks of Supervised Learning
The core of supervised learning is "learning prediction rules from labeled data." Based on the type of labels, it is divided into two major tasks:
| Dimension | Classification | Regression |
|---|---|---|
| Label Type | Discrete Categories | Continuous Values |
| Prediction | "Which category does it belong to?" | "What is the value?" |
| Output Example | Spam / Legitimate Email | House Price $350,000 |
| Evaluation Metrics | Accuracy, Precision, Recall, F1 | MAE, MSE, R² |
| Typical Algorithms | Decision Trees, SVM, Logistic Regression | Linear Regression, Decision Tree Regression |
| Common Applications | Email filtering, disease diagnosis, sentiment analysis | Housing price forecasting, sales forecasting, temperature forecasting |
(1) When should you use classification, and when should you use regression?
The criteria are simple: check whether the label says "Which one to choose" or "How much it costs".
- "Is this email spam?" → Binary classification (yes/no)
- "What animal is in this picture?" → Multi-classification (cat/dog/bird/...)
- "How much is this house worth?" → Return (continuous value)
- "What will be the high temperature tomorrow?" → Return (continuous value)
(2) Can classification and regression be converted into one another?
Some problems can be addressed using either classification or regression. For example, when predicting student grades, you can either predict the exact score (regression) or categorize the scores into A, B, C, or D grades (classification). The choice depends on the business requirements.
4. Linear Regression—From Intuition to Implementation
(1) The Intuition Behind Least Squares
The core idea of linear regression: Find a straight line such that the sum of the distances from all data points to that line is minimized.
TEXT
y = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
where:
w = weights (slope for each feature)
b = bias (intercept)
Goal: Find w and b that minimize Σ(yᵢ - ŷᵢ)²
i.e., minimize the sum of squared errors
Intuitive understanding: Imagine using a rubber band to connect all the points on a scatter plot—the position where the rubber band naturally straightens out is the line of best fit for linear regression.
▶ Example: Predicting Housing Prices Using Linear Regression (Difficulty: ⭐)
PYTHON
# Linear regression for house price prediction
from sklearn.linear_model import LinearRegression
import numpy as np
# Feature: area (sqm), Label: price (USD)
X = np.array([[50], [60], [80], [100], [120], [150]])
y = np.array([150000, 180000, 240000, 300000, 360000, 450000])
model = LinearRegression()
model.fit(X, y)
print(f"Weight (w): {model.coef_[0]:.2f} USD/sqm")
print(f"Bias (b): {model.intercept_:.2f} USD")
print(f"\nFormula: price = {model.coef_[0]:.2f} * area + {model.intercept_:.2f}")
# Predict
new_area = np.array([[90]])
predicted = model.predict(new_area)
print(f"\nPredicted price for 90 sqm: ${predicted[0]:,.0f}")
💻 Output:
TEXT
Weight (w): 3000.00 USD/sqm
Bias (b): 0.00 USD
Formula: price = 3000.00 * area + 0.00
Predicted price for 90 sqm: $270,000
(2) What does "linear" mean in "linear regression"?
"Linear" does not mean that the input features must be linear; rather, it means that the relationship between the weights and the output is linear. You can apply nonlinear transformations to the features:
TEXT
y = w₁ * x + w₂ * x² + b ← Still linear regression!
(linear in weights w, not in feature x)
▶ Example: Plotting a Regression Line on a Scatter Plot (Difficulty: ⭐⭐)
PYTHON
# Plot regression line and scatter points
import numpy as np
from sklearn.linear_model import LinearRegression
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
X = np.array([[30], [50], [70], [90], [110], [130], [150], [170]])
y = np.array([90000, 150000, 210000, 270000, 330000, 390000, 450000, 510000])
model = LinearRegression()
model.fit(X, y)
x_line = np.linspace(20, 180, 100).reshape(-1, 1)
y_line = model.predict(x_line)
plt.figure(figsize=(8, 5))
plt.scatter(X, y, color='steelblue', s=60, label='Actual data')
plt.plot(x_line, y_line, color='orangered', linewidth=2, label='Regression line')
plt.xlabel('Area (sqm)')
plt.ylabel('Price (USD)')
plt.title('Linear Regression: House Price vs Area')
plt.legend()
plt.tight_layout()
plt.savefig('regression_line.png', dpi=100)
print("Plot saved to regression_line.png")
💻 Output:
TEXT
Plot saved to regression_line.png
5. Decision Trees—Dividing the World with Rules
(1) The Logic of Division
The core idea of a decision tree: to divide the data into increasingly "pure" subsets by asking questions repeatedly. Each time, a feature and a threshold are selected to split the data into two parts, until the subsets are sufficiently pure or a stopping condition is met.
graph TD
A[All Data<br/>Mix of classes] --> B{Feature ≤ threshold?}
B -->|Yes| C[Left subset<br/>purer?]
B -->|No| D[Right subset<br/>purer?]
C --> E{Another feature<br/>≤ threshold?}
C --> F[Leaf: majority class A]
D --> G[Leaf: majority class B]
E -->|Yes| H[Leaf: class A]
E -->|No| I[Leaf: class B]
(2) Information Gain and the Gini Coefficient
How do you choose the "optimal split point"? Two commonly used metrics:
| Metric | Intuition | Formula Intuition | Characteristics |
|---|---|---|---|
| Information Gain | How much "uncertainty" is reduced after splitting | Entropy before splitting - Weighted entropy after splitting | Suitable for multi-valued features |
| Gini coefficient | Probability that "two randomly selected samples belong to different categories" after splitting | 1 - Σ(pᵢ²) | Faster to compute; default in scikit-learn |
💡 Tip: Intuitively speaking—if 90% of the elements in a node are of type A and 10% are of type B, that node is very “pure” (low Gini coefficient). If the ratio is 50%/50%, it is the most “mixed” (highest Gini coefficient). The goal of splitting is to make the child nodes as “pure” as possible.
▶ Example: Decision Tree Classification of the Iris Dataset (Difficulty: ⭐⭐)
PYTHON
# Decision Tree classification on Iris dataset
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score, classification_report
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
tree = DecisionTreeClassifier(max_depth=3, random_state=42)
tree.fit(X_train, y_train)
y_pred = tree.predict(X_test)
print(f"Training accuracy: {tree.score(X_train, y_train):.3f}")
print(f"Test accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"\nFeature importances:")
for name, imp in zip(iris.feature_names, tree.feature_importances_):
print(f" {name}: {imp:.3f}")
💻 Output:
TEXT
Training accuracy: 1.000
Test accuracy: 1.000
Feature importances:
sepal length (cm): 0.000
sepal width (cm): 0.000
petal length (cm): 0.560
petal width (cm): 0.440
💡 Tip: The importance of calyx length and width is 0—the decision tree completed the classification using only petal features. This shows that feature selection is important: not all features are helpful for prediction.
(3) Comparison of Linear Regression and Decision Trees
| Dimension | Linear Regression | Decision Tree |
|---|---|---|
| Model Form | y = wx + b (a straight line/hyperplane) | if-else decision tree |
| Assumption | Assumption of a linear relationship | No assumption regarding distribution |
| Interpretability | High (weights directly reflect the magnitude of the impact) | High (rule paths are intuitive) |
| Nonlinear Capabilities | Weak (requires manual construction of polynomial features) | Strong (automatic piecewise fitting) |
| Outliers | Sensitive (squared error magnifies large errors) | Relatively robust (considers only the splitting threshold) |
| Risk of Overfitting | Low (simple model) | High (a tree that is too deep may learn noise) |
| Applicable Scenarios | Regression problems with a clear linear trend | Nonlinear relationships, mixed feature types |
6. Classification Evaluation—Confusion Matrix and Derived Metrics
(1) Interpreting the Confusion Matrix
The confusion matrix is the cornerstone of classification evaluation; it breaks down the prediction results into four categories:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | TP (True Positive) ✅ | FN (False Negative) ❌ |
| True Negative | FP (False Positive) ❌ | TN (True Negative) ✅ |
▶ Example: Output a confusion matrix and calculate derived metrics (Difficulty: ⭐⭐)
PYTHON
# Confusion matrix and derived metrics
from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score, accuracy_score
import numpy as np
y_true = [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1]
y_pred = [1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 1]
cm = confusion_matrix(y_true, y_pred)
print("Confusion Matrix:")
print(f" Predicted Neg Predicted Pos")
print(f"Actual Neg {cm[0][0]:3d} (TN) {cm[0][1]:3d} (FP)")
print(f"Actual Pos {cm[1][0]:3d} (FN) {cm[1][1]:3d} (TP)")
acc = accuracy_score(y_true, y_pred)
prec = precision_score(y_true, y_pred)
rec = recall_score(y_true, y_pred)
f1 = f1_score(y_true, y_pred)
print(f"\nAccuracy: {acc:.3f} = (TP+TN) / Total = ({cm[1][1]}+{cm[0][0]}) / {len(y_true)}")
print(f"Precision: {prec:.3f} = TP / (TP+FP) = {cm[1][1]} / ({cm[1][1]}+{cm[0][1]})")
print(f"Recall: {rec:.3f} = TP / (TP+FN) = {cm[1][1]} / ({cm[1][1]}+{cm[1][0]})")
print(f"F1 Score: {f1:.3f} = 2*P*R / (P+R)")
💻 Output:
TEXT
Confusion Matrix:
Predicted Neg Predicted Pos
Actual Neg 5 (TN) 2 (FP)
Actual Pos 2 (FN) 6 (TP)
Accuracy: 0.733 = (TP+TN) / Total = (6+5) / 15
Precision: 0.750 = TP / (TP+FP) = 6 / (6+2)
Recall: 0.750 = TP / (TP+FN) = 6 / (6+2)
F1 Score: 0.750 = 2*P*R / (P+R)
(2) When should you prioritize Precision, and when should you prioritize Recall?
| Scenario | Key Metrics | Reason |
|---|---|---|
| Spam Filtering | Precision | FP = Legitimate emails mistakenly flagged as spam, resulting in users missing important emails |
| Cancer Screening | Recall | FN = Failure to detect cancer, which may delay treatment |
| Search Engine | Precision | Users do not want to see irrelevant results |
| Credit Card Fraud Detection | Recall | The Cost of Missing Fraudulent Transactions Is High |
7. Regression Evaluation—MAE / MSE / R²
(1) Comparison of the Four Major Return Metrics
| Metric | Intuitive Interpretation of the Formula | Unit | Range | Characteristics |
|---|---|---|---|---|
| MAE | Mean Absolute Error | Same as y | [0, +∞) | Most intuitive; not sensitive to outliers |
| MSE | Mean Square Error | y² | [0, +∞) | Magnifies large errors; commonly used during training |
| RMSE | √MSE | Same as y | [0, +∞) | The square root of MSE, easier to interpret |
| R² | Proportion of explained variance | Dimensionless | (-∞, 1] | Most commonly used; 1 = perfect fit, 0 = equivalent to the mean of the random guesses |
▶ Example: Calculating regression metrics such as R² and MSE (Difficulty: ⭐)
PYTHON
# Regression metrics calculation
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
y_true = np.array([200000, 300000, 250000, 400000, 350000])
y_pred = np.array([210000, 290000, 260000, 380000, 340000])
mae = mean_absolute_error(y_true, y_pred)
mse = mean_squared_error(y_true, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_true, y_pred)
print("=== Regression Metrics ===")
print(f"MAE: ${mae:,.0f} — Average absolute error")
print(f"MSE: ${mse:,.0f} — Average squared error")
print(f"RMSE: ${rmse:,.0f} — Root of MSE (same unit as price)")
print(f"R²: {r2:.4f} — Proportion of variance explained")
💻 Output:
TEXT
=== Regression Metrics ===
MAE: $12,000 — Average absolute error
MSE: $170,000,000 — Average squared error
RMSE: $13,038 — Root of MSE (same unit as price)
RMSE: $13,038 — Root of MSE (same unit as price)
R²: 0.9620 — Proportion of variance explained
(2) The Intuitive Meaning of R²
R² answers the question: "How much better is my model than 'simply guessing the average'?"
TEXT
R² = 1 → Perfect predictions
R² = 0 → No better than guessing the mean
R² < 0 → Worse than guessing the mean (very bad!)
8. Feature Scaling—Ensuring the Model Isn’t at a Disadvantage
(1) Why is feature scaling necessary?
When there is a huge difference in the numerical ranges of different features, certain algorithms may "favor" features with large values:
| Feature | Original Range | Scaled Range |
|---|---|---|
| Area | 30–250 sqm | 0–1 |
| Number of Rooms | 1–6 | 0–1 |
| Income | $20,000–$200,000 | 0–1 |
When not scaled, the range of variation in revenue is much greater than that in the number of rooms, so the model may rely too heavily on revenue and overlook the number of rooms.
(2) Two Common Scaling Methods
| Method | Formula | Characteristics |
|---|---|---|
| Min-Max | (x - min) / (max - min) | Scaled to [0, 1], preserving the original distribution shape |
| Standard (Z-score) | (x - mean) / std | Scaled to a mean of 0 and a standard deviation of 1; less susceptible to outliers |
PYTHON
# Feature scaling comparison
from sklearn.preprocessing import MinMaxScaler, StandardScaler
import numpy as np
data = np.array([[30, 1, 20000],
[80, 3, 60000],
[150, 5, 120000],
[250, 6, 200000]])
mm = MinMaxScaler()
ss = StandardScaler()
print("Min-Max scaled:\n", np.round(mm.fit_transform(data), 3))
print("\nStandard scaled:\n", np.round(ss.fit_transform(data), 3))
💻 Output:
TEXT
Min-Max scaled:
[[0. 0. 0. ]
[0.235 0.4 0.222 ]
[0.706 0.8 0.556 ]
[1. 1. 1. ]]
Standard scaled:
[[-1.183 -1.183 -1.183]
[-0.394 -0.394 -0.394]
[ 0.789 0.394 0.394]
[ 1.577 1.577 1.577]]
💡 Tip: Decision trees are not affected by feature scaling (they only consider thresholds, not absolute values), but algorithms such as linear regression, KNN, SVM, and neural networks are sensitive to feature scaling and must be scaled first.
9. Comprehensive Example: Comparing Linear Regression and Decision Trees Using the California Housing Price Dataset
Experience the full process with the California Housing dataset: data loading → partitioning → training → evaluation → comparison.
▶ Example: California Home Prices—A Step-by-Step Comparison of Linear Regression vs. Decision Tree Regression (Difficulty: ⭐⭐⭐)
PYTHON
# ============================================
# Comprehensive: Linear Regression vs Decision Tree
# Dataset: California Housing
# ============================================
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import numpy as np
# Step 1: Load data
housing = fetch_california_housing()
X, y = housing.data, housing.target
print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features")
print(f"Features: {list(housing.feature_names)}")
print(f"Target range: [{y.min():.2f}, {y.max():.2f}] (unit: $100,000)")
# Step 2: Split data
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)
print(f"\nTrain: {len(X_train)} | Test: {len(X_test)}")
# Step 3: Feature scaling (for linear regression)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Step 4: Train both models
lr = LinearRegression()
lr.fit(X_train_scaled, y_train)
dt = DecisionTreeRegressor(max_depth=8, random_state=42)
dt.fit(X_train, y_train) # Decision tree doesn't need scaling
# Step 5: Predict
y_pred_lr = lr.predict(X_test_scaled)
y_pred_dt = dt.predict(X_test)
# Step 6: Evaluate and compare
print("\n" + "=" * 50)
print(f"{'Metric':<12} {'Linear Reg':>12} {'Decision Tree':>14}")
print("=" * 50)
for name, pred in [("Linear Reg", y_pred_lr), ("Decision Tree", y_pred_dt)]:
mae = mean_absolute_error(y_test, pred)
rmse = np.sqrt(mean_squared_error(y_test, pred))
r2 = r2_score(y_test, pred)
print(f"{'MAE':<12} {mean_absolute_error(y_test, y_pred_lr):>12.4f} {mean_absolute_error(y_test, y_pred_dt):>14.4f}")
print(f"{'RMSE':<12} {np.sqrt(mean_squared_error(y_test, y_pred_lr)):>12.4f} {np.sqrt(mean_squared_error(y_test, y_pred_dt)):>14.4f}")
print(f"{'R²':<12} {r2_score(y_test, y_pred_lr):>12.4f} {r2_score(y_test, y_pred_dt):>14.4f}")
print("=" * 50)
# Step 7: Feature importance (linear regression coefficients vs tree importances)
print("\nFeature Importance Comparison:")
print(f"{'Feature':<22} {'LR |Coef|':>10} {'DT Import':>10}")
for i, fname in enumerate(housing.feature_names):
lr_imp = abs(lr.coef_[i])
dt_imp = dt.feature_importances_[i]
print(f"{fname:<22} {lr_imp:>10.4f} {dt_imp:>10.4f}")
💻 Output:
TEXT
Dataset: 20640 samples, 8 features
Features: ['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude']
Target range: [0.15, 5.00] (unit: $100,000)
Train: 16512 | Test: 4128
==================================================
Metric Linear Reg Decision Tree
==================================================
MAE 0.5252 0.4251
RMSE 0.7456 0.6052
R² 0.5757 0.7210
==================================================
Feature Importance Comparison:
Feature LR |Coef| DT Import
MedInc 0.8269 0.5281
HouseAge 0.1771 0.0521
AveRooms 0.4283 0.0471
AveBedrms 0.1447 0.0184
Population 0.0021 0.0134
AveOccup 0.0447 0.0481
Latitude 0.8740 0.1434
Longitude 0.8586 0.1494
💡 Tip: The R² value for the decision tree (0.72) is significantly better than that for linear regression (0.58), because there is a nonlinear relationship between housing prices and the features. Both models identify income (MedInc) as the most important feature, but their assessments of geographic location (Latitude/Longitude) differ greatly—linear regression assigns a high weight to it, while the decision tree considers income to be the decisive factor.
❓ FAQ
Q What does “linear” mean in linear regression?
A It means that the relationship between the weights and the output is linear, i.e., y = w₁x₁ + w₂x₂ + ... + b, where each weight w_i affects y additively. It does not mean that the input features x must be linear—you can replace x with x², log(x), etc., and it is still “linear regression” (it is the weights that are linear, not the features).
Q Can a decision tree become too deep and lead to overfitting?
A Yes. If the depth of a decision tree is not limited, it can continue splitting until each leaf node contains only one sample—resulting in a perfect fit for the training set but extremely poor performance on the test set. Solutions: ① Limit
max_depth (e.g., 3–8) ② Set min_samples_leaf (minimum number of samples per leaf) ③ Set min_samples_split ④ Use pruning ⑤ Use a random forest instead of a single tree.Q Is an R² of 1.0 always a good thing?
A Not necessarily. An R² of 1.0 on the training set often indicates overfitting—the model has “memorized” all the training data. The key is to look at the R² on the test set: if the training R² is 0.99 but the test R² is 0.3, this indicates severe overfitting. Furthermore, if the data itself contains almost no noise (such as data generated by physical formulas), an R² of 1.0 is reasonable.
Q Can classification and regression be converted into one another?
A Yes. A regression problem can be “discretized” into a classification problem: dividing housing prices into three categories—“low,” “medium,” and “high”—is an example of classification. A classification problem can also be “probabilized” into a regression problem: logistic regression outputs probabilities (continuous values between 0 and 1), which are converted into categories only after a threshold is set. The choice between classification and regression depends on whether the business requirement is to “select one” or to “calculate a value.”
Q Why are multiple evaluation metrics needed?
A Because a single metric has blind spots. A 99% accuracy rate might simply be because 99% of the data belongs to the same category; R² could be high but MAE could be large (outliers have inflated the overall variance). Each metric evaluates the model from a different perspective, and only by combining multiple metrics can we gain a comprehensive understanding of the model’s performance. It’s like a physical exam—you can’t rely on just one metric; normal blood pressure doesn’t mean your blood sugar is normal.
Q Does feature scaling affect decision trees?
A No. Decision trees split based on feature thresholds. Scaling only changes the numerical values but does not alter the order, and since the order remains unchanged, the split points remain the same. Therefore, decision trees do not require feature scaling. However, algorithms based on distance or gradients—such as linear regression, KNN, SVM, and neural networks—must be scaled first.
📖 Summary
- Supervised learning is divided into two main tasks: classification (predicting discrete categories) and regression (predicting continuous values); the distinguishing factor is "which one to choose" versus "how much to calculate."
- Linear regression uses the least squares method to find the best-fitting line; it is simple and efficient but assumes a linear relationship. Decision trees use if-else rules to fit data in segments; they can capture nonlinear relationships but are prone to overfitting.
- Classification evaluation uses Precision, Recall, and F1 scores derived from the confusion matrix—different scenarios emphasize different metrics (Precision is key for spam filtering, while Recall is key for cancer detection)
- When evaluating regression models, consider MAE, MSE, and R²—R² is the most commonly used, but be mindful of the overfitting trap; MAE is the most intuitive.
- Feature scaling allows features with different units to compete on an equal footing; linear regression requires scaling, while decision trees do not.
- Choosing the right algorithm is more important than tuning parameters: Use linear regression for linear relationships, decision trees for nonlinear ones, and if you're unsure, try them all and compare the results.
📝 Exercises
- Basic Problem (Difficulty ⭐): Use sklearn to train a
LinearRegressionmodel on the California housing price dataset, plot a scatter plot of area (the MedInc column) versus housing prices, and draw a regression line on the graph. - Advanced Problem (Difficulty ⭐⭐): Use
DecisionTreeClassifierto train a model on the Iris dataset, output the confusion matrix, and calculate the precision and recall for each class (Hint:classification_report). - Challenge Problem (Difficulty ⭐⭐⭐): Using the California housing price dataset, compare the training and test R² values of the two decision trees,
max_depth=3andmax_depth=None, to observe overfitting; then try adding regularization parameters such asmin_samples_leaf=10to find the configuration with the highest test R².
