Introduction to Machine Learning
Machine Learning (ML) is the most fundamental subfield of AI—virtually all modern AI applications (recommendation systems, speech recognition, autonomous driving) are built on ML. This chapter will help you understand the essence of ML, its three major paradigms, and the types of tasks it addresses, and will guide you through implementing your first complete ML classifier using sklearn.
1. What You'll Learn
- The Fundamental Difference Between Machine Learning and Traditional Programming
- The three major paradigms: supervised learning, unsupervised learning, and reinforcement learning
- Classification vs. Regression vs. Clustering Task Types
- Overview of the ML Workflow (Problem Definition → Data → Features → Training → Evaluation → Deployment)
- Implement your first classifier using sklearn
2. Story: When There Are Too Many Rules to List
(1) Pain Point: The more rules are written, the lower the accuracy becomes
Bob's e-commerce company needs to predict whether users will purchase a particular product. He began writing rules:
TEXT
IF age > 30 AND browse_count > 3 THEN buy = yes
IF age < 25 AND cart_count > 0 THEN buy = maybe
IF region = "east" AND discount > 20% THEN buy = yes
... (300 more rules later) ...
We expanded the rules from 10 to 300, but the accuracy rate rose from 60% to 72% and then plateaued. Every time a new product category is introduced, we have to rewrite the rules all over again—it’s simply impossible to finish.
(2) Machine Learning Solutions
Alice suggested, "Let's not write any rules—let the algorithm find patterns on its own from 100K historical records."
▶ Example: Rule-Driven vs. Data-Driven Prediction of Purchasing Behavior (Difficulty: ⭐)
PYTHON
# Rule-based approach: you write the logic
def will_buy_rules(user):
"""Predict purchase using hand-written rules"""
if user['age'] > 30 and user['browse_count'] > 3:
return True
if user['cart_count'] > 0 and user['discount_pct'] > 20:
return True
return False
# ML approach: algorithm learns from data
# from sklearn.neighbors import KNeighborsClassifier
# model = KNeighborsClassifier(n_neighbors=5)
# model.fit(X_train, y_train) # Learns patterns from 100K records
# prediction = model.predict(new_user_data)
(3) Returns: From 72% to 89%
Bob trained a model using machine learning, and the accuracy jumped from 72% to 89%. Moreover, for new product categories, all that’s needed is to add more data and retrain the model—without writing a single line of rules.
3. What Is Machine Learning?
(1) Arthur Samuel’s Definition (1959)
"Machine Learning is the field of study that gives computers the ability to learn without being explicitly programmed."
Key idea: It's not about you writing rules for the computer to follow; it's about providing data so the computer can learn the rules on its own.
(2) Tom Mitchell’s definition (1997)
"A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E."
In plain English: When a program’s performance P on task T improves as its experience E increases, it is said to be “learning.”
Take Bob's scenario as an example:
| Symbol | Meaning | Correspondence |
|---|---|---|
| T | Task | Predict whether a user will make a purchase |
| E | Experience | 100K historical purchase records |
| P | Performance | Prediction Accuracy |
(3) Traditional Rules vs. Machine Learning
| Dimension | Traditional Rules | Machine Learning |
|---|---|---|
| Rule Source | Manually written | Learned by an algorithm from data |
| Scalability | Poor (rules expand with the scenario) | Good (can be retrained with new data) |
| Adapting to Changes | Requires manual rule adjustments | Automatically adapts to changes in data distribution |
| Discovering Hidden Patterns | Difficult (Patterns that people cannot think of cannot be found) | Possible (Algorithms can discover nonlinear combinations) |
| Interpretability | Strong (rules are clear at a glance) | Weak (the model is a "black box," but can be interpreted) |
| Development Costs | Low when rules are simple; extremely high when complex | Requires data and computing power in the early stages; low costs later on |
| Comparison | You teach the child each step step-by-step | You show the child 1,000 problems and let them figure it out on their own |
4. The Three Major Paradigms: Supervised / Unsupervised / Reinforcement Learning
Machine learning is divided into three major paradigms based on "learning methods":
graph TB
ML[Machine Learning] --> SL[Supervised Learning]
ML --> UL[Unsupervised Learning]
ML --> RL[Reinforcement Learning]
SL --> SL1[Classification]
SL --> SL2[Regression]
UL --> UL1[Clustering]
UL --> UL2[Dim. Reduction]
RL --> RL1[Game AI]
RL --> RL2[Robot Control]
style ML fill:#e8f5e9
style SL fill:#e1f5fe
style UL fill:#fff3e0
style RL fill:#f3e5f5
(1) Supervised Learning
Key Feature: The training data is labeled (with answers).
It’s like a teacher guiding students through practice problems—each problem has a standard answer, and students learn by following the process of “solving problems → checking answers → correcting mistakes.”
| Element | Description |
|---|---|
| Input | Features X + Label y |
| Learning Process | Find the mapping function from X to Y |
| Output | A model capable of predicting y for a new X |
| Typical Tasks | Classification (discrete labels), Regression (continuous labels) |
| Examples | Predicting housing prices based on property size and location (regression); determining whether an email is spam based on its content (classification) |
▶ Example: Loading the Iris dataset with sklearn and performing classification (Difficulty: ⭐)
PYTHON
# Load the Iris dataset and inspect its structure
from sklearn.datasets import load_iris
iris = load_iris()
print(f"Feature names: {iris.feature_names}")
print(f"Target names: {list(iris.target_names)}")
print(f"Samples: {iris.data.shape[0]}")
print(f"Features: {iris.data.shape[1]}")
print(f"\nFirst 5 samples:")
for i in range(5):
print(f" {iris.data[i]} → {iris.target_names[iris.target[i]]}")
💻 Output:
TEXT
Feature names: ['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)']
Target names: ['setosa', 'versicolor', 'virginica']
Samples: 150
Features: 4
First 5 samples:
[5.1 3.5 1.4 0.2] → setosa
[4.9 3.0 1.4 0.2] → setosa
[4.7 3.2 1.3 0.2] → setosa
[4.6 3.1 1.5 0.2] → setosa
[5.0 3.6 1.4 0.2] → setosa
(2) Unsupervised Learning
Key Feature: The training data is unlabeled.
It's like giving a child a pile of photos and letting him sort them on his own based on what they "look like"—without anyone telling him, "This is a cat" or "That's a dog."
| Element | Description |
|---|---|
| Input | Only feature X, no label y |
| Learning Process | Discovering the intrinsic structure and patterns of data |
| Output | Grouping, structure, and compressed representation of data |
| Typical Tasks | Clustering, Dimension Reduction, Anomaly Detection |
| Examples | Customer segmentation (the number of segments is unknown; the algorithm automatically identifies them), product recommendations (identifying purchasing patterns) |
(3) Reinforcement Learning
Key Feature: Learning through trial and error and reward signals.
It's just like training a puppy—give it a treat (reward) when it does something right, and ignore it (punishment) when it does something wrong. The puppy gradually learns which behaviors earn it a reward.
| Element | Description |
|---|---|
| Input | Environmental State + Action Space |
| Learning Process | Trial and error → Receive reward/punishment → Adjust strategy |
| Output | The strategy that maximizes cumulative rewards in the environment |
| Typical Tasks | Game AI, robot control, autonomous driving decision-making |
| Examples | AlphaGo playing Go, ChatGPT’s human-guided reinforcement learning (RLHF) |
(4) Comparison of the Three Paradigms
| Dimension | Supervised Learning | Unsupervised Learning | Reinforcement Learning |
|---|---|---|---|
| Data | Labeled (X + y) | Unlabeled (X only) | State + Reward Signal |
| Objective | Predict Label | Discover Structure | Maximize Reward |
| Feedback | Immediate (tells you whether you're right or wrong) | None (there is no right or wrong) | Delayed (you don't know if it's good or bad until you've taken many steps) |
| Typical Algorithms | KNN, SVM, Decision Trees | K-Means, PCA, DBSCAN | Q-Learning, PPO, SAC |
| Typical Applications | Spam detection, housing price prediction | Customer segmentation, dimensionality reduction | Game AI, robotics, autonomous driving |
| Data Requirements | High annotation costs | Data is easily accessible | Requires a simulation environment |
| Analogy | A teacher guiding students through problems | Reading a book on your own to identify patterns | Training a puppy to perform tricks |
5. Task Types: Classification / Regression / Clustering
(1) Classification
Predict discrete category labels. For example: Is an email spam or not? Is the image a cat or a dog?
| Type | Description | Example |
|---|---|---|
| Binary Classification | Only 2 categories | Spam/Normal, Fraud/Legitimate |
| Multiple Categories | 3 or more categories | 3 types of irises, handwritten digits 0–9 |
(2) Regression
Predict continuous values. For example: What is the price of a house? What will the temperature be tomorrow?
(3) Clustering
Automatically group data by similarity, without predefined categories. For example: Customers are automatically divided into 3 groups.
(4) Comparison of Classification, Regression, and Clustering
| Dimension | Classification | Regression | Clustering |
|---|---|---|---|
| Output Type | Discrete Categories | Continuous Values | Group Labels |
| Are labels required? | Yes (supervised learning) | Yes (supervised learning) | No (unsupervised learning) |
| Objective | Predict which class a sample belongs to | Predict a specific value | Identify natural groupings in the data |
| Evaluation Metric | Accuracy / F1 | MAE / RMSE / R² | Silhouette / Inertia |
| Example | Determine whether a user has made a purchase | Predict a user's spending amount | Automatically segment users into several groups |
| Analogy | Determine whether a piece of fruit is an apple or an orange | Estimate the weight of a piece of fruit | Sort a basket of mixed fruits into several piles based on their appearance |
6. An Overview of the ML Workflow
A complete ML project consists of six steps, none of which can be skipped:
graph TB
A[1. Define Problem] --> B[2. Collect Data]
B --> C[3. Feature Engineering]
C --> D[4. Train Model]
D --> E[5. Evaluate]
E --> F{Good enough?}
F -->|No| C
F -->|Yes| G[6. Deploy]
G --> H[Monitor & Retrain]
style A fill:#e1f5fe
style B fill:#e8f5e9
style C fill:#fff3e0
style D fill:#f3e5f5
style E fill:#fce4ec
style G fill:#e0f2f1
style H fill:#f1f8e9
(1) Defining the Problem
Clearly define the problem to be solved, whether the task involves classification or regression, and what the criteria for success are.
| Good Question Definition | Poor Question Definition |
|---|---|
| "Predict whether users will purchase an item within 7 days" | "Let AI help us sell more" |
| "Detect fraudulent credit card transactions (F1 ≥ 0.9)" | "Detect anomalous transactions" |
(2) Collecting Data
Data is the cornerstone of machine learning. Data quality directly determines the upper limit of a model's performance.
| Data Dimension | Good Data | Poor Data |
|---|---|---|
| Volume | Sufficient (at least several thousand entries) | Too few (several dozen entries) |
| Quality | Accurate labels, complete features | Incorrect labels, many missing values |
| Diversity | Covers a variety of scenarios | Covers only a single scenario |
(3) Feature Engineering
Convert the raw data into numerical features that the model can understand:
TEXT
Raw Data → Feature Engineering → Features
"25-year-old male from Beijing" → [25, 1, 39] (age, gender_code, city_code)
(4) Train the model
Select an algorithm, set the hyperparameters, and fit the model to the training data.
(5) Evaluating the Model
Evaluate performance on the test set (data the model has never seen before) to ensure that the model can generalize.
(6) Deployment and Launch
Deploy the model as an API or embed it in an application, continuously monitor its performance, and retrain it periodically.
7. Unified Paradigm of the sklearn API
scikit-learn (sklearn) is the most popular ML library for Python, and its API design is extremely consistent—all models follow the same set of interfaces:
TEXT
model = SomeClassifier(params) # 1. Create model
model.fit(X_train, y_train) # 2. Train model
model.predict(X_new) # 3. Predict
model.score(X_test, y_test) # 4. Evaluate
▶ Example: Training and Prediction with the KNN Classifier (Difficulty: ⭐⭐)
PYTHON
# KNN classifier: train and predict
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
# Load and split data
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
# Create and train model
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
# Predict
y_pred = knn.predict(X_test)
# Evaluate
train_acc = knn.score(X_train, y_train)
test_acc = knn.score(X_test, y_test)
print(f"Train accuracy: {train_acc:.3f}")
print(f"Test accuracy: {test_acc:.3f}")
# Predict a new sample
import numpy as np
new_sample = np.array([[5.0, 3.4, 1.5, 0.2]])
pred = knn.predict(new_sample)
print(f"New sample predicted class: {load_iris().target_names[pred[0]]}")
💻 Output:
TEXT
Train accuracy: 0.971
Test accuracy: 0.978
New sample predicted class: setosa
(1) Quick Reference for Common sklearn Classifiers
| Classifier | Class Name | Key Parameters | Features |
|---|---|---|---|
| KNN | KNeighborsClassifier |
n_neighbors |
Simple and intuitive, suitable for small datasets |
| Decision Tree | DecisionTreeClassifier |
max_depth |
Highly interpretable, prone to overfitting |
| Random Forest | RandomForestClassifier |
n_estimators |
Ensemble method, highly robust |
| SVM | SVC |
C, kernel |
High accuracy with small datasets |
| Logistic Regression | LogisticRegression |
C |
Linear baseline, probability output |
| Naive Bayes | GaussianNB |
— | Commonly used for text classification; extremely fast |
💡 Tip: All sklearn classifiers support the
fit() / predict() / score() trio—to switch algorithms, you only need to change the class name in one line.
8. Complete Example: Using KNN to Complete the Full ML Workflow on the Iris Dataset
Below is a complete example that walks through the entire ML workflow, from data loading to the decision boundary for visualization:
▶ Example: Comparing Accuracy Rates for Different k Values (Difficulty: ⭐⭐)
PYTHON
# Compare KNN with different k values
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
print("=== KNN: Effect of k value ===")
for k in [1, 3, 5, 7, 9, 15, 30]:
knn = KNeighborsClassifier(n_neighbors=k)
knn.fit(X_train, y_train)
train_acc = knn.score(X_train, y_train)
test_acc = knn.score(X_test, y_test)
print(f"k={k:2d} | Train: {train_acc:.3f} | Test: {test_acc:.3f}")
💻 Output:
TEXT
=== KNN: Effect of k value ===
k= 1 | Train: 1.000 | Test: 0.956
k= 3 | Train: 0.971 | Test: 0.978
k= 5 | Train: 0.971 | Test: 0.978
k= 7 | Train: 0.971 | Test: 0.978
k= 9 | Train: 0.971 | Test: 0.978
k=15 | Train: 0.971 | Test: 0.978
k=30 | Train: 0.952 | Test: 0.956
💡 Tip: When k=1, the model achieves 100% accuracy on the training set (risk of overfitting); when k is too large, accuracy decreases (underfitting). k=3–7 is a good choice—this is a practical example of the "bias-variance trade-off."
▶ Example: Interpreting a Classification Report (Difficulty: ⭐⭐)
PYTHON
# Detailed classification report
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print("=== Classification Report ===")
print(classification_report(y_test, y_pred, target_names=load_iris().target_names))
💻 Output:
TEXT
=== Classification Report ===
precision recall f1-score support
setosa 1.00 1.00 1.00 15
versicolor 1.00 0.93 0.97 15
virginica 0.94 1.00 0.97 15
accuracy 0.98 45
macro avg 0.98 0.98 0.98 45
weighted avg 0.98 0.98 0.98 45
How to Interpret the Segment Report:
| Metric | Meaning | Interpretation |
|---|---|---|
| precision | The proportion of samples predicted to be in this class that are actually in that class | versicolor precision 1.00 = all samples predicted to be versicolor are correct |
| recall | The proportion of true samples of that class that were correctly predicted | versicolor recall 0.93 = 1 versicolor sample was misclassified |
| f1-score | The harmonic mean of precision and recall | A composite metric; the closer to 1, the better |
| support | Number of true samples in this class | 15 test samples per class |
▶ Example: Visualizing Decision Boundaries (Difficulty: ⭐⭐⭐)
PYTHON
# Visualize KNN decision boundaries on Iris (2 features)
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.neighbors import KNeighborsClassifier
iris = load_iris()
X = iris.data[:, :2] # Use only first 2 features for 2D plot
y = iris.target
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X, y)
# Create mesh grid for decision boundary
h = 0.02
x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h))
Z = knn.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)
# Plot
plt.figure(figsize=(8, 6))
plt.contourf(xx, yy, Z, alpha=0.3, cmap=plt.cm.Set1)
scatter = plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Set1, edgecolors='black')
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')
plt.title('KNN (k=5) Decision Boundary on Iris')
plt.legend(handles=scatter.legend_elements()[0], labels=list(iris.target_names))
plt.savefig('knn_decision_boundary.png', dpi=150, bbox_inches='tight')
plt.show()
print("Decision boundary plot saved.")
💡 Tip: The figure above illustrates how KNN divides the feature space into three regions—each corresponding to a different class. The smoother the boundaries, the more "conservative" the model (the larger the value of k); the more jagged the boundaries, the more "sensitive" the model (the smaller the value of k).
▶ Example: Comprehensive Example—Complete ML Workflow Using KNN on the Iris Dataset (Difficulty: ⭐⭐⭐)
PYTHON
# ============================================
# Complete ML Workflow: KNN on Iris Dataset
# Step 1-6: Full pipeline with evaluation
# ============================================
import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import (accuracy_score, classification_report,
confusion_matrix)
# Step 1: Load data
iris = load_iris()
X, y = iris.data, iris.target
print(f"Step 1 - Data loaded: {X.shape[0]} samples, {X.shape[1]} features")
print(f" Classes: {list(iris.target_names)}")
print(f" Class distribution: {np.bincount(y)}")
# Step 2: Split data (70% train, 30% test)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
print(f"\nStep 2 - Data split: Train={len(X_train)}, Test={len(X_test)}")
# Step 3: Feature scaling (important for KNN!)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
print(f"\nStep 3 - Features scaled (mean≈0, std≈1)")
print(f" Before scaling - mean: {X_train.mean(axis=0).round(2)}")
print(f" After scaling - mean: {X_train_scaled.mean(axis=0).round(2)}")
# Step 4: Train model
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)
print(f"\nStep 4 - Model trained: KNN (k=5)")
# Step 5: Evaluate
y_pred = knn.predict(X_test_scaled)
test_acc = accuracy_score(y_test, y_pred)
print(f"\nStep 5 - Evaluation:")
print(f" Test accuracy: {test_acc:.3f}")
print(f"\n{classification_report(y_test, y_pred, target_names=iris.target_names)}")
print("Confusion Matrix:")
print(confusion_matrix(y_test, y_pred))
# Step 6: Predict new samples
new_data = np.array([
[5.1, 3.5, 1.4, 0.2], # Likely setosa
[6.7, 3.0, 5.2, 2.3], # Likely virginica
[5.9, 3.0, 4.2, 1.5], # Likely versicolor
])
new_data_scaled = scaler.transform(new_data)
predictions = knn.predict(new_data_scaled)
print("\nStep 6 - New predictions:")
for i, pred in enumerate(predictions):
print(f" Sample {i+1} → {iris.target_names[pred]}")
💻 Output:
TEXT
Step 1 - Data loaded: 150 samples, 4 features
Classes: ['setosa', 'versicolor', 'virginica']
Class distribution: [50 50 50]
Step 2 - Data split: Train=105, Test=45
Step 3 - Features scaled (mean≈0, std≈1)
Before scaling - mean: [5.86 3.06 3.78 1.2 ]
After scaling - mean: [ 0. -0. 0. -0.]
Step 4 - Model trained: KNN (k=5)
Step 5 - Evaluation:
Test accuracy: 1.000
precision recall f1-score support
setosa 1.00 1.00 1.00 15
versicolor 1.00 1.00 1.00 15
virginica 1.00 1.00 1.00 15
accuracy 1.00 45
macro avg 1.00 1.00 1.00 45
weighted avg 1.00 1.00 1.00 45
Confusion Matrix:
[[15 0 0]
[ 0 15 0]
[ 0 0 15]]
Step 6 - New predictions:
Sample 1 → setosa
Sample 2 → virginica
Sample 3 → versicolor
💡 Tip: Feature scaling is very important for KNN—KNN uses distance metrics (such as Euclidean distance), and if one feature is orders of magnitude larger than the others, it will “dominate” the distance calculations.
StandardScaler Scale all features so that their mean is 0 and their standard deviation is 1 to ensure that each feature contributes equally.
❓ FAQ
Q What is the difference between machine learning and deep learning?
A Deep learning is a subset of machine learning. ML encompasses all methods of “learning from data” (KNN, SVM, decision trees, etc.), while DL specifically refers to methods that use multi-layer neural networks. DL performs better on unstructured data such as images, speech, and natural language, but requires more data and computing power; traditional ML remains very practical for structured data. Relationship: AI ⊃ ML ⊃ DL.
Q Does supervised learning always require labels?
A Yes, supervised learning is defined as “learning from labeled data.” Without labels, unsupervised learning (such as clustering and dimensionality reduction) is used. However, “semi-supervised learning” can use a small amount of labeled data combined with a large amount of unlabeled data—which is very practical in scenarios where labeling is costly.
Q What is the difference between clustering and classification?
A Classification is supervised learning—the categories are predefined (cat/dog), and the training data is labeled. Clustering is unsupervised learning—the categories are automatically discovered by the algorithm, and the training data is unlabeled. Classification answers the question, "What category is this?", while clustering answers, "How can the data be grouped?"
Q What is sklearn?
A scikit-learn (sklearn) is the most popular machine learning library for Python, providing a complete set of tools for classification, regression, clustering, dimensionality reduction, feature engineering, model selection, and more. Its API is extremely consistent (fit/predict/score), making it easy to get started and the tool of choice for learning ML. Installation:
pip install scikit-learn.Q How long does it take to train a machine learning model?
A It depends on three factors: the amount of data, the model’s complexity, and the hardware. Training a KNN model on the Iris dataset (150 records) takes less than 1 second; training a random forest on 1 million records can take anywhere from a few minutes to a few hours; training a large GPT-level model requires thousands of GPUs and takes several weeks. For beginner projects, it usually takes just a few seconds to a few minutes.
Q How do you choose the value of k in KNN?
A Cross-validation is typically used to select it. If k is too small (e.g., k=1), overfitting is likely; if k is too large (e.g., k=50), underfitting is likely. Rule of thumb: Start with k=5, then try k=1, 3, 5, 7, 9, and 11 to see which one performs best on the validation set. Generally, choose an odd number to avoid a tie.
📖 Summary
- The essence of machine learning is “letting algorithms learn rules from data, rather than having humans write them”—a shift from rule-driven to data-driven approaches
- Three major paradigms: supervised learning (labeled data, prediction), unsupervised learning (unlabeled data, pattern discovery), and reinforcement learning (trial and error + rewards)
- Task types: classification (discrete labels), regression (continuous values), clustering (automatic grouping)—the choice depends on the problem and the data
- The six steps of an ML workflow—define the problem → collect data → feature engineering → training → evaluation → deployment—constitute an iterative process.
- Unified sklearn API paradigm:
fit()Training →predict()Prediction →score()Evaluation; when changing algorithms, only the class names need to be modified - KNN is the most intuitive classifier: the class of a new sample = the majority class of its k nearest neighbors
📝 Exercises
- Basic Problem (Difficulty ⭐): Use sklearn to train a
KNeighborsClassifier(k=5) on the Wine dataset (load_wine()), and print the training and testing accuracy. The Wine dataset has 13 features and 3 classes. - Advanced Problem (Difficulty ⭐⭐): Compare the accuracy of KNN and decision trees (
DecisionTreeClassifier) on the Wine dataset. Try different values of k (1, 3, 5, 7, 9) and different tree depths (2, 3, 5, None) to find the optimal parameters for each algorithm. - Challenge Question (Difficulty ⭐⭐⭐): Plot the accuracy curve for different values of k—with k (1–30) on the x-axis and test accuracy on the y-axis—and mark the peak. Hint: Use
matplotlib.pyplot.plot(), observe the trend of the curve, and explain why k being too large or too small is undesirable.
