Unsupervised Learning
What if your data isn’t labeled? When you’re faced with a large set of user behavior records, genetic sequences, or sensor data—but have no “correct answers”—unsupervised learning is a powerful tool for uncovering hidden patterns. This chapter explores two main approaches—clustering and dimensionality reduction—to help you understand the principles and practical applications of K-Means and PCA.
1. What You'll Learn
- The Fundamental Difference Between Unsupervised Learning and Supervised Learning
- Principles and Implementation of the K-Means Clustering Algorithm
- The Elbow Method for Selecting the Optimal k-Value
- The Intuition and Mathematical Rationale Behind PCA Dimension Reduction
- Applications of Clustering and Dimension Reduction in Real-World Business Scenarios
2. Story: 100K Users—Who Will Help Organize Them into Groups?
(1) Pain Point: With such a massive user base, it’s hard to know where to start
Charlie’s marketing team had 100K user data points—spending amounts, purchase frequency, registration duration, most recent activity time, and more—but no one knew how to segment the users. Relying on experience, the marketing department divided users into two groups—“new users” and “existing users”—and sent them coupons, but the conversion rate was only 2.3%. The CEO demanded that it be raised to 5%, leaving the team at a loss.
(2) The AI Approach: Using Clustering to Automatically Identify User Groups
After Alice took over, she used K-Means to cluster the users into four groups:
| Cluster ID | Feature | Percentage | Name |
|---|---|---|---|
| Group 0 | High-spending, low-frequency | 15% | High-value buyers |
| Group 1 | Low Spending, High Frequency | 35% | Active Deal Hunters |
| Group 2 | High Spending, High Frequency | 10% | Core VIP |
| Group 3 | Low Spending, Low Frequency | 40% | Inactive Users |
We sent different coupons to each group: recommending new products to high-spending buyers, sending discount coupons to active bargain hunters, offering exclusive benefits to core VIPs, and sending reactivation red packets to inactive users. The conversion rate jumped from 2.3% to 3.1%—a 35% increase.
(3) Benefits: From "Guesswork" to "Data-Driven"
Charlie discovered that the power of unsupervised learning lies not in prediction, but in discovering what you didn’t know existed. Even without labels or prior knowledge, algorithms can still uncover meaningful groupings in the data—and that is the value of unsupervised learning.
3. Unsupervised Learning vs. Supervised Learning
(1) Key Differences
| Dimension | Supervised Learning | Unsupervised Learning |
|---|---|---|
| Training Data | Labeled (X → y) | Unlabeled (only X) |
| Objective | Predict known categories/values | Discover the underlying structure of the data |
| Typical Tasks | Classification, Regression | Clustering, Dimension Reduction, Association Rules |
| Evaluation Methods | Accuracy, F1, RMSE, etc. | Contour coefficient, inertia, visualization |
| Analogy | A teacher teaches students to identify animals | A child sorts toys by color on their own |
| Representative Algorithms | Decision Trees, SVM, Linear Regression | K-Means, PCA, DBSCAN |
(2) Why Is Unsupervised Learning Needed?
- High labeling costs: Medical image annotation requires experts, at a cost of $50–$200 per image
- Label Does Not Exist: User Segmentation, Anomaly Detection—There Is No "Right Answer" at All
- Exploratory Analysis: First, perform clustering to see what the data looks like, then decide what to do using supervised learning
4. K-Means Clustering
(1) What Is Clustering?
Clustering is the process of grouping similar data points together and dissimilar ones into separate groups. The key question is: What does “similar” mean? — This is typically measured using a distance metric, the most common of which is Euclidean distance.
(2) K-Means Algorithm Flowchart
K-Means is the most classic and practical clustering algorithm. Its core idea is: to divide the data into k clusters such that the sum of the distances from each data point to its cluster center is minimized.
flowchart TD
A[Initialize k centroids randomly] --> B[Assign each point to nearest centroid]
B --> C[Recalculate centroids as mean of assigned points]
C --> D{Centroids changed?}
D -- Yes --> B
D -- No --> E[Converged! Output clusters]
style A fill:#e1f5fe
style E fill:#e8f5e9
style D fill:#fff3e0
Algorithm Steps:
- Initialization: Randomly select k points as the initial centroids
- Assignment: Assign each data point to the cluster containing the nearest center of mass
- Update: Recalculate the center of mass for each cluster (by taking the average of all points within the cluster)
- Iteration: Repeat steps 2–3 until the center of mass no longer changes or the maximum number of iterations is reached.
▶ Example: K-Means Clustering of Customer Data (Difficulty: ⭐)
PYTHON
from sklearn.cluster import KMeans
import numpy as np
np.random.seed(42)
group_a = np.random.normal(loc=[20, 80], scale=5, size=(30, 2))
group_b = np.random.normal(loc=[80, 20], scale=5, size=(30, 2))
group_c = np.random.normal(loc=[50, 50], scale=5, size=(30, 2))
X = np.vstack([group_a, group_b, group_c])
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
kmeans.fit(X)
print(f"Cluster centers:\n{kmeans.cluster_centers_}")
print(f"Inertia (sum of squared distances): {kmeans.inertia_:.2f}")
print(f"First 10 labels: {kmeans.labels_[:10]}")
💻 Output:
TEXT
Cluster centers:
[[50.14 49.60]
[20.38 79.63]
[80.27 19.44]]
Inertia (sum of squared distances): 1310.56
First 10 labels: [1 1 1 1 1 1 1 1 1 1]
(3) Comparison of Three Clustering Algorithms
| Dimension | K-Means | Hierarchical Clustering | DBSCAN |
|---|---|---|---|
| Requires specifying k | Yes | No | No (requires specifying eps/min_samples) |
| Cluster Shape | Spherical | Any | Any |
| Noise Handling | Poor | Fair | Good (marked as noise points) |
| Time complexity | O(nkt) | O(n²) ~ O(n³) | O(n log n) |
| Scalability | Good for large datasets | Slow with large datasets | Moderate |
| Applications | Customer segmentation, image compression | Small datasets, tree structures | Anomaly detection, spatial data |
5. How to Choose the Value of k
The biggest problem with K-Means: You need to specify k in advance. If you choose the wrong k, the clustering results will be meaningless. Here are two common methods to help you find the optimal k.
(1) The Elbow Method
Plot the inertia curves (i.e., the sum of squared distances within clusters) corresponding to different values of k, select the k value at the "inflection point"—much like the elbow of an arm—after which the benefit of increasing k diminishes.
▶ Example: Drawing the Rule of Thumbs—Selecting k (Difficulty ⭐)
PYTHON
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
import numpy as np
np.random.seed(42)
X = np.vstack([
np.random.normal(loc=[20, 80], scale=5, size=(50, 2)),
np.random.normal(loc=[80, 20], scale=5, size=(50, 2)),
np.random.normal(loc=[50, 50], scale=5, size=(50, 2)),
np.random.normal(loc=[80, 80], scale=5, size=(50, 2)),
])
inertias = []
K_range = range(1, 11)
for k in K_range:
km = KMeans(n_clusters=k, random_state=42, n_init=10)
km.fit(X)
inertias.append(km.inertia_)
plt.figure(figsize=(8, 5))
plt.plot(K_range, inertias, 'bo-', linewidth=2, markersize=8)
plt.xlabel('Number of clusters (k)')
plt.ylabel('Inertia')
plt.title('Elbow Method for Optimal k')
plt.axvline(x=4, color='red', linestyle='--', label='Elbow at k=4')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('elbow_method.png', dpi=150)
plt.show()
💡 Tip: The Elbow Rule looks for the "inflection point" in the inertial decline. After k=4, the inertial decline slows down noticeably, indicating that four clusters are reasonable.
(2) Silhouette Score
The silhouette coefficient measures the similarity of each point to its own cluster versus its similarity to the nearest other clusters; its values range from [-1, 1], with higher values being better.
▶ Example: Selecting k for Silhouette scoring (Difficulty ⭐⭐)
PYTHON
from sklearn.metrics import silhouette_score
from sklearn.cluster import KMeans
import numpy as np
np.random.seed(42)
X = np.vstack([
np.random.normal(loc=[20, 80], scale=5, size=(50, 2)),
np.random.normal(loc=[80, 20], scale=5, size=(50, 2)),
np.random.normal(loc=[50, 50], scale=5, size=(50, 2)),
np.random.normal(loc=[80, 80], scale=5, size=(50, 2)),
])
print("k | Silhouette Score")
print("--|------------------")
for k in range(2, 8):
km = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = km.fit_predict(X)
score = silhouette_score(X, labels)
print(f"{k} | {score:.4f}")
💻 Output:
TEXT
k | Silhouette Score
--|------------------
2 | 0.5812
3 | 0.6234
4 | 0.6891
5 | 0.5543
6 | 0.4672
7 | 0.3891
💡 Tip: The Silhouette score is highest when k=4, which is consistent with the findings of the Rule of Thumb. Cross-validation of the two methods yields more reliable results.
(3) Comparison of Clustering Evaluation Methods
| Method | What It Measures | Range of Values | Advantages | Disadvantages |
|---|---|---|---|---|
| Inertia | Intra-cluster density | ≥0; the smaller, the better | Fast computation; built into K-Means | Decreases monotonically as k increases; cannot be used on its own |
| Silhouette | Compactness + Dispersion | [-1, 1], the higher the better | Allows comparison of different k values | Slow computation, O(n²) |
| Calinski-Harabasz | Inter-cluster/intra-cluster variance ratio | ≥0; the higher, the better | Fast computation | Prefers convex clusters |
| Davies-Bouldin | Intra-cluster divergence/inter-cluster distance ratio | ≥0; the smaller, the better | Intuitive | Prefers convex clusters |
6. PCA Dimension Reduction
(1) Why is dimensionality reduction necessary?
- Dimension Catastrophe: Too many features cause distance measures to lose meaning, leading to a decline in model performance
- Visualization: Humans can only perceive 2D and 3D; high-dimensional data must be reduced to lower dimensions before it can be visualized.
- Noise Reduction: The main information is concentrated in the first few principal components; the subsequent ones may be noise.
- Acceleration: Faster training and lower memory usage after dimensionality reduction
(2) PCA: An Intuitive Explanation
The core idea of PCA (Principal Component Analysis) is to identify the directions with the greatest variance in the data and project the data onto those directions. Greater variance means more information, and the goal is to preserve as much of the original information as possible after projection.
Imagine a set of 3D data points arranged in a "flattened" ellipsoid—PCA identifies the longest axis (the one with the greatest variance) and projects the data onto that axis, transforming 3D into 1D while minimizing information loss.
▶ Example: Reducing Dimensions to 2D Using PCA and Visualizing the Results (Difficulty: ⭐⭐)
PYTHON
from sklearn.decomposition import PCA
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
import numpy as np
iris = load_iris()
X = iris.data
y = iris.target
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)
print(f"Original shape: {X.shape}")
print(f"After PCA: {X_pca.shape}")
print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
print(f"Total variance explained: {pca.explained_variance_ratio_.sum():.4f}")
plt.figure(figsize=(8, 6))
for target, color, label in zip([0, 1, 2], ['red', 'blue', 'green'], iris.target_names):
plt.scatter(X_pca[y == target, 0], X_pca[y == target, 1],
c=color, label=label, alpha=0.7, edgecolors='k', s=60)
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%} variance)')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%} variance)')
plt.title('PCA: Iris Dataset Reduced to 2D')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('pca_iris.png', dpi=150)
plt.show()
💻 Output:
TEXT
Original shape: (150, 4)
After PCA: (150, 2)
Explained variance ratio: [0.9246 0.0530]
Total variance explained: 0.9776
💡 Tip: Reducing 4 dimensions to 2 still preserves 97.8% of the information! The first two principal components almost “tell the whole story.”
(3) Comparison of Dimension Reduction Methods
| Method | Type | Core Concept | Preserves Global Structure | Preserves Local Structure | Interpretability | Applicable Scenarios |
|---|---|---|---|---|---|---|
| PCA | Linear | Direction of maximum variance | Good | Poor | High (feature weights are interpretable) | Fast dimension reduction, noise reduction |
| t-SNE | Nonlinear | Neighborhood Probability Preserved | Poor | Good | Low | Visualization (2D/3D) |
| UMAP | Nonlinear | Topological structure preservation | Fair | Good | Low | Visualization, large-scale data |
| Autoencoder | Nonlinear | Neural network compression | Structure-aware | Structure-aware | Medium | Image/text dimensionality reduction |
7. Introduction to t-SNE Visualization
PCA is a linear dimension reduction method that excels at preserving global structure but tends to "flatten" nonlinear relationships. t-SNE (t-Distributed Stochastic Neighbor Embedding) is specifically designed for visualization and excels at preserving local neighborhood structures—similar points remain close to each other in 2D.
▶ Example: Visualizing High-Dimensional Data with t-SNE (Difficulty: ⭐⭐)
PYTHON
from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt
import numpy as np
digits = load_digits()
X = digits.data
y = digits.target
print(f"Original shape: {X.shape}") # 64-dimensional handwritten digits
tsne = TSNE(n_components=2, random_state=42, perplexity=30)
X_tsne = tsne.fit_transform(X)
plt.figure(figsize=(10, 8))
scatter = plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap='tab10',
alpha=0.7, edgecolors='k', s=30)
plt.colorbar(scatter, label='Digit Class')
plt.title('t-SNE: 64D Handwritten Digits → 2D')
plt.xlabel('t-SNE Dimension 1')
plt.ylabel('t-SNE Dimension 2')
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('tsne_digits.png', dpi=150)
plt.show()
⚠️ Note: t-SNE is only suitable for visualization and should not be used as a preprocessing step for downstream models. It is not reproducible (results vary across runs), does not preserve global distances, and is computationally slow (O(n²)). Use PCA or UMAP for dimensionality reduction in production environments.
8. Practical Applications of Clustering and Dimension Reduction
(1) Customer Segmentation
The e-commerce, finance, and gaming industries use clustering to segment users into different groups and develop differentiated strategies:
| Industry | Clustering Features | Clustering Results | Business Actions |
|---|---|---|---|
| E-commerce | Spending Amount, Frequency, and Category Preferences | High-Value/Low-Value/At-Risk of Churn | Tailored Coupons |
| Finance | Income, Liabilities, Credit History | Low Risk/Medium Risk/High Risk | Tiered Interest Rates |
| Games | Playtime, Payments, Social Activity | Whales/Dolphins/Freeloaders | Differentiated Operations |
(2) Image Compression
K-Means can be used for image color quantization—grouping millions of colors into k categories, thereby significantly reducing file size.
(3) Anomaly Detection
Points that do not belong to any cluster may be outliers. Points flagged as noise by DBSCAN and points that are far from the cluster centers warrant attention.
(4) Feature Engineering
The principal components obtained through PCA can be used as new features to feed into supervised learning models, thereby removing noise, reducing multicollinearity, and accelerating training.
9. Comparison of Clustering Results with Original Labels
Sometimes you have labels but want to see if unsupervised clustering can "discover" these categories on its own—this is an excellent way to test the quality of the clustering.
▶ Example: Comparing Clustering Results with Original Labels (Difficulty: ⭐⭐)
PYTHON
from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.metrics import confusion_matrix, accuracy_score
from scipy.stats import mode
import numpy as np
iris = load_iris()
X = iris.data
y_true = iris.target
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
y_pred = kmeans.fit_predict(X)
# Cluster labels are arbitrary, so we align them with true labels
aligned_pred = np.zeros_like(y_pred)
for i in range(3):
mask = y_pred == i
aligned_pred[mask] = mode(y_true[mask], keepdims=True).mode[0]
print("Confusion Matrix (cluster vs true label):")
print(confusion_matrix(y_true, aligned_pred))
print(f"\nAligned accuracy: {accuracy_score(y_true, aligned_pred):.4f}")
💻 Output:
TEXT
Confusion Matrix (cluster vs true label):
[[50 0 0]
[ 0 48 2]
[ 0 14 36]]
Aligned accuracy: 0.8933
💡 Tip: Unsupervised clustering automatically achieved a classification accuracy of 89.3% without knowing the labels! This indicates that the three varieties of irises do indeed have a distinct clustering structure in the feature space.
10. Limitations of Unsupervised Learning
| Limitations | Description |
|---|---|
| No Guaranteed Results | Without labels, it is impossible to quantify whether something is "right" or "wrong"; one can only rely on business judgment |
| The choice of k is subjective | The inflection point of the "elbow rule" may not be obvious, and different methods may yield different conclusions |
| Sensitive to initialization | Different initial cluster centers in K-Means may produce different results (requires n_init > 1) |
| Can only detect convex clusters | K-Means assumes clusters are spherical; ring-shaped or crescent-shaped data require DBSCAN |
| Dimension reduction results in information loss | PCA discards dimensions with low variance, which may result in the loss of important subtle differences |
| Challenges in Evaluation | There is no "standard answer"; evaluation relies on domain knowledge and visualization |
11. Comprehensive Example: Customer Segmentation Using the "Mall Customer" Dataset
▶ Example: The Complete Process of Data-Driven Customer Segmentation (Difficulty: ⭐⭐⭐)
PYTHON
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
# ============================================
# Step 1: Load and explore the Mall Customer dataset
# ============================================
np.random.seed(42)
n = 200
data = {
'CustomerID': range(1, n + 1),
'Gender': np.random.choice(['Male', 'Female'], n),
'Age': np.random.randint(18, 70, n),
'Annual_Income_kUSD': np.random.randint(15, 137, n),
'Spending_Score': np.random.randint(1, 100, n),
}
df = pd.DataFrame(data)
# Inject realistic cluster structure
df.loc[:49, 'Annual_Income_kUSD'] = np.random.randint(15, 40, 50)
df.loc[:49, 'Spending_Score'] = np.random.randint(60, 100, 50)
df.loc[50:99, 'Annual_Income_kUSD'] = np.random.randint(70, 137, 50)
df.loc[50:99, 'Spending_Score'] = np.random.randint(60, 100, 50)
df.loc[100:149, 'Annual_Income_kUSD'] = np.random.randint(70, 137, 50)
df.loc[100:149, 'Spending_Score'] = np.random.randint(1, 40, 50)
df.loc[150:, 'Annual_Income_kUSD'] = np.random.randint(15, 40, 50)
df.loc[150:, 'Spending_Score'] = np.random.randint(1, 40, 50)
print("Dataset shape:", df.shape)
print(df.head(10))
print("\nDescriptive stats:")
print(df[['Age', 'Annual_Income_kUSD', 'Spending_Score']].describe())
# ============================================
# Step 2: Feature selection and scaling
# ============================================
features = ['Age', 'Annual_Income_kUSD', 'Spending_Score']
X = df[features].values
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# ============================================
# Step 3: Elbow method to find optimal k
# ============================================
inertias = []
silhouette_scores = []
K_range = range(2, 11)
for k in K_range:
km = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = km.fit_predict(X_scaled)
inertias.append(km.inertia_)
silhouette_scores.append(silhouette_score(X_scaled, labels))
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
axes[0].plot(K_range, inertias, 'bo-', linewidth=2)
axes[0].set_xlabel('Number of clusters (k)')
axes[0].set_ylabel('Inertia')
axes[0].set_title('Elbow Method')
axes[0].axvline(x=4, color='red', linestyle='--', label='Elbow at k=4')
axes[0].legend()
axes[0].grid(True, alpha=0.3)
axes[1].plot(K_range, silhouette_scores, 'go-', linewidth=2)
axes[1].set_xlabel('Number of clusters (k)')
axes[1].set_ylabel('Silhouette Score')
axes[1].set_title('Silhouette Method')
axes[1].axvline(x=4, color='red', linestyle='--', label='Best at k=4')
axes[1].legend()
axes[1].grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('mall_elbow_silhouette.png', dpi=150)
plt.show()
# ============================================
# Step 4: Final clustering with k=4
# ============================================
kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
df['Cluster'] = kmeans.fit_predict(X_scaled)
# ============================================
# Step 5: Analyze each cluster
# ============================================
cluster_summary = df.groupby('Cluster')[features].mean()
print("\nCluster profiles (mean values):")
print(cluster_summary)
cluster_counts = df['Cluster'].value_counts().sort_index()
print("\nCluster sizes:")
print(cluster_counts)
# ============================================
# Step 6: PCA for 2D visualization
# ============================================
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
plt.figure(figsize=(10, 7))
colors = ['red', 'blue', 'green', 'orange']
labels_text = [
'Cluster 0: Young High-Spenders',
'Cluster 1: Rich High-Spenders',
'Cluster 2: Rich Low-Spenders',
'Cluster 3: Budget Low-Spenders',
]
for i in range(4):
mask = df['Cluster'] == i
plt.scatter(X_pca[mask, 0], X_pca[mask, 1],
c=colors[i], label=labels_text[i],
alpha=0.6, edgecolors='k', s=50)
centroids_pca = pca.transform(kmeans.cluster_centers_)
plt.scatter(centroids_pca[:, 0], centroids_pca[:, 1],
c='black', marker='X', s=200, label='Centroids')
plt.xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%})')
plt.ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%})')
plt.title('Mall Customer Segmentation (K-Means + PCA)')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('mall_clusters_pca.png', dpi=150)
plt.show()
# ============================================
# Step 7: Business recommendations
# ============================================
print("\n=== Business Recommendations ===")
for i in range(4):
row = cluster_summary.loc[i]
count = cluster_counts[i]
print(f"\nCluster {i} ({count} customers):")
print(f" Avg Age: {row['Age']:.0f}, Income: ${row['Annual_Income_kUSD']:.0f}k, Spending: {row['Spending_Score']:.0f}/100")
💻 Output:
TEXT
Dataset shape: (200, 5)
CustomerID Gender Age Annual_Income_kUSD Spending_Score
0 1 Female 56 28 86
1 2 Male 25 32 79
2 3 Female 38 22 93
3 4 Female 36 37 74
4 5 Male 47 33 68
5 6 Female 32 27 82
6 7 Female 51 18 88
7 8 Male 19 38 95
8 9 Male 23 25 71
9 10 Female 27 39 91
Descriptive stats:
Age Annual_Income_kUSD Spending_Score
count 200.000000 200.000000 200.000000
mean 43.475000 60.720000 50.115000
std 15.441418 41.082376 31.494432
min 18.000000 15.000000 1.000000
max 69.000000 136.000000 99.000000
Cluster profiles (mean values):
Age Annual_Income_kUSD Spending_Score
Cluster
0 41.34 26.82 80.44
1 43.88 99.24 78.32
2 44.92 101.58 19.44
3 43.36 26.24 21.12
Cluster sizes:
0 50
1 50
2 50
3 50
Name: Cluster, dtype: int64
=== Business Recommendations ===
Cluster 0 (50 customers):
Avg Age: 41, Income: $27k, Spending: 80/100
Cluster 1 (50 customers):
Avg Age: 44, Income: $99k, Spending: 78/100
Cluster 2 (50 customers):
Avg Age: 45, Income: $102k, Spending: 19/100
Cluster 3 (50 customers):
Avg Age: 43, Income: $26k, Spending: 21/100
❓ FAQ
Q How do you evaluate the quality of clustering results?
A When no labels are available, use internal metrics: the Silhouette score ([-1,1], where higher is better), the Calinski-Harabasz index (where higher is better), and the Davies-Bouldin index (where lower is better). When labels are available, compare the results to the true labels (calculate accuracy after alignment). The most reliable method is business validation—check whether the clustering results align with business intuition and are actionable.
Q How do you choose the value of k for K-Means?
A First, use the “elbow rule” to identify the inflection point in the inertia plot, then cross-validate the results using the silhouette score. If the two methods yield consistent results, you can confidently use that value; if they do not, prioritize business needs—it’s easier to develop differentiated strategies for 4 clusters than for 8. k is not a mathematical problem; it’s a business problem.
Q Does PCA dimension reduction result in information loss?
A Yes, but the "lost" information isn’t necessarily bad. PCA sorts dimensions by variance from highest to lowest; discarding dimensions with low variance is equivalent to discarding noise. The key is how much variance is retained—typically, 85% or more is sufficient. Use
pca.explained_variance_ratio_ to view the contribution of each principal component, then decide how many to retain.Q What are the practical applications of unsupervised learning?
A Five typical scenarios: ① Customer segmentation (personalized marketing) ② Anomaly detection (fraud detection) ③ Dimension reduction (feature engineering) ④ Image compression (color quantization) ⑤ Cold-start in recommendation systems (clustering first when no behavioral data is available). At its core, it is “exploratory analysis”—first examining what the data looks like, then deciding what to do with it.
Q Which is harder, clustering or classification?
A Clustering is harder because there is no “correct answer.” Classification has labels, so you can measure accuracy to determine performance; clustering has no labels, so evaluation relies on internal metrics and business judgment. Furthermore, K-Means is sensitive to initialization, the choice of k is subjective, and it is only suitable for convex clusters. Classification is like an “exam with a standard answer,” while clustering is like an “open-ended question with no standard answer.”
📖 Summary
- Unsupervised learning does not use labels; its goal is to discover the intrinsic structure of the data—clustering to identify groups, and dimensionality reduction to identify patterns.
- K-Means is the most practical clustering algorithm: initialization → assignment → update → iteration—simple and efficient
- Select k using the rule of thumb + Silhouette score cross-validation; the final decision will be based on business requirements.
- PCA projects data along the direction of maximum variance, enabling rapid dimensionality reduction, noise reduction, and visualization, but discards nonlinear structures.
- t-SNE is suitable for 2D/3D visualization and preserves local neighborhoods, but it is not suitable for model preprocessing.
- Limitations of unsupervised learning: difficult to evaluate, subjective k-selection, sensitive to initialization, and can only detect convex clusters
📝 Exercises
-
Basic Problem (Difficulty ⭐): Use
sklearn.datasets.make_blobsto generate 3 clusters of data, perform K-Means clustering, and plot a scatter plot (using different colors for each cluster and labeling the cluster centers). -
Advanced Problem (Difficulty ⭐⭐): Plot a Rule of Thumbs diagram and a Silhouette score plot for the data above to determine the optimal value of k. Then, intentionally cluster the data using k=2 and k=8, respectively, compare the results, and explain why choosing the wrong value of k leads to poorer results.
-
Challenge (Difficulty: ⭐⭐⭐): Load the handwritten digit dataset (64-dimensional) using
sklearn.datasets.load_digits. First, use PCA to reduce it to a 2D visualization, then use t-SNE to reduce it to a 2D visualization, and compare the results of the two methods. Write an analysis: What structural features do PCA and t-SNE each preserve? Which is more suitable for visualizing this dataset?
