Dimensionality Reduction
In high-dimensional space, every point is far away from every other — dimensionality reduction isn't just compression, it's a way to uncover the true structure of your data.
1. What You'll Learn
- The curse of dimensionality: distance breakdown and data sparsity in high-dimensional spaces
- PCA (Principal Component Analysis): covariance matrices, eigenvalue decomposition, and explained variance ratio
- t-SNE manifold visualization: the perplexity parameter, KL divergence, and mapping from high dimensions to 2D/3D
- A quick look at UMAP: a faster alternative to t-SNE that preserves global structure
- E-commerce user behavior case study: 50-dim behavior features → 2D visualization that reveals natural user segments
2. A Real Data Scientist's Story
(1) The Pain: 50-Dimensional User Behavior Data You Can't See or Understand
Bob collected 50 user behavior features (session duration, click counts, engagement ratios per category, and so on), but he had no way to intuitively grasp how users were distributed — tables were unreadable, scatter plots cap out at two dimensions, and clustering results were hard to validate. High-dimensional data is a black box to human intuition.
(2) The Fix: PCA + t-SNE
PCA compresses the data to two dimensions while preserving maximum variance, and t-SNE maps local structure onto a 2D plane — making high-dimensional data something you can actually see.
PYTHON
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
X_scaled = StandardScaler().fit_transform(X_50d)
# PCA: fast, preserves global variance
X_pca = PCA(n_components=2).fit_transform(X_scaled)
# t-SNE: slow, preserves local structure
X_tsne = TSNE(n_components=2, perplexity=30).fit_transform(X_scaled)
(3) The Payoff: A 2D Plot Instantly Reveals Three Natural User Groups
After dimensionality reduction, Bob could see three distinct user groups at a glance — high-value, active, and dormant — which lined up closely with his clustering results and let the business team understand the data distribution in seconds.
3. The Curse of Dimensionality
(1) Counterintuitive Phenomena in High-Dimensional Space
- Distance breaks down: in high-dimensional space, the distance to the nearest point and the farthest point converge toward the same value
- Data sparsity: volume grows exponentially with dimension, so a fixed number of samples becomes increasingly sparse
- Overfitting risk: when features >> samples, models tend to "memorize the noise"
▶ Example: A Curse-of-Dimensionality Experiment
PYTHON
import numpy as np
rng = np.random.default_rng(42)
for dim in [2, 10, 50, 100, 500]:
n = 1000
X = rng.uniform(0, 1, (n, dim))
# Compute pairwise distances
from sklearn.metrics import pairwise_distances
dists = pairwise_distances(X)
np.fill_diagonal(dists, np.inf)
min_dist = dists.min()
max_dist = dists.max()
ratio = (max_dist - min_dist) / max_dist
print(f"Dim={dim:4d}: min={min_dist:.3f}, max={max_dist:.3f}, "
f"relative_range={ratio:.4f}")
Output:
TEXT
# Runs successfully
| Dimensions | Min Distance | Max Distance | Relative Range | Meaning |
|---|---|---|---|---|
| 2 | 0.02 | 1.41 | 0.99 | Distances are meaningful |
| 10 | 0.78 | 1.87 | 0.58 | Distances start to break down |
| 50 | 2.42 | 3.28 | 0.26 | Distances are nearly useless |
| 100 | 3.54 | 4.05 | 0.13 | All distances converge |
| 500 | 8.11 | 8.47 | 0.04 | Completely broken |
4. PCA (Principal Component Analysis)
(1) How PCA Works
PCA uses eigenvalue decomposition to find the directions of maximum variance (the principal components) and projects the data onto a lower-dimensional space.
graph TB
INPUT[Standardized Data] --> COV[Compute Covariance Matrix]
COV --> EIG[Eigenvalue Decomposition]
EIG --> SELECT[Select Top K Eigenvectors]
SELECT --> PROJECT[Project Data to K Dimensions]
PROJECT --> OUTPUT[Low-Dimensional Representation]
▶ Example: PCA Reduction and Explained Variance Ratio
PYTHON
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
import matplotlib.pyplot as plt
import numpy as np
X, y = load_iris(return_X_y=True)
# Standardize first (PCA is sensitive to scale)
X_scaled = StandardScaler().fit_transform(X)
# PCA with all components to see explained variance
pca_full = PCA()
pca_full.fit(X_scaled)
print("Explained variance ratio:")
for i, (evr, cum) in enumerate(zip(pca_full.explained_variance_ratio_,
pca_full.explained_variance_ratio_.cumsum())):
print(f" PC{i+1}: {evr:.3f} (cumulative: {cum:.3f})")
# Reduce to 2D
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print(f"\n2D PCA preserves {pca.explained_variance_ratio_.sum():.1%} variance")
# Visualize
fig, ax = plt.subplots(figsize=(8, 6))
for target in np.unique(y):
mask = y == target
ax.scatter(X_pca[mask, 0], X_pca[mask, 1], label=load_iris().target_names[target], s=40)
ax.set_xlabel(f"PC1 ({pca.explained_variance_ratio_[0]:.1%})")
ax.set_ylabel(f"PC2 ({pca.explained_variance_ratio_[1]:.1%})")
ax.set_title("PCA of Iris Dataset")
ax.legend()
plt.tight_layout()
plt.savefig("pca_iris.png", dpi=150)
Output:
TEXT
Explained variance ratio:
(2) Choosing How Many Components to Keep
▶ Example: Cumulative Explained Variance Plot
PYTHON
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np
rng = np.random.default_rng(42)
n, p = 500, 50
X = rng.standard_normal((n, p))
# Make first 5 dimensions have more signal
X[:, :5] *= 3
X_scaled = StandardScaler().fit_transform(X)
pca = PCA().fit(X_scaled)
fig, ax = plt.subplots(figsize=(10, 5))
cumvar = pca.explained_variance_ratio_.cumsum()
ax.plot(range(1, len(cumvar)+1), cumvar, "b-o", markersize=3)
ax.axhline(0.95, color="red", linestyle="--", label="95% variance")
ax.axhline(0.90, color="orange", linestyle="--", label="90% variance")
ax.set_xlabel("Number of Components")
ax.set_ylabel("Cumulative Explained Variance")
ax.set_title("PCA Explained Variance")
ax.legend()
n_95 = (cumvar < 0.95).sum() + 1
print(f"Components for 95% variance: {n_95}")
plt.tight_layout()
plt.savefig("pca_variance.png", dpi=150)
Output:
TEXT
# Runs successfully
| Threshold | Meaning | Typical Result |
|---|---|---|
| 90% | Keeps the main signal | 5–15 components (for 50 dims) |
| 95% | Keeps more detail | 10–25 components (for 50 dims) |
| 99% | Nearly lossless | 20–40 components (for 50 dims) |
5. t-SNE Manifold Visualization
(1) How t-SNE Works
t-SNE preserves local neighborhood relationships by minimizing the KL divergence between pairwise similarities in the high-dimensional and low-dimensional spaces.
▶ Example: t-SNE Visualization
PYTHON
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt
import numpy as np
X, y = load_digits(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)
# t-SNE with different perplexity
fig, axes = plt.subplots(1, 3, figsize=(18, 5))
for ax, perplexity in zip(axes, [5, 30, 50]):
tsne = TSNE(n_components=2, perplexity=perplexity, random_state=42, init="pca")
X_tsne = tsne.fit_transform(X_scaled)
scatter = ax.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap="tab10", s=5, alpha=0.7)
ax.set_title(f"t-SNE (perplexity={perplexity})")
ax.set_xlabel("t-SNE 1")
ax.set_ylabel("t-SNE 2")
plt.tight_layout()
plt.savefig("tsne_perplexity.png", dpi=150)
Output:
TEXT
# Runs successfully
(2) Things to Watch Out for with t-SNE
| Consideration | Notes |
|---|---|
| perplexity | Typically 5–50. Smaller → focuses on close neighbors; larger → focuses on global structure |
| Random seed | Different seeds can give different results — fix random_state |
| Not for downstream use | t-SNE doesn't preserve distances, so it's unsuitable as feature input to a model |
| Slow to compute | For large datasets (>10k), reduce with PCA first, then run t-SNE |
| Cluster size is meaningless | t-SNE can exaggerate or shrink the relative sizes of clusters |
▶ Example: Speeding Up t-SNE with PCA Preprocessing
PYTHON
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
rng = np.random.default_rng(42)
X = rng.standard_normal((5000, 50))
# Direct t-SNE (slow)
# tsne = TSNE(n_components=2, random_state=42)
# X_tsne = tsne.fit_transform(X) # Very slow on 5000x50!
# Best practice: PCA first, then t-SNE
X_scaled = StandardScaler().fit_transform(X)
X_pca_30 = PCA(n_components=30).fit_transform(X_scaled) # Fast PCA to 30D
X_tsne = TSNE(n_components=2, perplexity=30, random_state=42,
init="pca", learning_rate="auto").fit_transform(X_pca_30)
print(f"PCA→t-SNE shape: {X_tsne.shape}")
Output:
TEXT
# Runs successfully
6. Hands-On: Dimensionality Reduction for E-Commerce User Behavior
▶ Example: 50-Dim User Features → 2D Visualization
PYTHON
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import numpy as np
rng = np.random.default_rng(42)
n = 3000
# Generate 50-dim user behavior data with 3 natural groups
group1 = rng.normal(loc=2, scale=1, size=(800, 50)) # High-value users
group2 = rng.normal(loc=0, scale=1.5, size=(1200, 50)) # Regular users
group3 = rng.normal(loc=-2, scale=0.8, size=(1000, 50)) # Dormant users
X = np.vstack([group1, group2, group3])
# Add noise dimensions
X[:, 20:] += rng.normal(0, 3, (n, 30))
# Standardize
X_scaled = StandardScaler().fit_transform(X)
# PCA to 2D
X_pca = PCA(n_components=2).fit_transform(X_scaled)
# t-SNE to 2D (with PCA preprocessing)
X_pca30 = PCA(n_components=30).fit_transform(X_scaled)
X_tsne = TSNE(n_components=2, perplexity=30, random_state=42,
init="pca", learning_rate="auto").fit_transform(X_pca30)
# K-Means labels for reference
labels = KMeans(n_clusters=3, random_state=42, n_init=10).fit_predict(X_scaled)
# Visualize
fig, axes = plt.subplots(1, 2, figsize=(16, 6))
for ax, X_2d, title in [(axes[0], X_pca, "PCA"), (axes[1], X_tsne, "t-SNE")]:
scatter = ax.scatter(X_2d[:, 0], X_2d[:, 1], c=labels, cmap="Set1", s=8, alpha=0.6)
ax.set_title(title)
ax.set_xlabel(f"{title} 1")
ax.set_ylabel(f"{title} 2")
plt.tight_layout()
plt.savefig("user_behavior_dim_reduction.png", dpi=150)
Output:
TEXT
# Runs successfully
(1) PCA vs t-SNE vs UMAP
| Aspect | PCA | t-SNE | UMAP |
|---|---|---|---|
| What it preserves | Global variance | Local neighborhoods | Global + local |
| Speed | Fast (<1s) | Slow (minutes) | Medium (seconds) |
| Reproducibility | Deterministic | Seed-dependent | Seed-dependent |
| Interpretability | High (explained variance) | Low | Medium |
| Downstream use | ✅ Usable as features | ❌ Visualization only | ⚠️ Needs validation |
| Mapping new data | ✅ transform | ❌ Must rerun | ✅ transform |
❓ FAQ
Q Can I use PCA-reduced data for model training?
A Yes. The principal components from PCA can be used as model input, and they're especially useful for high-dimensional, small-sample scenarios. Just remember: fit PCA on the training set and only transform the test set.
Q Why can't t-SNE be used as model features?
A t-SNE doesn't preserve global distance relationships, and it can't transform new data (you have to recompute from scratch each time). It's only suitable for visualization, not as model input.
Q How do I choose the perplexity parameter?
A The typical range is 5–50. For small datasets (~1k), use 5–20; for large data (10k+), use 30–50. Try a few values and pick whichever produces the clearest clusters.
Q Does PCA require standardization?
A Absolutely. PCA maximizes variance, so if your features are on different scales (e.g., age vs. income), the high-variance features will dominate the principal components. Always run StandardScaler before PCA.
Q What makes UMAP better than t-SNE?
A Three advantages — 1) it's 10–100x faster; 2) it preserves more global structure; 3) it supports transforming new data. The catch is that you need to install it separately with
pip install umap-learn.Q How much information is lost in dimensionality reduction?
A It depends on how many components you keep. Retaining 95% of the variance usually means losing only about 5% of the information — and what's dropped may just be noise. Use
cumsum(explained_variance_ratio_) to decide.📖 Summary
- The curse of dimensionality: distances break down, data gets sparse, and overfitting risk rises — dimensionality reduction is essential
- PCA: preserves the directions of maximum variance; great for fast reduction and model input, but you must standardize first
- t-SNE: preserves local neighborhood relationships; ideal for visualization, but unusable as model features and slow to compute
- UMAP: a t-SNE alternative that's faster, preserves global structure, and supports mapping new data
- A typical pipeline: StandardScaler → PCA (speedup + denoising) → t-SNE/UMAP (visualization)
- Choosing the number of PCA components: look at the cumulative explained variance ratio; 90–95% is a common threshold
📝 Exercises
- Basic (Difficulty ⭐): Run PCA on the Iris dataset to reduce it to 2 dimensions, plot a scatter chart, and annotate the explained variance ratio. Hint:
PCA(n_components=2)+ scatter. - Intermediate (Difficulty ⭐⭐): On the
load_digitsdata (64 dimensions), first reduce to 30 dimensions with PCA, then to 2 dimensions with t-SNE. Compare the runtime and visualization quality of direct t-SNE versus PCA + t-SNE. Hint: time it withtime.time(). - Challenge (Difficulty ⭐⭐⭐): Generate 50-dimensional data with 3 user groups, reduce it to 2D using PCA, t-SNE, and UMAP respectively, and evaluate with K-Means clustering (ARI/NMI) to determine which method best preserves the original group structure. Hint:
sklearn.metrics.adjusted_rand_score.
← Previous: Ensemble Learning | Next: NLP Fundamentals →
