KNN & Clustering
Birds of a feather flock together — KNN finds "the neighbors most like you" for classification, while K-Means finds "the tightest groups" for segmentation.
1. What You'll Learn
- KNN classification: distance metrics (Euclidean/Manhattan/Cosine), choosing K, voting mechanism
- K-Means clustering: algorithm workflow, choosing K (elbow method/silhouette score), convergence criteria
- DBSCAN density clustering: core points/border points/noise points, tuning eps and min_samples
- Hierarchical clustering: AgglomerativeClustering and dendrograms
- Bob's user profiling segmentation: RFM clustering to identify high-value/dormant/churned user groups
2. A Real Story from a User Operations Manager
(1) The Pain: One-Size-Fits-All Marketing for 500K Users Is Inefficient
Bob's platform has 500 thousand users, and every user receives the same promotional emails. The result: high-value users find the promotions too low-end, dormant users never open the emails, and the overall conversion rate is only 2%. Users are diverse — a one-size-fits-all strategy wastes 80% of the marketing budget.
(2) The Clustering Solution
Clustering can automatically divide users into high-value/active/dormant/churned groups, enabling precision marketing.
PYTHON
from sklearn.cluster import KMeans
# RFM clustering: group users by Recency, Frequency, Monetary
rfm = df[["recency", "frequency", "monetary"]]
kmeans = KMeans(n_clusters=4, random_state=42)
df["segment"] = kmeans.fit_predict(rfm)
for i in range(4):
segment = df[df["segment"] == i]
print(f"Segment {i}: {len(segment)} users, "
f"Avg Monetary={segment['monetary'].mean():.0f} USD")
(3) The Payoff: Precision Marketing Boosts Conversion 4x
After Bob used clustering to divide users into 4 segments, the high-value group received premium offers, the dormant group received re-engagement packages, and the overall conversion rate jumped from 2% to 8% — a 3x improvement in marketing ROI.
3. KNN Classification
(1) Distance Metrics and Choosing K
▶ Example: KNN Classification + Comparing Different K Values
PYTHON
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
X, y = load_iris(return_X_y=True)
# Compare different K values
for k in [1, 3, 5, 7, 11, 21]:
pipe = Pipeline([
("scaler", StandardScaler()),
("knn", KNeighborsClassifier(n_neighbors=k)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print(f"K={k:2d}: Accuracy={scores.mean():.3f} +/- {scores.std():.3f}")
Output:
TEXT
# Executed successfully
| Distance Metric | Formula Characteristics | Best For |
|---|---|---|
| Euclidean | Straight-line distance | Continuous features, all dimensions equally important |
| Manhattan | City-block distance | High-dimensional data, many outliers |
| Cosine | Angular distance | Text vectors, direction matters more than magnitude |
| Minkowski | Generalized distance (p=2→Euclidean, p=1→Manhattan) | Flexible tuning |
(2) The Effect of K
| K Value | Decision Boundary | Risk |
|---|---|---|
| K=1 | Extremely irregular | Overfitting (sensitive to noise) |
| K=small (3-7) | Moderately curved | Usually optimal |
| K=large (20+) | Nearly linear | Underfitting |
▶ Example: KNN User Purchase Prediction
PYTHON
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import numpy as np
rng = np.random.default_rng(42)
n = 1000
X = np.column_stack([
rng.uniform(0, 100, n), # browsing_time
rng.uniform(0, 50, n), # cart_value_usd
rng.integers(1, 30, n), # pages_viewed
rng.integers(0, 10, n), # previous_purchases
])
y = (X[:, 1] * 0.05 + X[:, 3] * 0.3 + rng.normal(0, 0.5, n) > 2).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
pipe = Pipeline([
("scaler", StandardScaler()),
("knn", KNeighborsClassifier(n_neighbors=7, weights="distance")),
])
pipe.fit(X_train, y_train)
print(classification_report(y_test, pipe.predict(X_test)))
Output:
TEXT
# Executed successfully
4. K-Means Clustering
(1) Algorithm Workflow
sequenceDiagram
participant Init as Initialize K Centers
participant Assign as Assign Points
participant Update as Update Centers
participant Check as Converged?
Init->>Assign: Random K center positions
loop Until convergence
Assign->>Update: Each point → nearest center
Update->>Check: Centers = mean of assigned points
Check->>Assign: Not converged (centers moved)
end
Check-->>Done: Converged! Return clusters
▶ Example: K-Means Clustering + Elbow Method
PYTHON
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import numpy as np
rng = np.random.default_rng(42)
# Generate 4 natural clusters
X = np.vstack([
rng.normal([20, 500], [3, 50], (200, 2)), # High value
rng.normal([50, 200], [5, 30], (300, 2)), # Medium
rng.normal([80, 50], [8, 20], (350, 2)), # Low
rng.normal([10, 800], [2, 40], (150, 2)), # VIP
])
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Elbow method
inertias = []
K_range = range(2, 10)
for k in K_range:
km = KMeans(n_clusters=k, random_state=42, n_init=10)
km.fit(X_scaled)
inertias.append(km.inertia_)
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(K_range, inertias, "bo-", linewidth=2)
ax.set_xlabel("Number of Clusters (K)")
ax.set_ylabel("Inertia (Within-Cluster Sum of Squares)")
ax.set_title("Elbow Method for Optimal K")
ax.axvline(x=4, color="red", linestyle="--", label="Elbow at K=4")
ax.legend()
plt.tight_layout()
plt.savefig("elbow_method.png", dpi=150)
Output:
TEXT
# Executed successfully
(2) Silhouette Score
▶ Example: Choosing K with the Silhouette Score
PYTHON
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.preprocessing import StandardScaler
import numpy as np
# Using same X_scaled from previous example
for k in range(2, 8):
km = KMeans(n_clusters=k, random_state=42, n_init=10)
labels = km.fit_predict(X_scaled)
score = silhouette_score(X_scaled, labels)
print(f"K={k}: Silhouette Score={score:.3f}")
Output:
TEXT
# Executed successfully
| Metric | Meaning | How to Determine Optimal K |
|---|---|---|
| Inertia (elbow method) | Within-cluster sum of squares | At the elbow point |
| Silhouette Score | Cohesion vs. separation | At the maximum value |
| Gap Statistic | Comparison against random distribution | First K below the upper bound |
5. DBSCAN Density Clustering
▶ Example: DBSCAN Discovers Arbitrarily Shaped Clusters
PYTHON
from sklearn.cluster import DBSCAN
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score
import numpy as np
rng = np.random.default_rng(42)
# Create non-spherical clusters (two circles)
from sklearn.datasets import make_circles
X, _ = make_circles(n_samples=500, factor=0.5, noise=0.05, random_state=42)
X_scaled = StandardScaler().fit_transform(X)
# K-Means vs DBSCAN
km = KMeans(n_clusters=2, random_state=42, n_init=10)
km_labels = km.fit_predict(X_scaled)
db = DBSCAN(eps=0.3, min_samples=10)
db_labels = db.fit_predict(X_scaled)
n_clusters = len(set(db_labels)) - (1 if -1 in db_labels else 0)
n_noise = list(db_labels).count(-1)
print(f"K-Means Silhouette: {silhouette_score(X_scaled, km_labels):.3f}")
print(f"DBSCAN Clusters: {n_clusters}, Noise points: {n_noise}")
if n_clusters > 1:
db_valid = db_labels != -1
print(f"DBSCAN Silhouette: {silhouette_score(X_scaled[db_valid], db_labels[db_valid]):.3f}")
Output:
TEXT
# Executed successfully
| Dimension | K-Means | DBSCAN |
|---|---|---|
| Cluster shape | Spherical | Arbitrary shapes |
| K value | Must be preset | Discovered automatically |
| Noise points | Not handled | Labeled as -1 |
| Uneven density | Performs poorly | Still handles well |
| Computation speed | Fast | Moderate |
6. Bob's RFM User Profiling Segmentation
▶ Example: Complete RFM Clustering Project
PYTHON
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 5000
df = pd.DataFrame({
"user_id": range(10001, 10001 + n),
"recency_days": rng.integers(1, 365, n),
"frequency": rng.integers(1, 50, n),
"monetary_usd": rng.exponential(500, n),
})
# RFM features
rfm = df[["recency_days", "frequency", "monetary_usd"]]
# Log transform for skewed features
rfm_log = rfm.copy()
rfm_log["frequency"] = np.log1p(rfm_log["frequency"])
rfm_log["monetary_usd"] = np.log1p(rfm_log["monetary_usd"])
# Scale
scaler = StandardScaler()
rfm_scaled = scaler.fit_transform(rfm_log)
# K-Means with K=4
km = KMeans(n_clusters=4, random_state=42, n_init=10)
df["segment"] = km.fit_predict(rfm_scaled)
# Analyze segments
segment_summary = df.groupby("segment").agg({
"recency_days": "mean",
"frequency": "mean",
"monetary_usd": "mean",
"user_id": "count",
}).round(1)
segment_summary.columns = ["avg_recency", "avg_frequency", "avg_monetary", "count"]
segment_summary = segment_summary.sort_values("avg_monetary", ascending=False)
# Label segments
labels = ["Champions", "Loyal", "At Risk", "Lost"]
for idx, (_, row) in enumerate(segment_summary.iterrows()):
print(f"{labels[idx]:10s}: {int(row['count']):5d} users, "
f"Recency={row['avg_recency']:.0f}d, "
f"Freq={row['avg_frequency']:.1f}, "
f"Monetary={row['avg_monetary']:.0f} USD")
Output:
TEXT
# Executed successfully
| Segment | Recency | Frequency | Monetary | Marketing Strategy |
|---|---|---|---|---|
| Champions | Low (recently active) | High | High | VIP service, premium recommendations |
| Loyal | Medium | Medium-high | Medium | Loyalty programs, cross-selling |
| At Risk | High (long time since purchase) | Medium | Medium | Re-engagement packages, limited-time offers |
| Lost | Very high | Low | Low | Low-cost outreach, survey research |
❓ FAQ
Q What's the difference between K in KNN and K in K-Means?
A K in KNN means "how many neighbors vote" — it's a classification algorithm. K in K-Means means "how many groups to form" — it's a clustering algorithm. They are entirely different concepts.
Q Does KNN require standardization?
A Absolutely. KNN is distance-based, so unscaled features (e.g., income 0–100,000 vs. age 0–100) will completely dominate the distance calculation. Always pair it with StandardScaler.
Q How do you choose K for K-Means?
A Combine the elbow method and silhouette score. The elbow method finds the inflection point; the silhouette score finds the maximum. They may disagree — defer to business reasonableness.
Q How do you choose eps for DBSCAN?
A Plot the K-distance graph (sorted k-nearest neighbor distances) and find the elbow. Typical eps values range from 0.1–1.0 (after standardization). min_samples is usually set to 2 × number of dimensions.
Q How do you evaluate clustering results?
A Unsupervised clustering has no "ground truth." Use the silhouette score (mathematical evaluation) + business interpretability (domain evaluation) for dual validation. Only clusters that map to meaningful business concepts are valuable.
Q Why apply a log transform before RFM clustering?
A Frequency and Monetary typically have right-skewed distributions — a few high-value users stretch the inter-cluster distances. A log transform makes the distribution more symmetric, improving K-Means performance.
📖 Summary
- KNN classification: pick K nearest neighbors by distance metric and vote; small K overfits, large K underfits; standardization is mandatory
- K-Means clustering: iteratively assign points and update centers; use the elbow method and silhouette score to choose K; only works well for spherical clusters
- DBSCAN density clustering: automatically discovers the number of clusters, labels noise, handles arbitrary shapes, but is sensitive to parameters
- RFM user segmentation is a classic e-commerce clustering application: Recency + Frequency + Monetary → user profiles
- Apply log transforms to handle skewed distributions before clustering; standardize to eliminate scale differences
- Cluster evaluation = mathematical metrics (silhouette score) + business interpretability — both are essential
📝 Exercises
- Basic (Difficulty ⭐): Use KNN to classify Iris. Compare cross-validation accuracy for K=1, 5, 10, 20 and find the optimal K. Hint: Pipeline(StandardScaler + KNN) + cross_val_score.
- Intermediate (Difficulty ⭐⭐): Generate 4-cluster data with make_blobs. Cluster with both K-Means and DBSCAN, then compare silhouette scores and noise-point handling. Hint: DBSCAN labels noise as -1.
- Challenge (Difficulty ⭐⭐⭐): Implement Bob's RFM clustering — generate simulated user data, apply log transform + standardization, choose K with the elbow method, run K-Means, name each segment meaningfully (Champions/Loyal/At Risk/Lost), and output the average RFM values per segment. Hint: Refer to the complete example in Section 6.
← Previous: SVM Support Vector Machines | Next: Feature Engineering →
