Support Vector Machines
SVM finds the "widest road" — pushing the decision boundary as far as possible from both classes, naturally making it more robust.
1. What You'll Learn
- Maximum-margin classifier: support vectors, margin boundaries, hard margin vs. soft margin (C parameter)
- Kernel trick: intuition and selection strategies for linear/RBF/polynomial kernels
- SVM regression (SVR): ε-insensitive loss function
- Why scaling matters: SVM's sensitivity to feature scales
- Charlie's customer segmentation: classifying European market customers into value tiers with SVM
2. A Real-World Story from a Market Analyst
(1) The Pain Point: Blurry Boundaries in Customer Value Tiers
Charlie is responsible for customer segmentation in the European market, dividing customers into High/Medium/Low value tiers to allocate service resources. But the boundary between High and Medium is hard to draw — using a single metric (e.g., spending amount) for tiering causes 20% of customers near the boundary to be frequently misclassified. A linear boundary cannot capture the complex relationships across multiple customer dimensions.
(2) The SVM Kernel Trick Solution
SVM uses the kernel trick to map data into a higher-dimensional space, finding a clear boundary where the data was previously inseparable.
PYTHON
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
# RBF kernel maps to high-dimensional space
pipe = Pipeline([
("scaler", StandardScaler()),
("svm", SVC(kernel="rbf", C=1.0, gamma="scale")),
])
pipe.fit(X_train, y_train)
print(f"Accuracy: {pipe.score(X_test, y_test):.4f}")
(3) The Result: Boundary Customer Accuracy Improved from 65% to 89%
After replacing linear tiering with SVM-RBF, Charlie saw classification accuracy for boundary customers jump from 65% to 89%, reducing service resource misallocation by 40%.
3. Maximum-Margin Classifier
(1) Core Idea of SVM
SVM finds the separating hyperplane that maximizes the margin. Only the samples lying on the margin boundaries (support vectors) determine the classification outcome.
graph TB
INPUT[Input Data] --> MAP[Kernel Mapping<br/>Low-D → High-D]
MAP --> HYPER[Find Max-Margin<br/>Hyperplane]
HYPER --> SV[Support Vectors<br/>Define the boundary]
SV --> MARGIN[Margin Width<br/>Larger = More Robust]
MARGIN --> CLASSIFY[Classification<br/>Decision Function]
▶ Example: Linear SVM Visualization
PYTHON
from sklearn.svm import SVC
from sklearn.datasets import make_blobs
import matplotlib.pyplot as plt
import numpy as np
X, y = make_blobs(n_samples=100, centers=2, random_state=42, cluster_std=1.5)
# Train SVM with linear kernel
svm = SVC(kernel="linear", C=1.0)
svm.fit(X, y)
fig, ax = plt.subplots(figsize=(8, 6))
# Plot data points
ax.scatter(X[:, 0], X[:, 1], c=y, cmap="bwr", s=30, edgecolors="black")
# Plot decision boundary and margins
xlim = ax.get_xlim()
ylim = ax.get_ylim()
xx = np.linspace(xlim[0], xlim[1], 30)
yy = np.linspace(ylim[0], ylim[1], 30)
YY, XX = np.meshgrid(yy, xx)
xy = np.vstack([XX.ravel(), YY.ravel()]).T
Z = svm.decision_function(xy).reshape(XX.shape)
ax.contour(XX, YY, Z, colors="k", levels=[-1, 0, 1], alpha=0.5, linestyles=["--", "-", "--"])
# Highlight support vectors
ax.scatter(svm.support_vectors_[:, 0], svm.support_vectors_[:, 1],
s=100, facecolors="none", edgecolors="k", linewidths=2)
ax.set_title(f"Linear SVM (C={svm.C})\nSupport vectors: {len(svm.support_vectors_)}")
plt.tight_layout()
plt.savefig("svm_linear.png", dpi=150)
Output:
TEXT
# Executed successfully
(2) Hard Margin vs. Soft Margin (C Parameter)
| C Value | Margin | Misclassification Tolerance | Overfitting Risk | Use Case |
|---|---|---|---|---|
| Large (100+) | Narrow | Almost none | High | Nearly linearly separable data |
| Medium (1) | Moderate | Some tolerance | Low | General cases |
| Small (0.01) | Wide | High tolerance | Low | Noisy/overlapping data |
▶ Example: Comparing the Effect of C Parameter
PYTHON
from sklearn.svm import SVC
from sklearn.datasets import make_blobs
from sklearn.model_selection import cross_val_score
import numpy as np
X, y = make_blobs(n_samples=200, centers=2, random_state=42, cluster_std=2.5)
for C in [0.01, 0.1, 1.0, 10.0, 100.0]:
svm = SVC(kernel="rbf", C=C, gamma="scale")
scores = cross_val_score(svm, X, y, cv=5, scoring="accuracy")
svm.fit(X, y)
print(f"C={C:6.2f}: Accuracy={scores.mean():.3f} +/- {scores.std():.3f}, "
f"Support vectors={len(svm.support_vectors_)}")
Output:
TEXT
# Executed successfully
4. The Kernel Trick
(1) Choosing a Kernel Function
▶ Example: Comparing Different Kernel Functions
PYTHON
from sklearn.svm import SVC
from sklearn.datasets import make_moons, make_circles
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
# Non-linear data (moons)
X_moons, y_moons = make_moons(n_samples=500, noise=0.2, random_state=42)
kernels = {
"Linear": SVC(kernel="linear", C=1),
"Polynomial (deg=3)": SVC(kernel="poly", degree=3, C=1),
"RBF": SVC(kernel="rbf", C=1, gamma="scale"),
}
for name, svm in kernels.items():
pipe = Pipeline([("scaler", StandardScaler()), ("svm", svm)])
scores = cross_val_score(pipe, X_moons, y_moons, cv=5, scoring="accuracy")
print(f"{name:20s}: Accuracy={scores.mean():.3f}")
Output:
TEXT
# Executed successfully
| Kernel | Formula | Use Case | Key Parameters |
|---|---|---|---|
| Linear | $\langle x, x' \rangle$ | High-dimensional, linearly separable | C |
| RBF | $e^{-\gamma|x-x'|^2}$ | General-purpose, non-linear | C, gamma |
| Polynomial | $(\langle x, x' \rangle + c)^d$ | Feature interactions | C, degree, coef0 |
| Sigmoid | $\tanh(\gamma\langle x, x' \rangle + c)$ | Neural-network-like | C, gamma, coef0 |
(2) The Gamma Parameter
▶ Example: Effect of Gamma on the RBF Kernel
PYTHON
from sklearn.svm import SVC
from sklearn.datasets import make_moons
from sklearn.model_selection import cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
X, y = make_moons(n_samples=500, noise=0.15, random_state=42)
for gamma in [0.01, 0.1, 1.0, 10.0, "scale", "auto"]:
pipe = Pipeline([
("scaler", StandardScaler()),
("svm", SVC(kernel="rbf", C=1, gamma=gamma)),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print(f"gamma={str(gamma):6s}: Accuracy={scores.mean():.3f}")
Output:
TEXT
# Executed successfully
| Gamma | Decision Boundary | Effect |
|---|---|---|
| Very small (0.01) | Smooth | Underfitting |
| Moderate (1) | Moderately curved | Usually optimal |
| Very large (10+) | Highly irregular | Overfitting |
| "scale" | 1/(n_features * X.var()) | sklearn default |
| "auto" | 1/n_features | Legacy default |
5. Why Scaling Matters and SVR
(1) SVM's Sensitivity to Feature Scales
▶ Example: Impact of Scaling on SVM
PYTHON
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
# Without scaling
svm_raw = SVC(kernel="rbf", C=1, gamma="scale")
scores_raw = cross_val_score(svm_raw, X, y, cv=5, scoring="accuracy")
# With scaling
pipe = Pipeline([("scaler", StandardScaler()), ("svm", SVC(kernel="rbf", C=1, gamma="scale"))])
scores_scaled = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print(f"Without scaling: {scores_raw.mean():.4f}")
print(f"With scaling: {scores_scaled.mean():.4f}")
Output:
TEXT
# Executed successfully
⚠️ Note: SVM uses distance metrics to compute the margin. If feature scales differ greatly (e.g., age 0–100 vs. income 0–1,000,000), the large-scale feature will dominate the margin calculation. Always pair SVM with StandardScaler.
(2) SVR Regression
▶ Example: SVR Sales Forecasting
PYTHON
from sklearn.svm import SVR
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error, r2_score
import numpy as np
rng = np.random.default_rng(42)
X = rng.uniform(0, 100, (200, 3))
y = 50 + 0.8 * X[:, 0] + rng.normal(0, 5, 200)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# SVR with RBF kernel
pipe = Pipeline([
("scaler", StandardScaler()),
("svr", SVR(kernel="rbf", C=100, epsilon=5)),
])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
print(f"SVR R²: {r2_score(y_test, y_pred):.4f}")
print(f"SVR MAE: {mean_absolute_error(y_test, y_pred):.2f}")
Output:
TEXT
# Executed successfully
| Parameter | Meaning | Effect |
|---|---|---|
| C | Regularization strength | Large C → less error tolerance → possible overfitting |
| epsilon (ε) | Insensitive band width | Large ε → more points ignored → smoother fit |
| kernel | Kernel function type | Same as classification SVM |
6. Charlie's European Customer Value Segmentation
▶ Example: Complete SVM Classification Project
PYTHON
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.metrics import classification_report
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 2000
df = pd.DataFrame({
"annual_spending_eur": rng.exponential(5000, n),
"purchase_frequency": rng.poisson(8, n),
"avg_order_value_eur": rng.exponential(150, n),
"tenure_months": rng.integers(1, 60, n),
"support_tickets": rng.poisson(3, n),
})
# Value tier logic (EUR-based)
score = (df["annual_spending_eur"] / 5000 * 0.3
+ df["purchase_frequency"] / 20 * 0.25
+ df["avg_order_value_eur"] / 200 * 0.2
+ df["tenure_months"] / 60 * 0.15
+ rng.normal(0, 0.1, n))
df["tier"] = pd.cut(score, bins=[0, 0.3, 0.6, 1.5], labels=["Low", "Medium", "High"])
X = df.drop(columns=["tier"])
y = df["tier"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
# Grid search for best SVM parameters
pipe = Pipeline([("scaler", StandardScaler()), ("svm", SVC())])
param_grid = {
"svm__kernel": ["linear", "rbf"],
"svm__C": [0.1, 1, 10],
"svm__gamma": ["scale", 0.1, 1],
}
grid = GridSearchCV(pipe, param_grid, cv=3, scoring="accuracy", n_jobs=-1)
grid.fit(X_train, y_train)
print(f"Best params: {grid.best_params_}")
print(f"Best CV accuracy: {grid.best_score_:.4f}")
print(f"\nTest report:\n{classification_report(y_test, grid.predict(X_test))}")
Output:
TEXT
# Executed successfully
❓ FAQ
Q Why must SVM data be scaled?
A SVM computes margins based on distance metrics. If feature A ranges 0–1 and feature B ranges 0–1,000,000, B will completely dominate the margin calculation. StandardScaler ensures all features contribute equally.
Q Which matters more for the RBF kernel — gamma or C?
A Both are important and they interact with each other. Use GridSearchCV to search them jointly. Typical search ranges: C=[0.1, 1, 10, 100], gamma=[0.001, 0.01, 0.1, 1, "scale"].
Q Is SVM suitable for large datasets?
A Not really. SVM training complexity is O(n²) to O(n³), making it very slow beyond 100 thousand samples. For large datasets, use LinearSVC (linear kernel) or Random Forest/XGBoost.
Q Can SVM output probabilities?
A Not by default. Setting probability=True trains an additional Platt Scaling model to output probabilities, but training becomes slower and the probabilities are not as well-calibrated as logistic regression.
Q What's the difference between a linear kernel and logistic regression?
A Both are linear classifiers. The difference — SVM maximizes the margin (focusing only on boundary samples), while logistic regression maximizes likelihood (considering all samples). SVM is more robust near the boundary; logistic regression gives better-calibrated probabilities.
Q How do you choose the epsilon parameter for SVR?
A Epsilon is the width of the insensitive band — errors smaller than ε are not counted in the loss. A typical value is an estimate of the noise level in the target variable. For example, if sales forecast errors are within ±5k USD, set ε=5.
📖 Summary
- SVM finds the maximum-margin hyperplane; only support vectors define the boundary — all other samples have no influence
- The C parameter controls the trade-off between margin width and misclassification: large C → narrow margin, fewer misclassifications; small C → wide margin, more tolerance
- The kernel trick solves non-linear classification by mapping data to a higher-dimensional space; the RBF kernel is the default first choice
- Gamma controls the influence range of the RBF kernel: small gamma → smooth boundary, large gamma → overfitting
- SVM requires feature scaling — distance metrics are extremely sensitive to feature scales
- SVR uses ε-insensitive loss for regression; prediction errors smaller than ε are not penalized
📝 Exercises
- Basic (Difficulty ⭐): Use SVC(kernel="rbf") to classify the make_moons dataset. Compare accuracy with and without StandardScaler. Hint: Pipeline vs. direct fit.
- Intermediate (Difficulty ⭐⭐): Use GridSearchCV to search for optimal SVM parameters (kernel, C, gamma) on the Iris dataset and find the best combination. Hint: SVC inside a Pipeline + param_grid.
- Challenge (Difficulty ⭐⭐⭐): Reproduce Charlie's customer segmentation — generate 3-class customer data, compare classification reports for SVM/LogisticRegression/RandomForest, and analyze which model works best for multi-class scenarios with blurry boundaries. Hint: Use classification_report to compare precision/recall/f1.
← Previous: Decision Trees and Random Forests | Next: KNN and Clustering →
