Logistic Regression
Logistic regression isn't really "regression" — it's a powerful classifier that outputs probabilities, not raw numbers.
1. What You'll Learn
- From linear to logistic: the sigmoid function, probability outputs, and decision boundaries
- The loss function: cross-entropy loss and the intuition behind maximum likelihood estimation
- Multiclass extensions: One-vs-Rest and Softmax (Multinomial)
- Evaluation metrics: Accuracy/Precision/Recall/F1/ROC-AUC and how to choose for your business scenario
- Alice's user churn prediction: identifying high-risk customers in a SaaS setting
2. A Real Story from SaaS Operations
(1) The Pain Point: Churn Detected Too Late, Costly to Win Back
Alice runs a SaaS platform with 50 thousand active users and a monthly churn rate of 8% — that's 4 thousand users lost every month. With each user worth about 2 thousand USD per year, the monthly loss is roughly 670 thousand USD. The problem: by the time a user cancels their subscription, it's already too late — the win-back success rate is under 5%.
(2) The Logistic Regression Solution
Logistic regression can predict churn probability as soon as user behavior shows warning signs, giving a 2–4 week early warning and lifting the win-back success rate to 30%.
PYTHON
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
model = LogisticRegression(class_weight="balanced")
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred, target_names=["Retained", "Churned"]))
(3) The Payoff: Each Saved Customer Is Worth 2k USD
Using logistic regression, Alice identifies high-churn-risk users in advance and saves 1.2 thousand users per month, adding roughly 2.4 million USD in annual revenue. Winning back a customer costs about 50 USD — far less than the 500 USD cost of acquiring a new one.
3. From Linear to Logistic
(1) The Sigmoid Function
Logistic regression uses the sigmoid function to squash the linear output into the [0,1] range as a probability: $\sigma(z) = \frac{1}{1+e^{-z}}$
graph LR
INPUT[Input Features x] --> LINEAR[Linear Combination<br/>z = w·x + b]
LINEAR --> SIGMOID[Sigmoid Function<br/>σ = 1/(1+e^(-z))]
SIGMOID --> PROB[P(y=1) = σ]
PROB --> DECISION{P ≥ 0.5?}
DECISION -->|Yes| CLASS1[Class 1<br/>e.g. Churned]
DECISION -->|No| CLASS0[Class 0<br/>e.g. Retained]
▶ Example: Visualizing the Sigmoid Function
PYTHON
import numpy as np
import matplotlib.pyplot as plt
z = np.linspace(-8, 8, 100)
sigmoid = 1 / (1 + np.exp(-z))
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(z, sigmoid, linewidth=2, color="#2196F3")
ax.axhline(0.5, color="red", linestyle="--", alpha=0.5, label="Decision boundary")
ax.axvline(0, color="gray", linestyle="--", alpha=0.3)
ax.set_xlabel("z (linear combination)")
ax.set_ylabel("σ(z) = P(y=1)")
ax.set_title("Sigmoid Function")
ax.legend()
ax.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig("sigmoid.png", dpi=150)
Output:
TEXT
# Executed successfully
(2) Cross-Entropy Loss
The loss function for logistic regression is cross-entropy (Log Loss):
$L = -\frac{1}{n}\sum[y\log(\hat{y}) + (1-y)\log(1-\hat{y})]$
| True Label | Predicted Probability | Loss | Intuition |
|---|---|---|---|
| y=1 | ŷ=0.99 | 0.01 | Almost no penalty |
| y=1 | ŷ=0.5 | 0.69 | Moderate penalty |
| y=1 | ŷ=0.01 | 4.60 | Huge penalty |
| y=0 | ŷ=0.01 | 0.01 | Almost no penalty |
| y=0 | ŷ=0.99 | 4.60 | Huge penalty |
4. Logistic Regression with Scikit-learn
▶ Example: Binary Classification — Predicting User Purchase Intent
PYTHON
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
import numpy as np
rng = np.random.default_rng(42)
n = 1000
X = rng.standard_normal((n, 4)) # [browsing_time, pages_viewed, cart_value, visit_count]
y = (X[:, 0] * 0.5 + X[:, 1] * 1.2 + X[:, 2] * 2.0 + rng.normal(0, 0.5, n) > 1.5).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
pipe = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression(random_state=42)),
])
pipe.fit(X_train, y_train)
y_pred = pipe.predict(X_test)
y_prob = pipe.predict_proba(X_test)[:, 1] # Probability of class 1
print(f"Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"\nConfusion Matrix:\n{confusion_matrix(y_test, y_pred)}")
print(f"\nClassification Report:\n{classification_report(y_test, y_pred)}")
print(f"\nSample probabilities: {y_prob[:5].round(3)}")
Output:
TEXT
# Executed successfully
▶ Example: Alice's SaaS User Churn Prediction
PYTHON
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, precision_recall_fscore_support
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 5000
df = pd.DataFrame({
"tenure_months": rng.integers(1, 60, n),
"monthly_usage_hours": rng.exponential(20, n),
"support_tickets": rng.poisson(2, n),
"payment_delay_days": rng.integers(0, 30, n),
"plan_tier": rng.choice([1, 2, 3], n, p=[0.5, 0.35, 0.15]),
})
# Churn logic: short tenure + low usage + many tickets + delays
churn_score = (
-0.05 * df["tenure_months"]
- 0.1 * df["monthly_usage_hours"]
+ 0.5 * df["support_tickets"]
+ 0.1 * df["payment_delay_days"]
+ rng.normal(0, 1, n)
)
df["churned"] = (churn_score > 2).astype(int)
print(f"Churn rate: {df['churned'].mean():.1%}")
# Train with class_weight="balanced" for imbalanced data
X = df.drop(columns=["churned"])
y = df["churned"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
model = LogisticRegression(class_weight="balanced", max_iter=500, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, y_prob)
precision, recall, f1, _ = precision_recall_fscore_support(y_test, y_pred, average="binary")
print(f"AUC-ROC: {auc:.4f}")
print(f"Precision: {precision:.4f}, Recall: {recall:.4f}, F1: {f1:.4f}")
Output:
TEXT
# Executed successfully
5. Multiclass Extensions
(1) One-vs-Rest vs Softmax
▶ Example: Multiclass — Automatic Product Category Classification
PYTHON
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
from sklearn.model_selection import cross_val_score
X, y = load_iris(return_X_y=True)
# One-vs-Rest (default)
ovr_model = LogisticRegression(multi_class="ovr", max_iter=200)
ovr_scores = cross_val_score(ovr_model, X, y, cv=5, scoring="accuracy")
# Softmax (Multinomial)
softmax_model = LogisticRegression(multi_class="multinomial", max_iter=200)
softmax_scores = cross_val_score(softmax_model, X, y, cv=5, scoring="accuracy")
print(f"One-vs-Rest: {ovr_scores.mean():.4f} +/- {ovr_scores.std():.4f}")
print(f"Softmax: {softmax_scores.mean():.4f} +/- {softmax_scores.std():.4f}")
Output:
TEXT
# Executed successfully
| Dimension | One-vs-Rest | Softmax (Multinomial) |
|---|---|---|
| Principle | K binary classifiers | 1 multiclass classifier |
| Probability calibration | No guarantee they sum to 1 | Strictly sums to 1 |
| Computational efficiency | Parallelizable | Requires joint optimization |
| Best for | Many classes | Few classes, probabilities needed |
6. Classification Metrics and Business Choices
(1) The Confusion Matrix and Derived Metrics
▶ Example: ROC Curve and PR Curve
PYTHON
from sklearn.metrics import roc_curve, auc, precision_recall_curve
import matplotlib.pyplot as plt
import numpy as np
# Assume y_test and y_prob from previous example
rng = np.random.default_rng(42)
y_test = rng.choice([0, 1], 500, p=[0.85, 0.15])
y_prob = np.clip(y_test * 0.8 + rng.normal(0, 0.3, 500), 0, 1)
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# ROC Curve
fpr, tpr, _ = roc_curve(y_test, y_prob)
roc_auc = auc(fpr, tpr)
axes[0].plot(fpr, tpr, color="#2196F3", lw=2, label=f"AUC = {roc_auc:.3f}")
axes[0].plot([0, 1], [0, 1], "r--", alpha=0.5)
axes[0].set_xlabel("False Positive Rate")
axes[0].set_ylabel("True Positive Rate")
axes[0].set_title("ROC Curve")
axes[0].legend()
# PR Curve
precision_vals, recall_vals, _ = precision_recall_curve(y_test, y_prob)
axes[1].plot(recall_vals, precision_vals, color="#4CAF50", lw=2)
axes[1].set_xlabel("Recall")
axes[1].set_ylabel("Precision")
axes[1].set_title("Precision-Recall Curve")
plt.tight_layout()
plt.savefig("roc_pr_curves.png", dpi=150)
Output:
TEXT
# Executed successfully
(2) Choosing Metrics for Your Business Scenario
| Scenario | Priority Metric | Reason | Threshold Strategy |
|---|---|---|---|
| User churn prediction | Recall | Missing a churned customer is costly | Lower threshold (0.3) |
| Spam filtering | Precision | Mislabeling a legit email is unacceptable | Raise threshold (0.7) |
| Product recommendation | F1 | Balance precision and coverage | Default (0.5) |
| Fraud detection | Recall | Missing fraud is hugely costly | Lower threshold (0.2) |
📌 Key Point: In churn prediction, Recall matters more than Precision — missing one churned customer costs 2k USD, while a false alarm on a retained customer only wastes 50 USD in win-back costs. Tuning the decision threshold can optimize business value.
❓ FAQ
Q Why is logistic regression called "regression"?
A For historical reasons — it models with a regression method (a linear combination), but the output passes through a sigmoid to become a class probability. It's fundamentally a classification algorithm; the name is misleading.
Q What does class_weight=balanced mean?
A It automatically weights each class by the inverse of its frequency, giving the minority class a higher weight. With an 8% churn rate, the churned class weight = 1/0.08 = 12.5, and the retained class weight = 1/0.92 = 1.09.
Q Does the decision threshold have to be 0.5?
A Not at all. 0.5 is just the default, but business scenarios often call for tuning. For churn prediction you might lower it to 0.3 (boosting Recall); for spam you might raise it to 0.7 (boosting Precision). Let business costs drive the threshold.
Q Which is better, ROC-AUC or PR-AUC?
A Look at ROC-AUC when classes are balanced, and PR-AUC when they're heavily imbalanced. For churn prediction (8% vs 92%), PR-AUC is recommended.
Q Can logistic regression handle non-linear relationships?
A Not by default. But you can add polynomial features (
PolynomialFeatures) so it learns a non-linear decision boundary — essentially doing linear classification in a higher-dimensional space.Q What is L1 regularization good for in logistic regression?
A Same as in linear regression — L1 produces sparse solutions and performs automatic feature selection. Use
LogisticRegression(penalty="l1", solver="saga") to enable L1.📖 Summary
- Logistic regression uses the sigmoid to turn linear output into probabilities, with 0.5 as the default decision boundary
- Cross-entropy loss heavily penalizes predictions that are confident but wrong
- class_weight="balanced" handles class imbalance — essential for churn prediction
- Multiclass: One-vs-Rest (multiple binary classifiers) vs Softmax (unified probabilities); the latter calibrates probabilities better
- Choosing business metrics: churn → Recall, spam → Precision, recommendation → F1
- Tuning the decision threshold can optimize business value — don't be tied to the 0.5 default
📝 Exercises
- Basic (Difficulty ⭐): Use logistic regression for multiclass classification on the Iris dataset, and print the accuracy and classification_report. Hint:
LogisticRegression(max_iter=200)+classification_report. - Intermediate (Difficulty ⭐⭐): Simulate an imbalanced dataset (95:5) and compare the Recall of default logistic regression vs class_weight="balanced". Hint: use
make_classification(weights=[0.95, 0.05]). - Challenge (Difficulty ⭐⭐⭐): Implement Alice's full churn prediction pipeline — generate simulated SaaS user data, train a logistic regression model, plot the ROC/PR curves, compute Recall/Precision/F1 at different thresholds, and find the business-optimal threshold. Hint:
roc_curve+precision_recall_curve+ iterate over thresholds.
← Previous: Linear Regression | Next: Decision Trees and Random Forests →
