Feature Engineering
Data determines the ceiling; the model merely approaches it — feature engineering is what raises that ceiling.
1. What You'll Learn
- Numeric features: standardization/normalization, log transform, binning, polynomial features
- Categorical features: One-Hot/Ordinal/Target Encoding, handling high cardinality
- Time feature extraction: year/month/week/hour, holiday flags, days since an event
- Feature selection: Filter (correlation/mutual information), Wrapper (RFE), Embedded (L1 regularization/tree importance)
- Bob's e-commerce feature factory: building business features like "purchase frequency in the last 7 days" and "category preference entropy"
2. A Real-World Story from an ML Engineer
(1) The Pain Point: 3 Days of Tuning for 1% Gain vs. One Feature for 10%
Bob spent 3 days tuning XGBoost hyperparameters, pushing R² from 0.78 to 0.79 — a mere 1% improvement. But when he added a single new feature, "purchase frequency in the last 7 days," R² jumped straight to 0.86 — an 8% gain. Feature engineering delivers far greater returns than hyperparameter tuning, yet most engineers lack a systematic methodology for it.
(2) A Systematic Approach to Feature Engineering
Feature engineering is not random experimentation — it follows a systematic workflow: understand the business → construct features → select features → validate effectiveness.
PYTHON
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
# Systematic feature engineering pipeline
num_features = ["ad_spend", "traffic", "log_monetary", "purchase_freq_7d"]
cat_features = ["category", "tier"]
preprocessor = ColumnTransformer([
("num", StandardScaler(), num_features),
("cat", OneHotEncoder(drop="first", handle_unknown="ignore"), cat_features),
])
pipe = Pipeline([("prep", preprocessor), ("model", XGBRegressor())])
(3) The Result: Systematic Feature Engineering Lifts R² from 0.78 to 0.88
After Bob built his feature factory, every new feature addition was backed by evidence. R² climbed systematically from 0.78 to 0.88 — far surpassing what tuning alone could achieve.
3. Numeric Feature Transforms
(1) Standardization vs. Normalization
▶ Example: StandardScaler vs. MinMaxScaler Comparison
PYTHON
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler
import numpy as np
data = np.array([[1], [5], [10], [50], [100], [500], [1000]])
scalers = {
"Original": data,
"StandardScaler": StandardScaler().fit_transform(data),
"MinMaxScaler": MinMaxScaler().fit_transform(data),
"RobustScaler": RobustScaler().fit_transform(data),
}
for name, scaled in scalers.items():
print(f"{name:16s}: min={scaled.min():8.3f}, max={scaled.max():8.3f}, "
f"mean={scaled.mean():8.3f}, std={scaled.std():8.3f}")
Output:
TEXT
# Executed successfully
| Method | Formula | Output Range | Outlier Robustness | Use Case |
|---|---|---|---|---|
| StandardScaler | (x-μ)/σ | Unbounded | Poor | SVM/Logistic Regression/Neural Networks |
| MinMaxScaler | (x-min)/(max-min) | [0,1] | Poor | Neural network inputs |
| RobustScaler | (x-median)/IQR | Unbounded | Strong | Data with outliers |
| Log Transform | log(1+x) | Unbounded | Strong | Right-skewed distributions |
(2) Log Transform and Binning
▶ Example: Handling Right-Skewed Distributions with Binning
PYTHON
import numpy as np
from sklearn.preprocessing import KBinsDiscretizer
import matplotlib.pyplot as plt
rng = np.random.default_rng(42)
income = rng.exponential(50000, 1000) # Right-skewed income data
fig, axes = plt.subplots(1, 3, figsize=(15, 4))
# (1) Original distribution
axes[0].hist(income, bins=30, color="#2196F3", edgecolor="white")
axes[0].set_title("Original (Right-Skewed)")
# (2) Log-transformed
log_income = np.log1p(income)
axes[1].hist(log_income, bins=30, color="#4CAF50", edgecolor="white")
axes[1].set_title("Log-Transformed (More Symmetric)")
# (3) Binned
binner = KBinsDiscretizer(n_bins=5, encode="ordinal", strategy="quantile")
binned = binner.fit_transform(income.reshape(-1, 1))
axes[2].hist(binned, bins=5, color="#FF9800", edgecolor="white")
axes[2].set_title("Quantile Binned (5 Bins)")
plt.tight_layout()
plt.savefig("feature_transform.png", dpi=150)
Output:
TEXT
# Executed successfully
(3) Polynomial Features
▶ Example: Capturing Non-Linear Relationships
PYTHON
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
import numpy as np
rng = np.random.default_rng(42)
X = rng.uniform(-3, 3, (200, 1))
y = 2 * X.squeeze() ** 2 - 3 * X.squeeze() + 1 + rng.normal(0, 1, 200)
for degree in [1, 2, 3, 5]:
pipe = Pipeline([
("poly", PolynomialFeatures(degree=degree, include_bias=False)),
("model", LinearRegression()),
])
scores = cross_val_score(pipe, X, y, cv=5, scoring="r2")
print(f"Degree={degree}: R²={scores.mean():.3f} +/- {scores.std():.3f}")
Output:
TEXT
# Executed successfully
4. Categorical Feature Encoding
(1) Encoding Method Comparison
▶ Example: Different Encoding Strategies
PYTHON
import pandas as pd
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
import numpy as np
df = pd.DataFrame({
"category": ["Electronics", "Clothing", "Food", "Electronics", "Food"],
"tier": ["Low", "Medium", "High", "Medium", "Low"],
"city": ["New York", "London", "Paris", "Tokyo", "Berlin"],
})
# One-Hot Encoding (no ordinal assumption)
ohe = OneHotEncoder(drop="first", sparse_output=False)
ohe_result = ohe.fit_transform(df[["category"]])
print(f"One-Hot category:\n{ohe_result}")
print(f"Feature names: {ohe.get_feature_names_out()}")
# Ordinal Encoding (has order)
ord_enc = OrdinalEncoder(categories=[["Low", "Medium", "High"]])
ord_result = ord_enc.fit_transform(df[["tier"]])
print(f"\nOrdinal tier: {ord_result.flatten()}")
# Target Encoding (encode by target mean per category)
target_means = df.groupby("city")["some_target"].transform("mean")
Output:
TEXT
# Executed successfully
| Encoding | Use Case | Dimension Increase | Ordinality | High-Cardinality Risk |
|---|---|---|---|---|
| One-Hot | Nominal categories (<10 classes) | K-1 | None | Dimension explosion |
| Ordinal | Ordinal categories (ranks) | 0 | Preserved | None |
| Target | High cardinality (cities/products) | 0 | None | Overfitting (requires CV) |
| Frequency | High cardinality | 0 | None | Information loss |
| Embedding | Very high cardinality (deep learning) | Controllable | Learned | Requires training |
(2) Handling High Cardinality
▶ Example: Target Encoding for City Features
PYTHON
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 1000
cities = rng.choice([f"City_{i}" for i in range(50)], n)
df = pd.DataFrame({
"city": cities,
"income_k": rng.exponential(50, n),
"age": rng.integers(18, 70, n),
"purchased": rng.choice([0, 1], n, p=[0.7, 0.3]),
})
# Target encoding with CV to avoid leakage
from category_encoders import TargetEncoder
pipe = Pipeline([
("target_enc", TargetEncoder(cols=["city"])),
("scaler", StandardScaler()),
("model", LogisticRegression(max_iter=500)),
])
X = df[["city", "income_k", "age"]]
y = df["purchased"]
scores = cross_val_score(pipe, X, y, cv=5, scoring="accuracy")
print(f"Target Encoding CV Accuracy: {scores.mean():.3f}")
Output:
TEXT
# Executed successfully
5. Time Feature Extraction
▶ Example: E-Commerce Time Feature Engineering
PYTHON
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 1000
df = pd.DataFrame({
"order_date": pd.date_range("2023-01-01", periods=n, freq="6H"),
"user_register_date": pd.to_datetime(rng.choice(pd.date_range("2020-01-01", "2023-01-01"), n)),
})
# Extract time-based features
df["order_year"] = df["order_date"].dt.year
df["order_month"] = df["order_date"].dt.month
df["order_day"] = df["order_date"].dt.day
df["order_hour"] = df["order_date"].dt.hour
df["day_of_week"] = df["order_date"].dt.dayofweek # 0=Mon, 6=Sun
df["is_weekend"] = (df["day_of_week"] >= 5).astype(int)
df["is_night"] = ((df["order_hour"] >= 22) | (df["order_hour"] <= 6)).astype(int)
# Quarter and season
df["quarter"] = df["order_date"].dt.quarter
# Time since registration
df["days_since_register"] = (df["order_date"] - df["user_register_date"]).dt.days
# Is holiday (simplified - mark Dec 20-31 as holiday season)
df["is_holiday_season"] = ((df["order_month"] == 12) & (df["order_day"] >= 20)).astype(int)
# Days to next month end (urgency feature)
df["days_to_month_end"] = (df["order_date"] + pd.offsets.MonthEnd(0) - df["order_date"]).dt.days
print(f"Time features: {[c for c in df.columns if c not in ['order_date', 'user_register_date']]}")
print(df.head(3))
Output:
TEXT
# Executed successfully
| Feature Type | Examples | Business Meaning |
|---|---|---|
| Cyclical | hour, day_of_week, month | Peak purchase periods |
| Indicator | is_weekend, is_holiday | Spending patterns during special periods |
| Elapsed time | days_since_register | User lifecycle stage |
| Urgency | days_to_month_end | Impulse buying signals |
6. Feature Selection
(1) Three Major Approaches
mindmap
root((Feature Selection))
Filter
Correlation Coefficient
Mutual Information
Variance Threshold
Chi-Square Test
Wrapper
Recursive Feature Elimination RFE
Sequential Feature Selection
Cross-Validation Based
Embedded
L1 Regularization Lasso
Tree Feature Importance
XGBoost Importance
▶ Example: Comparing Three Feature Selection Methods
PYTHON
from sklearn.feature_selection import SelectKBest, f_regression, mutual_info_regression, RFE
from sklearn.linear_model import Lasso, LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import numpy as np
rng = np.random.default_rng(42)
n, p = 500, 20
X = rng.standard_normal((n, p))
true_coefs = np.zeros(p)
true_coefs[:5] = [3, -2, 1.5, 0.8, -0.5]
y = X @ true_coefs + rng.normal(0, 1, n)
feature_names = [f"feat_{i}" for i in range(p)]
# Method 1: Filter (F-regression)
selector_filter = SelectKBest(f_regression, k=5)
selector_filter.fit(X, y)
filter_selected = np.array(feature_names)[selector_filter.get_support()]
print(f"Filter top 5: {filter_selected}")
# Method 2: Wrapper (RFE)
rfe = RFE(LinearRegression(), n_features_to_select=5)
rfe.fit(X, y)
rfe_selected = np.array(feature_names)[rfe.get_support()]
print(f"RFE top 5: {rfe_selected}")
# Method 3: Embedded (Lasso)
lasso = Lasso(alpha=0.1)
lasso.fit(StandardScaler().fit_transform(X), y)
lasso_selected = np.array(feature_names)[np.abs(lasso.coef_) > 0.01]
print(f"Lasso top 5: {lasso_selected}")
print(f"True features: {feature_names[:5]}")
Output:
TEXT
# Executed successfully
| Method | Speed | Feature Interactions | Overfitting Risk | Use Case |
|---|---|---|---|---|
| Filter | Fast | Not considered | Low | Initial screening, high-dimensional data |
| Wrapper | Slow | Considered | High | Fine-grained selection |
| Embedded | Medium | Partially considered | Low | Practical first choice |
▶ Example: Bob's E-Commerce Feature Factory
PYTHON
import pandas as pd
import numpy as np
rng = np.random.default_rng(42)
n = 5000
# Raw transaction data
transactions = pd.DataFrame({
"user_id": rng.choice(range(1000), n),
"order_date": pd.date_range("2023-01-01", periods=n, freq="3H"),
"amount_usd": rng.exponential(100, n).clip(1, 5000),
"category": rng.choice(["Elec", "Cloth", "Food", "Book", "Home"], n),
})
# Feature factory: aggregate transactions into user features
user_features = transactions.groupby("user_id").agg(
total_orders=("amount_usd", "count"),
total_spending_usd=("amount_usd", "sum"),
avg_order_value=("amount_usd", "mean"),
std_order_value=("amount_usd", "std"),
last_order_date=("order_date", "max"),
first_order_date=("order_date", "min"),
favorite_category=("category", lambda x: x.mode().iloc[0] if len(x.mode()) > 0 else "Unknown"),
category_diversity=("category", "nunique"),
).reset_index()
# Derived business features
user_features["recency_days"] = (pd.Timestamp("2024-01-01") - user_features["last_order_date"]).dt.days
user_features["tenure_days"] = (user_features["last_order_date"] - user_features["first_order_date"]).dt.days
user_features["purchase_frequency"] = user_features["total_orders"] / (user_features["tenure_days"] + 1) * 30
# Category preference entropy (how diverse is user's category preference)
from scipy.stats import entropy
def calc_entropy(series):
probs = series.value_counts(normalize=True)
return entropy(probs)
cat_entropy = transactions.groupby("user_id")["category"].apply(calc_entropy).reset_index()
cat_entropy.columns = ["user_id", "category_entropy"]
user_features = user_features.merge(cat_entropy, on="user_id")
# Price sensitivity (std/mean of order values)
user_features["price_sensitivity"] = user_features["std_order_value"] / (user_features["avg_order_value"] + 1)
print(f"User features shape: {user_features.shape}")
print(f"Feature columns: {list(user_features.columns)}")
print(user_features.head(3))
Output:
TEXT
# Function defined successfully
❓ FAQ
Q Should feature engineering be done before or after the train/test split?
A Operations involving target information (e.g., Target Encoding) must be done after the split, fitting only on the training set. Operations that don't use the target (e.g., log transform, extracting month) can be done before the split. A Pipeline handles this ordering automatically.
Q How many features is too many?
A There's no absolute standard. A common rule of thumb: the number of samples should be at least 10x the number of features (n > 10p). When features exceed samples (high-dimensional, small-sample), you must apply feature selection or regularization.
Q How do you interpret model coefficients after a log transform?
A In a linear model with log(y), the coefficient means "a 1-unit increase in x changes y by approximately coef×100%." This is an elasticity interpretation, commonly used in economics.
Q What is the dummy variable trap after One-Hot encoding?
A Creating K dummy variables for K categories causes perfect multicollinearity (all columns sum to 1). You must set drop_first=True to remove one, keeping K-1 columns.
Q Which feature selection method should you prioritize?
A Start with Filter methods for fast screening (removing obviously irrelevant features), then use Embedded methods (Lasso/tree importance) for fine-grained selection. Wrapper methods are most accurate but slowest — reserve them for final optimization.
Q How do you determine whether a new feature is useful?
A Three approaches — 1) Check its univariate correlation with the target; 2) Add it to the model and see if validation metrics improve; 3) Use permutation importance to measure how much performance drops when that feature is randomly shuffled.
📖 Summary
- The numeric feature toolkit: standardization/normalization removes scale differences, log transform handles skewness, binning captures non-linearity
- Choose categorical encoding based on ordinality: nominal → One-Hot, ordinal → Ordinal, high cardinality → Target Encoding
- Time features are a goldmine for e-commerce ML: cyclical + indicator + elapsed-time + urgency — four feature categories
- Three paths for feature selection: Filter (fast screening), Wrapper (precise but slow), Embedded (practical first choice)
- Feature factory: aggregate raw transaction data into business features like RFM, category preference entropy, and price sensitivity
- Feature engineering determines the model's ceiling — time invested here pays off far more than hyperparameter tuning
📝 Exercises
- Basic (Difficulty ⭐): Take a dataset with skewed distributions and process it with StandardScaler alone vs. Log + StandardScaler. Compare the distribution statistics after each treatment. Hint:
np.log1p()+StandardScaler(). - Intermediate (Difficulty ⭐⭐): Build a complete feature engineering Pipeline — StandardScaler for numeric columns, OneHotEncoder for categorical columns, extract month/week/is_weekend from datetime columns, then use RFE to select the 5 most important features. Hint: ColumnTransformer + RFE.
- Challenge (Difficulty ⭐⭐⭐): Build Bob's e-commerce feature factory — starting from raw transaction data, construct at least 10 business features (including RFM, category preference entropy, price sensitivity, purchase frequency, etc.), use Lasso for feature selection, and compare model performance before and after selection. Hint: Refer to the feature factory example in Section 6.
← Previous: KNN and Clustering | Next: Ensemble Learning — XGBoost and LightGBM →
