Getting Started with MLOps and MLflow
ML without experiment management is like code without Git—nobody knows which version is the best.
1. What You'll Learn
- MLOps concepts: DevOps extended to ML, managing the full ML lifecycle
- MLflow's four core components: Tracking / Projects / Models / Model Registry
- Experiment tracking in practice: logging parameters, metrics, models, and charts; comparing model versions
- Model Registry: managing model stages (Staging → Production), version rollback
- Bob's experiment management: how to systematically manage 50+ experiments for SalesPredict
2. A Real Story from an ML Team Lead
(1) The Pain: 50 Experiments Later, No Idea Which Model Is Best
Bob ran 50+ experiments in three weeks—different algorithms, different features, different hyperparameters. The parameters and results for each experiment were scattered across Jupyter Notebooks, Slack messages, and sticky notes. When the product manager asked, "Which model is the best right now?" Bob spent two hours digging through records and still wasn't sure. Experiment chaos is the norm in ML projects.
(2) The MLflow Solution
MLflow automatically logs the parameters, metrics, models, and charts for every experiment. All experiments are centralized, comparable, and traceable.
PYTHON
import mlflow
mlflow.set_experiment("SalesPredict-Revenue")
with mlflow.start_run(run_name="xgboost_v1"):
mlflow.log_params({"n_estimators": 300, "max_depth": 6, "learning_rate": 0.1})
mlflow.log_metrics({"r2": 0.89, "mae": 8.5})
mlflow.sklearn.log_model(model, "model")
(3) The Payoff: Find the Best Experiment in 2 Seconds, 3x Experiment Efficiency
After adopting MLflow, Bob can find the best historical model in 2 seconds. New experiments are automatically compared against past runs, and team collaboration efficiency tripled.
3. MLOps Concepts
(1) ML Lifecycle Management
MLOps is DevOps extended to the ML domain—it cares not only about code versions, but also data versions, experiment versions, and model versions.
graph TB
DATA[Data Engineering<br/>Version & Pipeline] --> EXP[Experiment<br/>Track & Compare]
EXP --> REG[Model Registry<br/>Version & Stage]
REG --> DEPLOY[Deployment<br/>Serve & Scale]
DEPLOY --> MON[Monitoring<br/>Drift & Performance]
MON -->|Retrain Trigger| DATA
(2) How ML Differs from Traditional Software Engineering
| Dimension | Software Engineering | ML Engineering |
|---|---|---|
| Version control | Code | Code + data + models |
| Testing | Unit tests | Data tests + model tests |
| Deployment | Deploy once | Continuous retraining |
| Degradation cause | Bugs | Data drift / concept drift |
| Reproducibility | Code = result | Code + data + params = result |
4. MLflow's Four Core Components
(1) MLflow Tracking
▶ Example: Logging Experiment Parameters and Metrics
PYTHON
import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import r2_score, mean_absolute_error
from sklearn.model_selection import train_test_split
import numpy as np
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("SalesPredict-Revenue")
rng = np.random.default_rng(42)
n = 1000
X = rng.uniform(0, 100, (n, 5))
y = 50 + 0.8 * X[:, 0] + 1.2 * X[:, 1] + rng.normal(0, 5, n)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Experiment 1: Random Forest
with mlflow.start_run(run_name="rf_baseline"):
params = {"n_estimators": 100, "max_depth": 10}
mlflow.log_params(params)
model = RandomForestRegressor(**params, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
r2 = r2_score(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
mlflow.log_metrics({"r2": r2, "mae": mae})
mlflow.sklearn.log_model(model, "model")
print(f"RF: R²={r2:.4f}, MAE={mae:.2f}")
# Experiment 2: XGBoost
import xgboost as xgb
with mlflow.start_run(run_name="xgboost_v1"):
params = {"n_estimators": 300, "max_depth": 6, "learning_rate": 0.1}
mlflow.log_params(params)
model = xgb.XGBRegressor(**params, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
r2 = r2_score(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
mlflow.log_metrics({"r2": r2, "mae": mae})
mlflow.xgboost.log_model(model, "model")
print(f"XGB: R²={r2:.4f}, MAE={mae:.2f}")
Output:
TEXT
# Execution successful
▶ Example: Querying and Comparing Experiments
PYTHON
import mlflow
# Query experiments
experiment = mlflow.get_experiment_by_name("SalesPredict-Revenue")
runs = mlflow.search_runs(experiment_ids=[experiment.experiment_id])
# Sort by R² (best first)
best_runs = runs.sort_values("metrics.r2", ascending=False)
print("Top 3 runs by R²:")
print(best_runs[["run_name", "metrics.r2", "metrics.mae",
"params.n_estimators", "params.max_depth"]].head(3))
Output:
TEXT
Top 3 runs by R²:
(2) MLflow Models and Registry
▶ Example: Model Registration and Stage Management
PYTHON
import mlflow
# Register a model
model_uri = f"runs:/{run_id}/model"
mlflow.register_model(model_uri, "SalesPredict-Revenue-Model")
# Transition model stage
client = mlflow.tracking.MlflowClient()
client.transition_model_version_stage(
name="SalesPredict-Revenue-Model",
version=1,
stage="Staging",
)
# Promote to Production
client.transition_model_version_stage(
name="SalesPredict-Revenue-Model",
version=1,
stage="Production",
)
# Load production model
model = mlflow.sklearn.load_model("models:/SalesPredict-Revenue-Model/Production")
Output:
TEXT
# Execution successful
| Registry Stage | Meaning | Typical Action |
|---|---|---|
| None | Just registered | Automated testing |
| Staging | Pending validation | A/B testing |
| Production | Live in production | Online serving |
| Archived | Retired | Record keeping |
5. MLflow Projects and Automatic Logging
▶ Example: Automatic Logging with autolog
PYTHON
import mlflow
import mlflow.sklearn
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
import numpy as np
# Auto-log: automatically record all parameters, metrics, and model
mlflow.sklearn.autolog()
rng = np.random.default_rng(42)
X = rng.uniform(0, 100, (1000, 5))
y = 50 + 0.8 * X[:, 0] + rng.normal(0, 5, n)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
with mlflow.start_run(run_name="gb_autolog"):
model = GradientBoostingRegressor(n_estimators=100, max_depth=5, learning_rate=0.1, random_state=42)
model.fit(X_train, y_train)
# autolog captures everything automatically!
Output:
TEXT
# Execution successful
| autolog Framework | What Gets Logged Automatically |
|---|---|
| sklearn | Parameters, metrics, model, confusion matrix |
| xgboost | Parameters, metrics, feature importance |
| lightgbm | Parameters, metrics, feature importance |
| pytorch | Parameters, metrics, model |
▶ Example: Logging Custom Charts
PYTHON
import mlflow
import matplotlib.pyplot as plt
import numpy as np
with mlflow.start_run(run_name="feature_analysis"):
# Create and log a plot
fig, ax = plt.subplots()
ax.barh(["feat_1", "feat_2", "feat_3", "feat_4", "feat_5"],
[0.35, 0.25, 0.15, 0.10, 0.05])
ax.set_xlabel("Importance")
ax.set_title("Feature Importance")
plt.tight_layout()
mlflow.log_figure(fig, "feature_importance.png")
plt.close()
# Log artifacts (any file)
# mlflow.log_artifact("local_file.csv")
Output:
TEXT
# Execution successful
6. Bob's 50+ Experiment Management in Practice
▶ Example: Batch Experiment Tracking Framework
PYTHON
import mlflow
import mlflow.sklearn
import xgboost as xgb
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import cross_val_score
from sklearn.metrics import r2_score, mean_absolute_error
import numpy as np
mlflow.set_experiment("SalesPredict-Benchmark")
# Simulated data
rng = np.random.default_rng(42)
X = rng.uniform(0, 100, (2000, 10))
y = 50 + X[:, :5] @ [0.8, 1.2, -0.5, 0.3, 0.1] + rng.normal(0, 5, 2000)
# Systematic experiment tracking
experiments = [
{"name": "linear_baseline", "model": LinearRegression(), "params": {}},
{"name": "ridge_alpha1", "model": Ridge(alpha=1), "params": {"alpha": 1}},
{"name": "rf_100", "model": RandomForestRegressor(n_estimators=100, random_state=42),
"params": {"n_estimators": 100}},
{"name": "rf_300", "model": RandomForestRegressor(n_estimators=300, random_state=42),
"params": {"n_estimators": 300}},
{"name": "xgb_d6_lr01", "model": xgb.XGBRegressor(max_depth=6, learning_rate=0.1, random_state=42),
"params": {"max_depth": 6, "learning_rate": 0.1}},
]
for exp in experiments:
with mlflow.start_run(run_name=exp["name"]):
mlflow.log_params(exp["params"])
cv_scores = cross_val_score(exp["model"], X, y, cv=5, scoring="r2")
mlflow.log_metrics({
"cv_r2_mean": cv_scores.mean(),
"cv_r2_std": cv_scores.std(),
})
# Find best experiment
runs = mlflow.search_runs(
experiment_ids=[mlflow.get_experiment_by_name("SalesPredict-Benchmark").experiment_id],
order_by=["metrics.cv_r2_mean DESC"],
)
print("Best experiments:")
print(runs[["run_name", "metrics.cv_r2_mean", "metrics.cv_r2_std"]].head(5))
Output:
TEXT
Best experiments:
❓ FAQ
Q Should I use MLflow or Weights & Biases (W&B)?
A MLflow is open-source, free, self-hostable, and feature-complete. W&B offers better visualizations and easier team collaboration, but is commercially licensed. For individuals or small teams, go with MLflow; for large teams, W&B is a great fit.
Q How do I deploy an MLflow Tracking Server?
A For local development, use
mlflow server --backend-store-uri sqlite:///mlflow.db. For production, use PostgreSQL + object storage (S3/GCS) + a reverse proxy.Q Can I mix autolog with manual logging?
A Yes. autolog captures standard information, while manual logging supplements it with custom content (e.g., business metrics, charts). Recommended approach: rely on autolog as the baseline and add manual logs for extras.
Q Is Model Registry really necessary?
A Absolutely. Without a Registry, model files are scattered everywhere, and you have no idea which one is in production or which is outdated. The Registry provides version management, stage management, and rollback capabilities.
Q How do I organize a large number of experiments?
A Use experiments for grouping (e.g., SalesPredict-Revenue / SalesPredict-Churn), tags for labeling (e.g., team=bob, priority=high), and run_name for distinction (e.g., xgboost_v3_tuned).
Q How do I ensure experiment reproducibility?
A MLflow records parameters + code version + data version + random seed. Use
mlflow.sklearn.load_model(run_id) to restore the complete model. Pair with DVC for data version management.📖 Summary
- MLOps = DevOps for ML: managing the full lifecycle of data versions, experiment versions, and model versions
- MLflow's four core components: Tracking (logging) + Projects (packaging) + Models (format) + Registry (registration)
- Tracking essentials: log_params + log_metrics + log_model—every experiment logged automatically
- Model Registry manages model stages: None → Staging → Production → Archived
- autolog automatically captures parameters, metrics, and models, reducing manual logging overhead
- Reproducibility = recording parameters + data + random seed + code version in full
📝 Exercises
- Basic (Difficulty ⭐): Install MLflow, start a local Tracking Server, and log one experiment with a parameter (lr=0.01) and a metric (r2=0.85). Hint:
pip install mlflow+mlflow.start_run(). - Intermediate (Difficulty ⭐⭐): Use autolog to record the training process of sklearn's RandomForestRegressor, then query all runs and sort by R². Hint:
mlflow.sklearn.autolog()+search_runs(). - Challenge (Difficulty ⭐⭐⭐): Implement Bob's experiment management framework—run 5 models on the SalesPredict data, automatically log parameters/metrics/models to MLflow for each run, register the best model to the Registry, and load the Production model for prediction. Hint: refer to the batch experiment example in Section 6.
← Previous: Model Evaluation and Tuning | Next: Model Deployment →
