Monitoring — Prometheus + Grafana Full-Stack Observability
Monitoring is like a car's dashboard—the speedometer (QPS), oil temperature (latency), and check engine light (error rate) keep you informed of the car's (system's) health at all times. Without a dashboard, you wouldn't know something was wrong until smoke started coming out of the engine.
1. What You'll Learn
- Prometheus Metrics Exposure:
prometheus-fastapi-instrumentatorIntegration - Custom business metrics: price update rate, API latency percentile, number of active users
- Grafana Dashboard Design: Request Volume/Latency/Error Rate (RED Method) + Business Metrics
- Alert Rules: P99 latency exceeds threshold, error rate spikes, Redis cache hit rate drops
- Alice Scenario: Charlie's PriceTracker Monitoring Dashboard
2. Alice's True Story
(1) Pain Point: You Only Realize There's a Problem Once It Happens
At 3 a.m., PriceTracker's database connection pool ran out, and the API began returning 500 errors. Alice didn't find out until 8 a.m., when users started complaining. Without real-time monitoring, Charlie could only review the logs after the fact and discovered that the connection pool had been triggering alerts as early as 5 a.m.—if monitoring had been in place, the issue could have been detected and resolved within five minutes.
(2) Solution Using Prometheus and Grafana
Prometheus collects metrics (number of requests, latency, error rate) every second; Grafana visualizes them in real time; and AlertManager automatically sends alerts when metrics deviate from normal ranges—shifting the process from "learning about issues only after user complaints" to "automatic notifications within 5 minutes."
(3) Revenue
P99 latency spikes trigger an alert within one minute; automatic notifications are sent before the database connection pool is exhausted; issue detection time has been reduced from "hours" to "minutes," so Charlie no longer has to stare at the screen for 24 hours straight.
3. Exposing Prometheus Metrics
(1) Monitoring Architecture
flowchart LR
App[FastAPI App] -->|/metrics| Prom[Prometheus]
Prom -->|Query| Grafana[Grafana Dashboard]
Prom -->|Alert| AlertMgr[AlertManager]
AlertMgr -->|Notify| Slack[Slack / Email]
Grafana -->|Visualize| Charlie[Charlie DevOps]
(1) ▶ Example: prometheus-fastapi-instrumentator integration
PYTHON
# Install: uv add prometheus-fastapi-instrumentator
from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
app = FastAPI()
# Add default metrics: request count, duration, size, exceptions
Instrumentator().instrument(app).expose(app)
# Now /metrics endpoint exposes Prometheus metrics
# Including: http_requests_total, http_request_duration_seconds, etc.
Output:
TEXT
# Execution Successful
(2) ▶ Example: Custom Metrics
PYTHON
from prometheus_client import Counter, Histogram, Gauge
from fastapi import FastAPI
app = FastAPI()
# Custom business metrics
PRICE_UPDATES = Counter(
"pricetracker_price_updates_total",
"Total number of price updates",
["category", "currency"],
)
REQUEST_LATENCY = Histogram(
"pricetracker_request_latency_seconds",
"Request latency in seconds",
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
)
ACTIVE_USERS = Gauge(
"pricetracker_active_users",
"Number of active users in last 5 minutes",
)
CACHE_HIT_RATE = Gauge(
"pricetracker_cache_hit_rate",
"Redis cache hit rate percentage",
)
@app.post("/api/v1/prices")
async def create_price(price: PriceCreate):
result = await service.create_price(price)
# Record custom metric
PRICE_UPDATES.labels(category="electronics", currency="USD").inc()
return result
Output:
TEXT
# Function defined successfully
(2) Key Metrics of the RED Method
| Metric | Type | Description | PromQL |
|---|---|---|---|
| Rate | Counter | Request Rate (QPS) | rate(http_requests_total[5m]) |
| Errors | Counter | Error Rate | rate(http_requests_total{status=~"5.."}[5m]) |
| Duration | Histogram | Delay Distribution | histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) |
4. Grafana Dashboard Design
(1) Dashboard Layout
graph TD
Title[PriceTracker Monitoring Dashboard]
Title --> Row1[Row 1: Overview]
Row1 --> QPS[QPS - Rate]
Row1 --> P50[P50 Latency]
Row1 --> P99[P99 Latency]
Row1 --> ErrRate[Error Rate]
Title --> Row2[Row 2: Business]
Row2 --> PriceUpd[Price Updates/min]
Row2 --> ActiveUsers[Active Users]
Row2 --> CacheHit[Cache Hit Rate]
Title --> Row3[Row 3: Infrastructure]
Row3 --> DBConns[DB Connections]
Row3 --> RedisConns[Redis Connections]
Row3 --> CeleryQ[Celery Queue Size]
(1) ▶ Example: Grafana Dashboard JSON Configuration (Excerpt)
JSON
{
"dashboard": {
"title": "PriceTracker Monitoring",
"panels": [
{
"title": "Request Rate (QPS)",
"type": "timeseries",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m]))",
"legendFormat": "Total QPS"
}
]
},
{
"title": "P99 Latency",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
"legendFormat": "P99"
}
],
"thresholds": {
"steps": [
{ "value": 0, "color": "green" },
{ "value": 0.1, "color": "yellow" },
{ "value": 0.5, "color": "red" }
]
}
},
{
"title": "Cache Hit Rate",
"type": "gauge",
"targets": [
{
"expr": "pricetracker_cache_hit_rate",
"legendFormat": "Hit Rate %"
}
]
}
]
}
}
Output:
JSON
{
"dashboard": {
"title": "PriceTracker Monitoring",
"panels": [
{
"title": "Request Rate (QPS)",
"type": "timeseries",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m]))",
"legendFormat": "Total QPS"
}
]
},
{
"title": "P99 Latency",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_buck
5. Alert Rules
(1) ▶ Example: Prometheus Alert Rules
YAML
# prometheus/alert_rules.yml
groups:
- name: pricetracker_alerts
rules:
- alert: HighP99Latency
expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5
for: 2m
labels:
severity: warning
annotations:
summary: "P99 latency exceeds 500ms"
description: "P99 latency is {{ $value }}s, threshold is 0.5s"
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 1m
labels:
severity: critical
annotations:
summary: "Error rate exceeds 5%"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: LowCacheHitRate
expr: pricetracker_cache_hit_rate < 60
for: 5m
labels:
severity: warning
annotations:
summary: "Cache hit rate below 60%"
description: "Current hit rate is {{ $value }}%"
- alert: DatabaseConnectionPoolExhausted
expr: pricetracker_db_connections_in_use / pricetracker_db_connections_max > 0.9
for: 3m
labels:
severity: critical
annotations:
summary: "Database connection pool > 90% utilized"
Output:
TEXT
Monitoring config loaded
Prometheus targets: 3 active
Grafana dashboard: ready
(2) ▶ Example: Adding a Monitoring Stack to Docker Compose
YAML
# Add to docker-compose.yml
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./docker/prometheus.yml:/etc/prometheus/prometheus.yml
- ./docker/alert_rules.yml:/etc/prometheus/alert_rules.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--alert.rule-files=/etc/prometheus/alert_rules.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
alertmanager:
image: prom/alertmanager:latest
ports:
- "9093:9093"
volumes:
- ./docker/alertmanager.yml:/etc/alertmanager/alertmanager.yml
Output:
TEXT
CONTAINER ID IMAGE STATUS PORTS
abc123 nginx:latest Up 2 hours 0.0.0.0:80->80/tcp
(3) ▶ Example: prometheus.yml configuration
YAML
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "pricetracker-api"
metrics_path: "/metrics"
static_configs:
- targets: ["api:8000"]
rule_files:
- "alert_rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
Output:
TEXT
The configuration has taken effect.
❓ FAQ
Q What is the difference between Prometheus and ELK?
A Prometheus is for metrics monitoring (numeric time series), while ELK is for log analysis (text search). The two are complementary—Prometheus is used for monitoring trends and generating alerts, while ELK is used for reviewing detailed logs to troubleshoot issues.
Q What should the metric collection interval be set to?
A The default of 15 seconds strikes a good balance. A shorter interval (5 seconds) provides more real-time data but incurs higher storage costs, while a longer interval (60 seconds) saves on storage but may miss brief fluctuations.
Q Will having too many custom metrics slow down the application?
A Each metric incurs a small CPU overhead. We recommend using fewer than 100 custom metrics. The cardinality of labels (the number of distinct values) has a greater impact on performance than the number of metrics; avoid labels with high cardinality (such as
user_id).Q How do I share a Grafana dashboard?
A Export it as a JSON file, and your team can import it. Grafana.com offers a wide variety of ready-made templates for reference.
Q What should I do if there are too many alerts (alert fatigue)?
A Set a reasonable "for" duration (to avoid triggers caused by momentary fluctuations), classify alerts as "warning" or "critical," group related alerts, and reduce noise in notification channels.
Q How do I monitor Celery tasks?
A Flower provides task-level monitoring and exposes Prometheus metrics. Key metrics include task success rate, queue length, and worker memory usage.
📖 Summary
prometheus-fastapi-instrumentatorOne line of code exposes default HTTP metrics- Customize three types of business metrics: Counter (count), Histogram (delay distribution), and Gauge (current value)
- Grafana Dashboard organized using the RED method: Rate (QPS), Errors (error rate), Duration (latency)
- AlertManager alert rule configuration: P99 > 500 ms, error rate > 5%, cache hit rate < 60%
- One-Click Deployment of a Complete Monitoring Stack with Docker Compose: Prometheus + Grafana + AlertManager
📝 Exercises
- Basic Exercise (Difficulty: ⭐): Add
prometheus-fastapi-instrumentatorto FastAPI and verify that the/metricsendpoint returns the default HTTP metrics (such as http_requests_total). Hint:Instrumentator().instrument(app).expose(app) - Advanced Exercise (Difficulty ⭐⭐): Add three custom metrics—PRICE_UPDATES (Counter by category), REQUEST_LATENCY (Histogram with percentile buckets), and CACHE_HIT_RATE (Gauge)—and log the metric values in the endpoint. Hint:
Counter(..., ["category"])+.labels(category="electronics").inc() - Challenge (Difficulty: ⭐⭐⭐): Add Prometheus, Grafana, and AlertManager to Docker Compose; configure
prometheus.ymlto collect FastAPI metrics; write two alert rules—one for P99 > 500 ms and another for error rate > 5%; and import the dashboard into Grafana to view real-time metrics. Hint:docker/prometheus.yml+alert_rules.yml+ Grafana data source configuration
---|
