FastAPI: 监控 — Prometheus + Grafana 全栈可观测
最后更新：2026-08-26
监控就像汽车仪表盘——速度表（QPS）、油温（延迟）、故障灯（错误率），随时告诉你车（系统）的健康状态。没有仪表盘，你只能等引擎冒烟才知道出问题了。
1. 你将学到
- Prometheus 指标暴露：
prometheus-fastapi-instrumentator集成 - 自定义业务指标：价格更新速率、API 延迟百分位、活跃用户数
- Grafana Dashboard 设计：请求量/延迟/错误率（RED 方法）+ 业务指标
- 告警规则：P99 延迟超阈值、错误率飙升、Redis 缓存命中率下降
- Alice 场景：Charlie 的 PriceTracker 监控大屏
2. Alice 的真实故事
(1) 痛点：出了问题才知道
PriceTracker 在凌晨 3 点数据库连接池耗尽，API 开始返回 500 错误，直到早上 8 点用户投诉 Alice 才知道。没有实时监控，Charlie 只能事后翻日志，发现连接池 5 点就开始报警了——如果有监控，5 分钟就能发现并修复。
(2) Prometheus + Grafana 的解法
Prometheus 每秒采集指标（请求数、延迟、错误率），Grafana 实时可视化，AlertManager 在指标异常时自动告警——问题从"用户投诉才知道"变成"5 分钟内自动通知"。
(3) 收益
P99 延迟飙升 1 分钟内触发告警，数据库连接池耗尽前自动通知，问题发现时间从"小时级"降到"分钟级"，Charlie 不用 24 小时盯屏幕。
3. Prometheus 指标暴露
(1) 监控架构
flowchart LR
App[FastAPI App] -->|/metrics| Prom[Prometheus]
Prom -->|Query| Grafana[Grafana Dashboard]
Prom -->|Alert| AlertMgr[AlertManager]
AlertMgr -->|Notify| Slack[Slack / Email]
Grafana -->|Visualize| Charlie[Charlie DevOps]
▶ 示例：prometheus-fastapi-instrumentator 集成
PYTHON
# Install: uv add prometheus-fastapi-instrumentator
from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
app = FastAPI()
# Add default metrics: request count, duration, size, exceptions
Instrumentator().instrument(app).expose(app)
# Now /metrics endpoint exposes Prometheus metrics
# Including: http_requests_total, http_request_duration_seconds, etc.
输出：
TEXT
📖 仅展示
# 执行成功
▶ 示例：自定义指标
PYTHON
from prometheus_client import Counter, Histogram, Gauge
from fastapi import FastAPI
app = FastAPI()
# Custom business metrics
PRICE_UPDATES = Counter(
"pricetracker_price_updates_total",
"Total number of price updates",
["category", "currency"],
)
REQUEST_LATENCY = Histogram(
"pricetracker_request_latency_seconds",
"Request latency in seconds",
buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
)
ACTIVE_USERS = Gauge(
"pricetracker_active_users",
"Number of active users in last 5 minutes",
)
CACHE_HIT_RATE = Gauge(
"pricetracker_cache_hit_rate",
"Redis cache hit rate percentage",
)
@app.post("/api/v1/prices")
async def create_price(price: PriceCreate):
result = await service.create_price(price)
# Record custom metric
PRICE_UPDATES.labels(category="electronics", currency="USD").inc()
return result
输出：
TEXT
📖 仅展示
# 函数定义成功
(2) RED 方法核心指标
| 指标 | 类型 | 说明 | PromQL |
|---|---|---|---|
| Rate | Counter | 请求速率（QPS） | rate(http_requests_total[5m]) |
| Errors | Counter | 错误率 | rate(http_requests_total{status=~"5.."}[5m]) |
| Duration | Histogram | 延迟分布 | histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) |
4. Grafana Dashboard 设计
(1) Dashboard 面板布局
graph TD
Title[PriceTracker Monitoring Dashboard]
Title --> Row1[Row 1: Overview]
Row1 --> QPS[QPS - Rate]
Row1 --> P50[P50 Latency]
Row1 --> P99[P99 Latency]
Row1 --> ErrRate[Error Rate]
Title --> Row2[Row 2: Business]
Row2 --> PriceUpd[Price Updates/min]
Row2 --> ActiveUsers[Active Users]
Row2 --> CacheHit[Cache Hit Rate]
Title --> Row3[Row 3: Infrastructure]
Row3 --> DBConns[DB Connections]
Row3 --> RedisConns[Redis Connections]
Row3 --> CeleryQ[Celery Queue Size]
▶ 示例：Grafana Dashboard JSON 配置（节选）
JSON
{
"dashboard": {
"title": "PriceTracker Monitoring",
"panels": [
{
"title": "Request Rate (QPS)",
"type": "timeseries",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m]))",
"legendFormat": "Total QPS"
}
]
},
{
"title": "P99 Latency",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
"legendFormat": "P99"
}
],
"thresholds": {
"steps": [
{ "value": 0, "color": "green" },
{ "value": 0.1, "color": "yellow" },
{ "value": 0.5, "color": "red" }
]
}
},
{
"title": "Cache Hit Rate",
"type": "gauge",
"targets": [
{
"expr": "pricetracker_cache_hit_rate",
"legendFormat": "Hit Rate %"
}
]
}
]
}
}
输出：
JSON
{
"dashboard": {
"title": "PriceTracker Monitoring",
"panels": [
{
"title": "Request Rate (QPS)",
"type": "timeseries",
"targets": [
{
"expr": "sum(rate(http_requests_total[5m]))",
"legendFormat": "Total QPS"
}
]
},
{
"title": "P99 Latency",
"type": "stat",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_buck
5. 告警规则
▶ 示例：Prometheus 告警规则
YAML
# prometheus/alert_rules.yml
groups:
- name: pricetracker_alerts
rules:
- alert: HighP99Latency
expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5
for: 2m
labels:
severity: warning
annotations:
summary: "P99 latency exceeds 500ms"
description: "P99 latency is {{ $value }}s, threshold is 0.5s"
- alert: HighErrorRate
expr: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
for: 1m
labels:
severity: critical
annotations:
summary: "Error rate exceeds 5%"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: LowCacheHitRate
expr: pricetracker_cache_hit_rate < 60
for: 5m
labels:
severity: warning
annotations:
summary: "Cache hit rate below 60%"
description: "Current hit rate is {{ $value }}%"
- alert: DatabaseConnectionPoolExhausted
expr: pricetracker_db_connections_in_use / pricetracker_db_connections_max > 0.9
for: 3m
labels:
severity: critical
annotations:
summary: "Database connection pool > 90% utilized"
输出：
TEXT
📖 仅展示
Monitoring config loaded
Prometheus targets: 3 active
Grafana dashboard: ready
▶ 示例：Docker Compose 添加监控栈
YAML
# Add to docker-compose.yml
prometheus:
image: prom/prometheus:latest
ports:
- "9090:9090"
volumes:
- ./docker/prometheus.yml:/etc/prometheus/prometheus.yml
- ./docker/alert_rules.yml:/etc/prometheus/alert_rules.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--alert.rule-files=/etc/prometheus/alert_rules.yml'
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
alertmanager:
image: prom/alertmanager:latest
ports:
- "9093:9093"
volumes:
- ./docker/alertmanager.yml:/etc/alertmanager/alertmanager.yml
输出：
TEXT
📖 仅展示
CONTAINER ID IMAGE STATUS PORTS
abc123 nginx:latest Up 2 hours 0.0.0.0:80->80/tcp
▶ 示例：prometheus.yml 配置
YAML
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "pricetracker-api"
metrics_path: "/metrics"
static_configs:
- targets: ["api:8000"]
rule_files:
- "alert_rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets: ["alertmanager:9093"]
输出：
TEXT
📖 仅展示
配置文件已生效
❓ 常见问题
Q Prometheus 和 ELK 有什么区别？
A Prometheus 是指标监控（数值时间序列），ELK 是日志分析（文本搜索）。两者互补——Prometheus 看趋势和告警，ELK 看详细日志排查问题。
Q 指标采集间隔设多久？
A 默认 15 秒是好的平衡。更短（5s）更实时但存储成本高，更长（60s）省存储但可能错过短暂抖动。
Q 自定义指标太多会拖慢应用吗？
A 每个指标有微小 CPU 开销。建议 < 100 个自定义指标。Label 基数（不同值数量）比指标数量更影响性能，避免高基数 Label（如 user_id）。
Q Grafana Dashboard 怎么共享？
A 导出为 JSON 文件，团队导入即可。Grafana.com 有大量现成模板可参考。
Q 告警太多怎么办（告警疲劳）？
A 设合理的 for 持续时间（避免瞬时抖动触发），分级 warning/critical，分组相关告警，减少通知渠道噪音。
Q 如何监控 Celery 任务？
A Flower 提供任务级监控，也暴露 Prometheus 指标。关键指标：任务成功率、队列长度、Worker 内存使用。
📖 小节
prometheus-fastapi-instrumentator一行代码暴露默认 HTTP 指标- 自定义 Counter（计数）、Histogram（延迟分布）、Gauge（当前值）三类业务指标
- Grafana Dashboard 用 RED 方法组织：Rate（QPS）、Errors（错误率）、Duration（延迟）
- AlertManager 配置告警规则：P99 > 500ms、错误率 > 5%、缓存命中率 < 60%
- Docker Compose 一键部署完整监控栈：Prometheus + Grafana + AlertManager
📝 作业
- 基础题（难度⭐）：为 FastAPI 添加
prometheus-fastapi-instrumentator，验证/metrics端点返回默认 HTTP 指标（http_requests_total 等）。提示：Instrumentator().instrument(app).expose(app) - 进阶题（难度⭐⭐）：添加 3 个自定义指标——PRICE_UPDATES（Counter 按 category）、REQUEST_LATENCY（Histogram 含百分位桶）、CACHE_HIT_RATE（Gauge），在端点中记录指标值。提示：
Counter(..., ["category"])+.labels(category="electronics").inc() - 挑战题（难度⭐⭐⭐）：Docker Compose 添加 Prometheus + Grafana + AlertManager，配置 prometheus.yml 采集 FastAPI 指标，编写 P99 > 500ms 和错误率 > 5% 两条告警规则，Grafana 导入 Dashboard 查看实时指标。提示：
docker/prometheus.yml+alert_rules.yml+ Grafana 数据源配置
---|
