Observability
Observability is the eyes of production operations—metrics reveal trends, logs pinpoint causes, and trace data identifies root causes. These three pillars are all indispensable.
1. What You'll Learn
- Micrometer Metric Collection: Counter / Gauge / Timer / DistributionSummary
- Prometheus Crawl Configuration and PromQL Query Alert Rules
- Grafana Dashboard Visualizations: JVM / HTTP / Database Metrics Dashboards
- OpenTelemetry Distributed Tracing and Span/Context Propagation
- Bob set up the OrderFlow SLO dashboard: P99 < 100 ms / Error Rate < 0.1% / Availability > 99.9%
2. A True Story of an SRE Engineer
(1) Pain Point: Black-Box Troubleshooting of Online Issues
When OrderFlow encounters a production error, Bob is left sifting through a sea of application logs: Which API is slow? Which service is having issues? Which microservices did the request pass through? He has no idea. Alice and Bob often spend four hours troubleshooting a single production issue, three of which are spent "guessing" where the problem lies.
(2) Approaches to the Three Pillars of Observability
graph LR
A["Observability<br/>Three Pillars"] --> B["Metrics<br/>What happened?<br/>Prometheus"]
A --> C["Logs<br/>Why happened?<br/>ELK / Loki"]
A --> D["Traces<br/>Where happened?<br/>Jaeger / Zipkin"]
Identify issues using metrics, analyze logs to determine the cause, and use trace analysis to pinpoint the root cause.
(3) Revenue
After Bob established the observability system, Grafana dashboards began displaying P99 latency and error rates in real time, Prometheus alerts triggered automatic notifications, and OpenTelemetry trace data pinpointed the root cause of issues within 30 seconds. The average time to resolve issues dropped from 4 hours to 15 minutes.
3. Collection of Micrometer Data
(1) Four Types of Metrics
| Type | Meaning | Increases Only | Typical Scenarios |
|---|---|---|---|
| Counter | Counter (Increment Only) | Yes | Total requests, number of orders created |
| Gauge | Current Value (Can Be Increased or Decreased) | No | Current Number of Connections, Queue Length |
| Timer | Duration Distribution | No | Request-Response Time |
| DistributionSummary | Distribution Statistics | No | Request Body Size Distribution |
(1) ▶ Example: Custom Business Metrics
JAVA
@Service
public class OrderMetrics {
private final Counter orderCreatedCounter;
private final Counter orderCancelledCounter;
private final Timer orderCreationTimer;
private final Gauge pendingOrdersGauge;
public OrderMetrics(MeterRegistry registry, OrderRepository orderRepo) {
this.orderCreatedCounter = Counter.builder("orderflow.orders.created")
.description("Total orders created")
.tag("service", "orderflow")
.register(registry);
this.orderCancelledCounter = Counter.builder("orderflow.orders.cancelled")
.description("Total orders cancelled")
.register(registry);
this.orderCreationTimer = Timer.builder("orderflow.orders.creation.duration")
.description("Order creation duration")
.publishPercentiles(0.5, 0.95, 0.99)
.publishPercentileHistogram()
.register(registry);
this.pendingOrdersGauge = Gauge.builder("orderflow.orders.pending",
orderRepo, repo -> repo.countByStatus("PENDING"))
.description("Current pending orders count")
.register(registry);
}
public void recordOrderCreated() {
orderCreatedCounter.increment();
}
public Timer.Sample startCreationTimer() {
return Timer.start(orderCreationTimer);
}
public void recordCreationComplete(Timer.Sample sample) {
sample.stop(orderCreationTimer);
}
}
Output:
TEXT
// Execution Successful
(2) ▶ Example: Using Metrics in a Service
JAVA
@Service
public class OrderServiceImpl implements OrderService {
private final OrderMetrics metrics;
@Override
@Transactional
public Order createOrder(CreateOrderRequest request) {
Timer.Sample sample = metrics.startCreationTimer();
try {
Order order = doCreateOrder(request);
metrics.recordOrderCreated();
return order;
} finally {
metrics.recordCreationComplete(sample);
}
}
}
Output:
TEXT
// Execution Successful
4. Prometheus Integration
(1) ▶ Example: Prometheus Dependencies and Configuration
XML
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
Output:
TEXT
// Execution Successful
YAML
# application.yml
management:
endpoints:
web:
exposure:
include: health,prometheus,metrics
metrics:
tags:
application: ${spring.application.name}
export:
prometheus:
enabled: true
(2) ▶ Example: Prometheus Scraping Configuration
YAML
# prometheus.yml
scrape_configs:
- job_name: 'orderflow'
metrics_path: '/actuator/prometheus'
scrape_interval: 15s
static_configs:
- targets: ['orderflow:8080']
Output:
TEXT
Monitoring config loaded
Prometheus targets: 3 active
Grafana dashboard: ready
(3) ▶ Example: PromQL Queries and Alerts
YAML
# PromQL queries
# P99 latency for order API
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket{uri=~"/api/v1/orders.*"}[5m])) by (le, uri))
# Error rate (5xx responses)
sum(rate(http_server_requests_seconds_total{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_total[5m]))
# Orders per minute
rate(orderflow_orders_created_total[1m]) * 60
Output:
TEXT
Configuration applied successfully
YAML
# alert_rules.yml
groups:
- name: orderflow
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_server_requests_seconds_total{status=~"5.."}[5m]))
/ sum(rate(http_server_requests_seconds_total[5m])) > 0.001
for: 5m
labels:
severity: critical
annotations:
summary: "OrderFlow error rate exceeds 0.1%"
- alert: HighP99Latency
expr: |
histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket[5m])) by (le)) > 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "OrderFlow P99 latency exceeds 100ms"
5. Grafana Dashboard
(1) Key Kanban Metrics
| Dashboard | Metric | PromQL |
|---|---|---|
| Request Rate | QPS | sum(rate(http_server_requests_seconds_total[5m])) |
| P50/P95/P99 Latency | Response Time Distribution | histogram_quantile(0.99, ...) |
| Error Rate | 5xx Ratio | rate(...{status=~"5.."})/rate(...) |
| JVM Heap Memory | Memory Usage | jvm_memory_used_bytes{area="heap"} |
| GC Pause | GC Duration | rate(jvm_gc_pause_seconds_sum[5m]) |
| HikariCP Active Connections | Connection Pool Usage | hikaricp_connections_active |
| Order Creation Rate | Business Metric | rate(orderflow_orders_created_total[1m]) |
(1) ▶ Example: Grafana Dashboard JSON snippet
JSON
{
"dashboard": {
"title": "OrderFlow Observability",
"panels": [
{
"title": "Request Rate (QPS)",
"type": "timeseries",
"targets": [{
"expr": "sum(rate(http_server_requests_seconds_total{application=\"orderflow-service\"}[5m]))"
}]
},
{
"title": "P99 Latency",
"type": "timeseries",
"targets": [{
"expr": "histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket{application=\"orderflow-service\"}[5m])) by (le))"
}]
},
{
"title": "Error Rate",
"type": "gauge",
"targets": [{
"expr": "sum(rate(http_server_requests_seconds_total{application=\"orderflow-service\",status=~\"5..\"}[5m])) / sum(rate(http_server_requests_seconds_total{application=\"orderflow-service\"}[5m]))"
}],
"fieldConfig": {
"defaults": {
"thresholds": {
"steps": [
{"value": 0, "color": "green"},
{"value": 0.001, "color": "red"}
]
}
}
}
}
]
}
}
Output:
JSON
{
"dashboard": {
"title": "OrderFlow Observability",
"panels": [
{
"title": "Request Rate (QPS)",
"type": "timeseries",
"targets": [
{
"expr": "sum(rate(http_server_requests_seconds_total{application=\"orderflow-service\"}[5m]))"
}
]
},
{
"title": "P99 Latency",
"type": "timeseries",
"targets": [
{
"expr": "histogram_quantile(0.99, sum(rate(http_server_request
6. OpenTelemetry Trace
(1) The Concept of Distributed Tracing
graph LR
A["Client"] --> B["API Gateway<br/>Trace: abc123<br/>Span 1"]
B --> C["Order Service<br/>Span 2<br/>parent: Span 1"]
C --> D["Product Service<br/>Span 3<br/>parent: Span 2"]
C --> E["Database<br/>Span 4<br/>parent: Span 2"]
| Concept | Meaning |
|---|---|
| Trace | Complete path of a single request |
| Span | An operation in the chain |
| Context | Trace context passed between spans |
| SpanId | Unique identifier for the current span |
| TraceId | Unique identifier for the entire trace |
(1) ▶ Example: OpenTelemetry Dependency
XML
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
</dependency>
Output:
TEXT
// Execution Successful
YAML
# application.yml
otel:
exporter:
otlp:
endpoint: http://otel-collector:4317
resource:
attributes:
service.name: orderflow-service
traces:
exporter: otlp
(2) ▶ Example: Custom Span
JAVA
@Service
public class OrderService {
private final Tracer tracer;
public OrderService(Tracer tracer) {
this.tracer = tracer;
}
public Order createOrder(CreateOrderRequest request) {
Span span = tracer.spanBuilder("create-order")
.setAttribute("product.id", request.productId())
.setAttribute("quantity", request.quantity())
.startSpan();
try (Scope scope = span.makeCurrent()) {
Order order = doCreateOrder(request);
span.setAttribute("order.id", order.getId());
return order;
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
span.end();
}
}
}
Output:
TEXT
// Execution Successful
7. Comprehensive Example: OrderFlow SLO Dashboard
YAML
# docker-compose.observability.yml
version: "3.9"
services:
prometheus:
image: prom/prometheus:latest
ports: ["9090:9090"]
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./alert_rules.yml:/etc/prometheus/alert_rules.yml
grafana:
image: grafana/grafana:latest
ports: ["3000:3000"]
environment:
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- grafana-data:/var/lib/grafana
otel-collector:
image: otel/opentelemetry-collector:latest
ports: ["4317:4317", "4318:4318"]
volumes:
- ./otel-collector-config.yml:/etc/otelcol/config.yaml
jaeger:
image: jaegertracing/all-in-one:latest
ports: ["16686:16686"]
app:
build: .
ports: ["8080:8080"]
environment:
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4317
OTEL_SERVICE_NAME: orderflow-service
MANAGEMENT_ENDPOINTS_WEB_EXPOSURE_INCLUDE: health,prometheus,metrics
| SLO Metric | Target | Alert Threshold | PromQL |
|---|---|---|---|
| P99 Latency | < 100 ms | > 100 ms for 5 minutes | histogram_quantile(0.99, ...) |
| Error Rate | < 0.1% | > 0.1% for 5 minutes | rate(5xx)/rate(all) |
| Availability | > 99.9% | < 99.9% for 5 minutes | 1 - rate(5xx)/rate(all) |
| Order Throughput | > 1,000/min | < 500/min for 10 minutes | rate(orders_created)[1m]*60 |
❓ FAQ
Q What is the difference between Metrics and Logs?
A Metrics are aggregated values (counters, histograms) and are suitable for monitoring trends and triggering alerts. Logs are records of discrete events and are suitable for analyzing specific causes. Use Metrics to identify issues and Logs to analyze their causes.
Q What are the respective roles of Prometheus and Grafana?
A Prometheus handles data collection and storage (time-series database) plus alerting rules. Grafana handles visualization (dashboards). When used together, Prometheus serves as the data source, and Grafana serves as the presentation layer.
Q What is the relationship between OpenTelemetry and Jaeger?
A OpenTelemetry is the "collection" standard (SDK + API), while Jaeger is the "storage and visualization" backend. Applications use the OTel SDK to collect trace data and send it to Jaeger for storage and querying.
Q What is the difference between
publishPercentiles and histogram_quantile?A
publishPercentiles calculates percentiles on the application side (saving Prometheus storage, but less accurate across multiple instances). histogram_quantile calculates percentiles on the Prometheus side (supports multi-instance aggregation and is more accurate). We recommend calculating on the Prometheus side in production environments.Q What is the difference between an SLO and an SLA?
A An SLO (Service Level Objective) is an internal target, such as P99 < 100 ms. An SLA (Service Level Agreement) is a commitment to customers that includes compensation for breaches. An SLO forms the basis of an SLA.
Q How do I choose a logging backend?
A ELK (Elasticsearch + Logstash + Kibana) is powerful but resource-intensive; Loki (part of the Grafana ecosystem) is lightweight but has limited query capabilities. We recommend Loki for small and medium-sized projects and ELK for large projects.
📖 Summary
- Four Micrometer metrics: Counter, Gauge (current value), Timer (elapsed time), and DistributionSummary (distribution)
- Prometheus scrapes
/actuator/prometheus, PromQL queries + alert rules - Grafana Dashboard displaying HTTP, JVM, connection pool, and business metrics
- OpenTelemetry collects distributed tracing data; Jaeger stores and visualizes it
- SLO Dashboard: P99 < 100 ms / Error Rate < 0.1% / Availability > 99.9%
- Three-Pillar Collaboration: Identify issues through metrics → Pin down the location via trace → Analyze logs to determine the root cause
📝 Exercises
-
Basic Exercise (Difficulty: ⭐): Configure Micrometer + Prometheus for OrderFlow, expose the
/actuator/prometheusendpoint, and define three custom business metrics (order creation count, order processing time, and number of pending orders). -
Advanced Exercise (Difficulty: ⭐⭐): Configure Prometheus to collect data and set up a Grafana dashboard to display HTTP QPS, P99 latency, error rate, and JVM heap memory. Write two Prometheus alert rules (for high error rate and high latency).
-
Challenge (Difficulty: ⭐⭐⭐): Integrate OpenTelemetry and Jaeger to implement distributed tracing. Add a custom span to the OrderService to trace the entire order placement flow (Controller → Service → Repository), and view the trace details in the Jaeger UI.
