Logging and Monitoring
A container crashes at 3 a.m.—logs and monitoring are your only tools for quickly pinpointing the root cause.
1. What You'll Learn
- Four Types of Log Drivers and Their Applicable Scenarios
- Tips for Viewing and Filtering Real-Time Logs
- Methods for Monitoring Container Resources
- Log rotation strategy
- Approach to Centralized Log Collection
2. A True Story of an Operations Engineer
(1) Pain Point: No Clues About the OOM Alert at 3 a.m.
At 3 a.m., a container in the production environment experienced a sudden spike in memory usage, causing an OOM error. Charlie was alerted and woken up, but he didn’t know the cause—the log file was 2 GB, docker logs was scrolling by too fast to find the key information, and there was no historical resource monitoring data, so he could only guess.
(2) Solutions for Log Filtering and Resource Monitoring
Charlie used the combination of docker logs --since and grep to find the abnormal request within 30 seconds.
BASH
# Find OOM-related logs in the last 2 hours
docker logs --since 2h --tail 500 app 2>&1 | grep -i "oom\|memory\|error"
# Check resource usage history
docker stats --no-stream
(3) Benefit: Identify the root cause in 30 seconds
From 2 GB of logs, he pinpointed the OOM event and the abnormal request within 30 seconds. Charlie then configured log rotation and Prometheus alerts so that future alerts in the early morning would automatically include the root cause.
3. Docker Log Drivers
(1) Four Types of Log Drivers
graph LR
CTN["Container<br/>stdout/stderr"] --> DRV["Log Driver"]
DRV --> JSON["json-file<br/>Default，Local Files"]
DRV --> JRN["journald<br/>systemd Log"]
DRV --> SYS["syslog<br/>Remote Log Server"]
DRV --> GELF["gelf<br/>ELK/Loki"]
| Drive | Storage Location | Log Rotation | Centralized | Use Cases |
|---|---|---|---|---|
| json-file (default) | /var/lib/docker/containers/ |
Requires manual configuration | ❌ | Standalone development |
| journald | systemd journal | ✅ Automatic | ❌ | systemd |
| syslog | Remote syslog server | ✅ Server | ✅ | Traditional logging system |
| gelf | ELK/Loki | ✅ Server-side | ✅ | Centralized logging platform |
| local | Local files (optimized format) | ✅ Automatic | ❌ | Lightweight local logs |
| none | No logging | - | - | High-performance/sensitive data |
(2) Configure the log driver
BASH
# Per-container: specify log driver at runtime
docker run -d \
--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=3 \
--name app myapp:1.0
JSON
// Global: /etc/docker/daemon.json
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}
4. Advanced Usage of docker logs
▶ Example: --since time filter (Difficulty: ⭐⭐)
BASH
# Show logs from the last 2 hours
docker logs --since 2h app
# Show logs from a specific time
docker logs --since "2024-01-15T03:00:00" app
# Show logs between two timestamps
docker logs --since "2024-01-15T03:00:00" --until "2024-01-15T04:00:00" app
▶ Example: --tail line limit (Difficulty ⭐)
BASH
# Show last 100 lines
docker logs --tail 100 app
# Follow last 100 lines in real-time
docker logs -f --tail 100 app
▶ Example: Real-time monitoring with docker stats (Difficulty: ⭐⭐)
BASH
# Real-time resource stats for all containers
docker stats
# One-time snapshot (no streaming)
docker stats --no-stream
# Stats for specific containers
docker stats app db --no-stream
# Custom format
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
💻 Output:
TEXT
NAME CPU % MEM USAGE / LIMIT
app 2.35% 150MiB / 512MiB
db 0.50% 85MiB / 256MiB
▶ Example: Docker Events Event Tracing (Difficulty: ⭐⭐)
BASH
# Stream Docker daemon events
docker events
# Filter by event type
docker events --filter type=container
# Filter by specific event
docker events --filter event=oom --filter event=die
# Filter by time range
docker events --since "2024-01-15T03:00:00" --until "2024-01-15T04:00:00"
💻 Output:
TEXT
2024-01-15T03:24:15Z container oom 67890... (name=app, image=myapp)
2024-01-15T03:24:15Z container die 67890... (exitCode=137, name=app)
2024-01-15T03:24:25Z container start 67890... (name=app, image=myapp)
5. Log Rotation Strategy
(1) JSON file log rotation
| Parameter | Function | Recommended Value |
|---|---|---|
max-size |
Maximum size of a single log file | 10m |
max-file |
Maximum number of log files to retain | 3 |
labels |
Post Tags | - |
tag |
Log Tag Template | - |
▶ Example: Log Rotation Configuration (Difficulty: ⭐⭐)
BASH
# Configure log rotation at container start
docker run -d \
--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=3 \
--name app myapp:1.0
💡 Tip: Containers without log rotation are "time bombs"—log files will grow indefinitely until they fill up the disk. In production environments, you must configure
max-size and max-file, or use the local log driver (which includes built-in rotation).
(1) Logging Configuration in Compose
YAML
services:
api:
image: myapp:1.0
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
6. Centralized Logging Solution
(1) Comparison of the Three Options
| Solution | Components | Complexity | Suitable Scale |
|---|---|---|---|
| docker logs | Built-in | ⭐ | Single-machine/Development |
| ELK Stack | Elasticsearch + Logstash + Kibana | ⭐⭐⭐ | Enterprise-grade |
| Loki + Grafana | Loki + Promtail + Grafana | ⭐⭐ | Small/Medium-sized/Cloud-native |
7. Complete Example: Configuring Log Rotation and Monitoring
BASH
# ============================================
# Complete walkthrough: Log rotation + monitoring
# Covers: log rotation, stats, events, OOM diagnosis
# ============================================
# 1. Start app with log rotation
docker run -d \
--name app \
--log-driver json-file \
--log-opt max-size=10m \
--log-opt max-file=3 \
--memory=256m \
--restart=on-failure:5 \
-p 5000:5000 \
myapp:1.0
# 2. Monitor resource usage
docker stats app --no-stream
# 3. Simulate: trigger OOM by exceeding memory
docker exec app python -c "
import numpy as np
x = np.zeros((50000, 50000)) # Try to allocate ~20GB
"
# 4. Capture the OOM event
docker events --filter event=oom --since 1m
# 5. Find OOM in logs with timestamp
docker logs --since 5m --tail 200 app 2>&1 | grep -i "oom\|memory\|killed"
# 6. Check exit code
docker inspect app --format='ExitCode: {{.State.ExitCode}}, OOMKilled: {{.State.OOMKilled}}'
# 7. Export last 1000 lines for analysis
docker logs --tail 1000 app > app-debug.log 2>&1
# 8. Verify log rotation is working
ls -la /var/lib/docker/containers/$(docker inspect app --format='{{.Id}}')/
❓ FAQ
Q Can a log file that’s too large affect the disk?
A Yes. By default, the json-file driver does not rotate logs, so the log file grows indefinitely. I’ve seen cases in production environments where a single container’s log file reached 50 GB. You must configure
--log-opt max-size=10m --log-opt max-file=3 at startup or set the default rotation globally in daemon.json.Q How do I configure log rotation?
A There are two ways: ① Use
--log-opt max-size=10m --log-opt max-file=3 when starting the container; ② Configure "log-opts": {"max-size": "10m", "max-file": "3"} in the global configuration /etc/docker/daemon.json. We recommend using the global configuration in a production environment to avoid omissions.Q How can I centrally collect logs from multiple containers?
A Use the gelf or syslog log driver to send all container logs to a centralized ELK/Loki platform. A more recommended approach: Keep the json-file driver and use Filebeat/Promtail to collect log files; this way,
docker logs will still work.Q Are the logs still there after the container restarts?
A Yes. Restarting the container does not delete the log files. However,
docker rm deletes the logs when the container is deleted. To persist logs: ① Write them to application files and mount them as a volume; ② Use a centralized logging platform for real-time collection.Q Does the memory shown by
docker stats refer to the container or the process?A The container.
docker stats displays the total memory usage of all processes within the container’s cgroup. LIMIT is the upper limit set by --memory. If --memory is not set, LIMIT displays the total host memory.📖 Summary
- Four log drivers: json-file (default), journald, syslog, and gelf—log rotation must be configured in production environments
docker logsAdvanced Filtering:--since/--untilTime Filter,--tailRow Limit,-fReal-Time Trackingdocker statsReal-time monitoring of CPU, memory, and I/O;--no-streamOne-time snapshotdocker eventsTrack container lifecycle events, including critical events such as OOM, die, and restart- Log Rotation:
max-size=10m max-file=3Prevents logs from filling up the disk - Centralized logging: ELK (enterprise-grade) or Loki+Grafana (lightweight/cloud-native)
📝 Exercises
- Basic Exercise (Difficulty ⭐): Start the container and configure
--log-opt max-size=5m --log-opt max-file=2, then usedocker logsto view the logs and verify that round-robin has taken effect. - Advanced Problem (Difficulty ⭐⭐): Use
docker statsto monitor a container during a load test and record the peak CPU and memory usage. - Challenge (Difficulty: ⭐⭐⭐): Use
docker eventsto capture a container OOM event, and combine it withdocker logs --sinceto identify the request or operation that triggered the OOM.
