Compose Advanced Orchestration
Development, testing, and production—one Compose file handles all three environments.
1. What You'll Learn
- Profiles: Multi-Environment Configuration
- Configuration Reuse and Inheritance (extends / override)
- Horizontal scaling
- Statement on Resource Limitations
- Health Check Dependency Control
2. A True Story of a Technical Lead
(1) Pain Point: Three Environments, Three Configuration Files
Charlie needs to maintain separate Compose configurations for three environments: development, testing, and production. The development environment requires hot reloading and debugging tools; the testing environment requires a full service stack; and the production environment requires resource limits and multiple replicas. The three separate YAML files contain a lot of duplicate code, so changing something in one file means changing it in all three.
(2) Profiles + Override Solution
Charlie used profiles and override files to implement a solution based on "one base configuration + environment-specific overrides."
BASH
# Development: activate dev profile + dev overrides
docker compose --profile dev -f docker-compose.yml -f docker-compose.dev.yml up -d
# Production: prod overrides only
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
(3) Output: 3 files → 1 + 2 files
From three completely separate files to one base file plus two difference-override files. The common configuration needs to be maintained in only one place.
3. Profiles: Multi-Environment Configuration
(1) The Profile Mechanism
graph TB
BASE["docker-compose.yml<br/>Base services"] --> DEV["--profile dev<br/>+ dev tools (hot-reload, adminer)"]
BASE --> TEST["--profile test<br/>+ test runners (selenium)"]
BASE --> PROD["--profile prod<br/>+ monitoring (prometheus, grafana)"]
▶ Example: --profile dev enabled (Difficulty: ⭐⭐)
YAML
# docker-compose.yml with profiles
services:
api:
build: .
ports:
- "5000:5000"
environment:
- FLASK_ENV=development
db:
image: postgres:15-alpine
volumes:
- pg-data:/var/lib/postgresql/data
# Dev-only: database management UI
adminer:
image: adminer
ports:
- "8081:8080"
profiles: ["dev"]
depends_on: [db]
# Dev-only: hot-reload with flask debug
api-dev:
build:
context: .
dockerfile: Dockerfile.dev
volumes:
- ./src:/app
ports:
- "5000:5000"
- "5678:5678" # Debug port
environment:
- FLASK_DEBUG=1
profiles: ["dev"]
# Prod-only: Prometheus monitoring
prometheus:
image: prom/prometheus
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "9090:9090"
profiles: ["prod"]
volumes:
pg-data:
BASH
# Development: base services + dev profile
docker compose --profile dev up -d
# Production: base services + prod profile
docker compose --profile prod up -d
# Only base services (no profile)
docker compose up -d
(1) Profiles vs Override Comparison
| Dimension | Profiles | Override Files |
|---|---|---|
| Number of files | 1 file | 2–3 files |
| Activation Method | --profile xxx |
-f base.yml -f override.yml |
| Suitable Scenarios | Scaling Services | Significant Configuration Differences |
| Composability | Multiple profiles can be combined | Override in order |
4. Override File Merging
▶ Example: Overriding File Configuration (Difficulty: ⭐⭐⭐)
YAML
# docker-compose.yml (base configuration)
services:
api:
build: .
environment:
FLASK_ENV: production
restart: unless-stopped
db:
image: postgres:15-alpine
volumes:
- pg-data:/var/lib/postgresql/data
restart: unless-stopped
volumes:
pg-data:
YAML
# docker-compose.dev.yml (dev overrides)
services:
api:
environment:
FLASK_ENV: development
FLASK_DEBUG: "1"
volumes:
- ./src:/app # Hot-reload: mount source code
ports:
- "5678:5678" # Debug port
# Dev-only service
adminer:
image: adminer
ports:
- "8081:8080"
YAML
# docker-compose.prod.yml (prod overrides)
services:
api:
deploy:
replicas: 3
resources:
limits:
cpus: "1.0"
memory: 512M
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
interval: 30s
timeout: 5s
retries: 3
# Prod-only: Nginx reverse proxy
nginx:
image: nginx:1.25-alpine
ports:
- "80:80"
volumes:
- ./nginx.prod.conf:/etc/nginx/conf.d/default.conf
depends_on:
- api
BASH
# Development
docker compose -f docker-compose.yml -f docker-compose.dev.yml up -d
# Production
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
# Preview merged config (dry-run)
docker compose -f docker-compose.yml -f docker-compose.prod.yml config
5. Horizontal Scaling
▶ Example: --scale Extension (Difficulty ⭐⭐)
BASH
# Scale the API service to 3 replicas
docker compose up -d --scale api=3
# Scale dynamically on a running stack
docker compose up -d --scale api=5
(1) Scaling Considerations
| Problem | Cause | Solution |
|---|---|---|
| Port Conflict | Multiple replicas mapped to the same host port | Only one entry point exposed (Nginx); the API does not map to a host port |
| Data Consistency | Writing to multiple copies in the same storage | Only stateless services are suitable for scaling; shared databases |
| Load Balancing | How requests are distributed across multiple replicas | Nginx upstream or Docker's built-in DNS round-robin |
▶ Example: Declarative scaling of deploy.replicas (Difficulty: ⭐⭐)
YAML
services:
api:
build: .
deploy:
replicas: 3
resources:
limits:
cpus: "0.5"
memory: 256M
reservations:
cpus: "0.25"
memory: 128M
# Do NOT map ports for scaled services
# ports: ["5000:5000"] ← This breaks with replicas > 1
nginx:
image: nginx:alpine
ports:
- "80:80"
# Nginx upstream uses DNS round-robin to api:5000
6. Resource Limitations
(1) deploy.resources Configuration
YAML
services:
api:
deploy:
resources:
limits: # Hard limits (kill if exceeded)
cpus: "1.0"
memory: 512M
reservations: # Soft guarantees (minimum)
cpus: "0.5"
memory: 256M
| Field | Purpose | Effect |
|---|---|---|
limits.cpus |
CPU Limit | Exceeded, Throttled |
limits.memory |
Memory Limit | Exceeded: OOM Kill |
reservations.cpus |
Minimum CPU Guarantee | Scheduling Guarantee |
reservations.memory |
Minimum Memory Guarantee | Scheduling Guarantee |
7. Complete Example: Compose Project for Three Environments
YAML
# ============================================
# docker-compose.yml - Base configuration
# ============================================
services:
api:
build:
context: .
dockerfile: Dockerfile
environment:
DATABASE_URL: postgresql://postgres:${DB_PASSWORD:-secret}@db:5432/${DB_NAME:-myapp}
depends_on:
db:
condition: service_healthy
restart: unless-stopped
networks:
- backend
db:
image: postgres:15-alpine
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD:-secret}
POSTGRES_DB: ${DB_NAME:-myapp}
volumes:
- pg-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U postgres"]
interval: 5s
timeout: 5s
retries: 5
restart: unless-stopped
networks:
- backend
# Dev-only: Adminer DB UI
adminer:
image: adminer
ports:
- "8081:8080"
profiles: ["dev"]
networks:
- backend
volumes:
pg-data:
networks:
backend:
YAML
# ============================================
# docker-compose.prod.yml - Production overrides
# ============================================
services:
api:
deploy:
replicas: 3
resources:
limits:
cpus: "1.0"
memory: 512M
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
interval: 30s
timeout: 5s
retries: 3
nginx:
image: nginx:1.25-alpine
ports:
- "80:80"
volumes:
- ./nginx.prod.conf:/etc/nginx/conf.d/default.conf
depends_on:
api:
condition: service_healthy
restart: unless-stopped
networks:
- backend
BASH
# Development
docker compose --profile dev up -d
# Production (3 replicas + Nginx + resource limits)
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
# Verify scaling
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
❓ FAQ
Q Can profiles and override files be used together?
A Yes.
docker compose --profile dev -f docker-compose.yml -f docker-compose.dev.yml up -d Use profiles and override files together. Profiles manage the addition and removal of services, while override files manage configuration differences; the two complement each other.Q Which takes precedence—
--scale or deploy.replicas?A The
--scale command-line argument takes precedence over the deploy.replicas YAML declaration. If both are set, the value of --scale overrides the value in the YAML. In a production environment, we recommend using the YAML declaration (which supports version control); use --scale for quick adjustments during debugging.Q How do I control the order in which services start?
A
depends_on + condition. Three conditions: ① service_started (default; wait only for startup); ② service_healthy (wait for health check to pass); ③ service_completed_successfully (wait for initialization tasks to complete). Always use service_healthy in production environments.Q How do I switch between variables in Compose files across different environments?
A
${VAR:-default} + .env files. Place different .env files in each environment: .env.dev / .env.prod. Specify --env-file at startup: docker compose --env-file .env.prod up -d.Q How are ports handled when there are multiple replicas?
A The scaled service cannot map host ports (multiple replicas would cause conflicts). Only allow Nginx/HAProxy to map ports; the API service communicates internally over the network. Nginx’s upstream uses Docker DNS round-robin to automatically distribute traffic across multiple replicas.
📖 Summary
- Profiles: Add or remove services within the same file based on the environment;
--profile devto activate - Override file:
-f base.yml -f override.ymlMerged configuration, suitable for a large number of differences - Scaling:
--scale api=3ordeploy.replicas: 3Horizontal Scaling of Stateless Services - The extended service cannot map to a host port—use an Nginx entry point + internal DNS round-robin
deploy.resourcesDeclaring CPU/Memory Limits and Reservationsdepends_on: { condition: service_healthy }Ensure that dependencies are truly ready
📝 Exercises
- Basic Exercise (Difficulty ⭐): Write two sets of override files (one for dev and one for prod) for the application from Lesson 12; add hot reloading for dev and resource limits for prod.
- Advanced Exercise (Difficulty ⭐⭐): Use
--scale api=3to scale the API service to 3 replicas, and usedocker compose psto verify it. - Challenge (Difficulty: ⭐⭐⭐): Configure an Nginx reverse proxy to an API with 3 replicas, and verify that requests are distributed to different replicas (by checking the logs of different containers).
