English العربية Português 日本語

Ollama: Docker Containerized Deployment

Docker turns Ollama into LEGO bricks — pull images, orchestrate combinations, deploy with one command.

⚠️ Note: Docker containers do not persist data by default — deleting a container loses all model files (often several GB). You must use Volume mounting (-v ollama_data:/root/.ollama) to persist model data to the host. This must never be omitted in production.

📋 Prerequisites: You should first master the following

1. What You Will Learn

2. A Real Story from a SaaS Entrepreneur

(1) The Pain Point: Manually Installing Ollama on Every Server

Alice needs to deploy SupportBot to 3 servers, each requiring manual Ollama installation, model pulling, and environment configuration. Version inconsistencies, missed configurations, and environment differences cause various problems.

(2) The Solution: One-Command Docker Deployment

BASH 
# One command to run Ollama in container
docker run -d --gpus all -v ollama_data:/root/.ollama ollama/ollama

# Or with Docker Compose for full stack
docker compose up -d

3. Official Ollama Image

(1) Image Variants

Image Size Purpose
ollama/ollama ~800 MB CPU + GPU (auto-detected)
ollama/ollama:rocm ~1.2 GB AMD GPU only

(2) Basic Run Commands

Scenario Command
CPU only docker run -d -p 11434:11434 ollama/ollama
NVIDIA GPU docker run -d --gpus all -p 11434:11434 ollama/ollama
Persistent storage docker run -d -v ollama_data:/root/.ollama -p 11434:11434 ollama/ollama
Custom port docker run -d -p 8080:11434 ollama/ollama
100% 
flowchart TD
    A[ollama/ollama image] --> B{GPU available?}
    B -->|NVIDIA| C[nvidia-container-toolkit<br/>--gpus all]
    B -->|AMD| D[rocm image<br/>--device /dev/kfd]
    B -->|None| E[CPU-only mode]
    C --> F[Ollama Server :11434]
    D --> F
    E --> F

▶ Example 1: CPU Container Run

⚠️ Warning: Running a container without mounting a Volume (-v ollama_data:/root/.ollama) stores model data inside the container. Once the container is deleted, all models are lost and must be re-downloaded (often several GB). Production environments must mount persistent storage.

BASH 
# Run Ollama in container (CPU mode)
docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

# Verify it's running
docker ps | grep ollama

# Pull a model inside the container
docker exec ollama ollama pull qwen2.5

# Test the API
curl http://localhost:11434/api/tags

Output:

TEXT 
I'm a helpful AI assistant running locally on your machine...

4. GPU Container Configuration

(1) NVIDIA GPU Container Prerequisites

Step Command Description
1 Install NVIDIA drivers nvidia-smi should work
2 Install nvidia-container-toolkit Required for GPU passthrough
3 Restart Docker sudo systemctl restart docker
4 Verify GPU availability docker run --rm --gpus all nvidia/cuda:12.0.0-runtime-ubuntu22.04 nvidia-smi

(2) nvidia-container-toolkit Installation

Platform Installation Command
Ubuntu/Debian `curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey
CentOS/RHEL `curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
Configure Docker sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker

▶ Example 2: NVIDIA GPU Container Run

BASH 
# Run Ollama with GPU support
docker run -d \
  --name ollama-gpu \
  --gpus all \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

# Verify GPU is visible inside container
docker exec ollama-gpu nvidia-smi

# Test GPU inference speed
docker exec ollama-gpu ollama run --verbose llama3.2 "Hello"

# Specify which GPUs to use
docker run -d \
  --gpus '"device=0,1"' \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama

Output:

TEXT 
I'm a helpful AI assistant running locally on your machine...

▶ Example 3: AMD ROCm Container

BASH 
# Run Ollama with AMD GPU (ROCm)
docker run -d \
  --name ollama-rocm \
  --device /dev/kfd \
  --device /dev/dri \
  -p 11434:11434 \
  -v ollama_data:/root/.ollama \
  ollama/ollama:rocm

Output:

TEXT 
I'm a helpful AI assistant running locally on your machine...

5. Docker Compose Multi-Service Orchestration

💡 Tip: Docker Compose can orchestrate multiple services in one command (Ollama + Chroma + WebUI + FastAPI). Using healthcheck + depends_on.condition: service_healthy ensures dependent services are truly ready before starting downstream, which is more reliable than simple startup ordering.

💡 Tip: Docker Compose's healthcheck + depends_on.condition: service_healthy ensures dependent services are truly ready before starting downstream, which is more reliable than simple startup order. Ollama needs a few seconds after starting before it can respond to API requests.

(1) Typical Multi-Service Architecture

100% 
flowchart TD
    A[Nginx<br/>:80 Reverse Proxy] --> B[FastAPI App<br/>:8000]
    B --> C[Ollama<br/>:11434]
    B --> D[Chroma DB<br/>:8001 Vector Store]
    C --> E[Model Volume]
    D --> F[Chroma Volume]

(2) docker-compose.yml Template

Service Image Port Dependencies
ollama ollama/ollama 11434 GPU runtime
webui open-webui/open-webui 3000 ollama
chroma chromadb/chroma 8000 —
app custom fastapi 8001 ollama, chroma

▶ Example 4: Complete Docker Compose Configuration

⚠️ Warning: In production configurations, OLLAMA_HOST=0.0.0.0:11434 makes Ollama listen on all network interfaces. This is safe within the Docker internal network, but extremely dangerous if the port is mapped to the host's public IP. Always use this with Nginx authentication or firewall rules.

YAML 
# docker-compose.yml
version: "3.8"

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

  chroma:
    image: chromadb/chroma
    container_name: chroma
    ports:
      - "8001:8000"
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      - ANONYMIZED_TELEMETRY=FALSE
    restart: unless-stopped

volumes:
  ollama_data:
  webui_data:
  chroma_data:
BASH 
# Start all services
docker compose up -d

# Pull model inside ollama container
docker exec ollama ollama pull qwen2.5
docker exec ollama ollama pull nomic-embed-text

# Check all services
docker compose ps

# View logs
docker compose logs -f ollama

Output:

TEXT 
[+] Running 4/4
 ✔ Network ollama_default  Created
 ✔ Container ollama        Started
 ✔ Container open-webui    Started
 ✔ Container chroma        Started

(1) Volume Mounting Comparison

Method Command Persistence Performance
Named Volume -v ollama_data:/root/.ollama ✅ Docker-managed Good
Bind Mount -v /data/ollama:/root/.ollama ✅ Host-managed Best
tmpfs --tmpfs /root/.ollama ❌ Memory Fastest

(2) Network Mode Comparison

Mode Command Use Case Description
bridge Default Inter-container communication Requires port mapping
host --network host Low latency Shares host network
Custom network docker network create mynet Multi-service Automatic DNS resolution

▶ Example 5: Production-Grade Docker Compose

YAML 
# docker-compose.prod.yml
version: "3.8"

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - /data/ollama/models:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_KEEP_ALIVE=30m
      - OLLAMA_NUM_PARALLEL=4
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
        limits:
          memory: 16G
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3

  app:
    build: ./app
    container_name: supportbot
    ports:
      - "8000:8000"
    environment:
      - OLLAMA_HOST=http://ollama:11434
    depends_on:
      ollama:
        condition: service_healthy
    restart: unless-stopped

networks:
  default:
    name: supportbot-net

Output:

TEXT 
# Verify configuration file syntax
docker compose config --quiet && echo "✅ Configuration file syntax is correct"
# Output: ✅ Configuration file syntax is correct

7. Comprehensive Example: SupportBot Full-Stack Docker Deployment

ℹ️ Info: The init-models container uses a "one-time task" pattern — it automatically pulls required models after Ollama's health check passes, then exits. This eliminates the need to manually enter the container to run ollama pull, achieving true "one-command deployment."

YAML 
# ============================================
# Comprehensive: SupportBot full-stack Docker
# Ollama + FastAPI + Chroma + Open WebUI
# ============================================

# docker-compose.yml
version: "3.8"

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:11434/api/tags || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
    restart: unless-stopped

  init-models:
    image: curlimages/curl
    container_name: init-models
    depends_on:
      ollama:
        condition: service_healthy
    command: >
      sh -c "
        curl -s http://ollama:11434/api/pull -d '{\"name\":\"qwen2.5\"}' &&
        curl -s http://ollama:11434/api/pull -d '{\"name\":\"nomic-embed-text\"}' &&
        curl -s http://ollama:11434/api/pull -d '{\"name\":\"llava\"}'
      "

  chroma:
    image: chromadb/chroma
    container_name: chroma
    ports:
      - "8001:8000"
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      - ANONYMIZED_TELEMETRY=FALSE
    restart: unless-stopped

  supportbot:
    build:
      context: ./app
      dockerfile: Dockerfile
    container_name: supportbot
    ports:
      - "8000:8000"
    environment:
      - OLLAMA_HOST=http://ollama:11434
      - CHROMA_HOST=http://chroma:8000
      - EMBED_MODEL=nomic-embed-text
      - CHAT_MODEL=qwen2.5
    depends_on:
      ollama:
        condition: service_healthy
      chroma:
        condition: service_started
    restart: unless-stopped

  webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_models:
  chroma_data:
  webui_data:
BASH 
#!/bin/bash
# Deploy SupportBot stack

# Start all services
docker compose up -d

# Wait for Ollama to be healthy
echo "Waiting for Ollama to be ready..."
until curl -s http://localhost:11434/api/tags > /dev/null 2>&1; do
    sleep 2
done
echo "Ollama is ready!"

# Pull required models
docker exec ollama ollama pull qwen2.5
docker exec ollama ollama pull nomic-embed-text

# Verify all services
docker compose ps
echo "SupportBot stack deployed!"
echo "  Ollama API:  http://localhost:11434"
echo "  SupportBot:  http://localhost:8000"
echo "  WebUI:       http://localhost:3000"
echo "  Chroma:      http://localhost:8001"

Output:

TEXT 
Waiting for Ollama to be ready...
Ollama is ready!
NAME                IMAGE                              STATUS
ollama              ollama/ollama                      Up (healthy)
chroma              chromadb/chroma                    Up
supportbot          custom/app                         Up
webui               ghcr.io/open-webui/open-webui      Up
SupportBot stack deployed!
  Ollama API:  http://localhost:11434
  SupportBot:  http://localhost:8000
  WebUI:       http://localhost:3000
  Chroma:      http://localhost:8001

❓ FAQ

Q Is GPU inference inside Docker containers slower than a direct installation?
A There should be no difference under normal conditions. Check: 1) Whether --gpus all is effective; 2) Whether nvidia-container-toolkit is installed; 3) Whether Docker runtime is configured as nvidia.
Q Are models still there after container restart?
A If you mounted a Volume at /root/.ollama, model data is persisted and retained. Without a Volume, models are lost and must be re-pulled.
Q How do I enter a container to run ollama commands?
A Use docker exec -it ollama bash or docker exec ollama ollama pull model_name to operate inside the container.
Q Does Docker Compose depends_on guarantee service readiness?
A No. depends_on only guarantees startup order. Add healthcheck + condition: service_healthy to ensure dependent services are truly ready.
Q What is Open WebUI?
A An open-source ChatGPT-style web interface that connects directly to Ollama. Suitable for non-technical users and for demonstrations. One-command Docker deployment.
Q How do multiple containers share a GPU?
A Use --gpus all to let all containers share the GPU. Ollama internally queues requests. For isolation, use --gpus '"device=0"' and --gpus '"device=1"' to assign different GPUs.

📖 Summary

📝 Exercises

  1. Basic (⭐): Run an Ollama container with Docker, pull a model, and test a conversation via the API.
  2. Intermediate (⭐⭐): Write a Docker Compose configuration to start Ollama + Open WebUI, and complete a conversation in the WebUI.
  3. Advanced (⭐⭐⭐): Deploy the complete SupportBot full stack (Ollama + FastAPI + Chroma + WebUI), implement RAG Q&A, and verify persistence.
Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站，几个人的小团队，精力有限。页面虽经检查，难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了，麻烦告诉我们，我们会在收到反馈后第一时间进行修复，再次感谢您的光临 🙏