English العربية Português 日本語

Ollama: Project Deployment and Optimization

Deployment and optimization is SupportBot's graduation ceremony — from development to production, from functional to excellent.

⚠️ Note: Before deployment, complete the security checklist — ① requests without API Key return 401; ② rate-limited requests return 429; ③ port 11434 is not externally accessible; ④ SSL certificate is valid and HTTPS is enforced; ⑤ input/output filtering is working. Security configuration "looks correct" ≠ "actually works" — test each item with curl.

📋 Prerequisites: You should first master the following

1. What You Will Learn

2. Project Background

💡 Tip: Best practice for production environment variable management — all keys and sensitive configuration should be injected via .env files or a Secrets Manager, never hard-coded in code or Docker Compose files. Add .env to .gitignore; provide only a .env.example template in the repository.

(1) Key Pre-Deployment Issues

Alice's SupportBot is developed but faces deployment challenges:

Issue Risk Solution
Single-machine deployment, no redundancy Service outage Docker Compose multi-service
No authentication, API exposed Security vulnerability Nginx + API Key
No monitoring blind spots Slow fault detection Prometheus + Grafana
Performance unverified Latency exceeds target Benchmarking + tuning
No operations documentation Slow fault recovery Operations runbook

3. Docker Compose Production Configuration

⚠️ Warning: In production, Ollama's OLLAMA_HOST=0.0.0.0 is only safe within the Docker internal network; port 11434 must never be directly mapped to the public internet. Nginx is the only external entry point and must be configured with API Key authentication and HTTPS.

(1) Production Architecture

100% 
flowchart TD
    A[Internet<br/>:443] --> B[Nginx<br/>SSL + Auth + Rate Limit]
    B --> C[FastAPI SupportBot<br/>:8000]
    C --> D[Ollama<br/>:11434<br/>GPU Accelerated]
    C --> E[Chroma<br/>:8001<br/>Vector Store]
    F[Prometheus<br/>:9090] --> G[All Services<br/>Metrics Collection]
    G --> H[Grafana<br/>:3001<br/>Dashboards]

(2) Service List

Service Image Port Resources Persistence
nginx nginx:alpine 80/443 1 vCPU, 512MB Config files
supportbot custom 8000 2 vCPU, 4GB —
ollama ollama/ollama 11434 8 vCPU, 16GB, 1x GPU ollama_data
chroma chromadb/chroma 8001 2 vCPU, 4GB chroma_data
prometheus prom/prometheus 9090 1 vCPU, 2GB prometheus_data
grafana grafana/grafana 3001 1 vCPU, 1GB grafana_data

▶ Example 1: Production Docker Compose

YAML 
# docker-compose.prod.yml
version: "3.8"

services:
  nginx:
    image: nginx:alpine
    container_name: supportbot-nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/ssl/certs:ro
    depends_on:
      supportbot:
        condition: service_healthy
    restart: unless-stopped

  supportbot:
    build:
      context: ./app
      dockerfile: Dockerfile
    container_name: supportbot-api
    ports:
      - "8000:8000"
    environment:
      - OLLAMA_HOST=http://ollama:11434
      - CHROMA_HOST=http://chroma:8000
      - API_KEY=${SUPPORTBOT_API_KEY}
      - CHAT_MODEL=qwen2.5
      - CLASSIFIER_MODEL=llama3.2:3b
      - EMBED_MODEL=nomic-embed-text
    depends_on:
      ollama:
        condition: service_healthy
      chroma:
        condition: service_started
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 15s
      timeout: 5s
      retries: 3
    restart: unless-stopped

  ollama:
    image: ollama/ollama
    container_name: supportbot-ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0:11434
      - OLLAMA_NUM_PARALLEL=4
      - OLLAMA_KEEP_ALIVE=30m
      - OLLAMA_MAX_LOADED_MODELS=2
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 5
    restart: unless-stopped

  chroma:
    image: chromadb/chroma
    container_name: supportbot-chroma
    ports:
      - "8001:8000"
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      - ANONYMIZED_TELEMETRY=FALSE
    restart: unless-stopped

  prometheus:
    image: prom/prometheus
    container_name: supportbot-prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana
    container_name: supportbot-grafana
    ports:
      - "3001:3000"
    volumes:
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    depends_on: [prometheus]
    restart: unless-stopped

volumes:
  ollama_data:
  chroma_data:
  prometheus_data:
  grafana_data:
⚠️ Security Warning: Grafana's default admin password is admin — you must change it immediately after first login! In production, set a strong password via the GRAFANA_PASSWORD environment variable, e.g.: GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}, and configure a randomly generated password value in the .env file.

Output:

TEXT 
Docker Compose production deployment successful, all services healthy

▶ Example 2: Nginx Production Configuration

NGINX 
# nginx/nginx.conf
worker_processes auto;

events {
    worker_connections 1024;
}

http {
    # Rate limiting zone
    limit_req_zone $binary_remote_addr zone=api:10m rate=60r/m;

    upstream supportbot {
        server supportbot:8000;
    }

    # Redirect HTTP to HTTPS
    server {
        listen 80;
        return 301 https://$host$request_uri;
    }

    # HTTPS server
    server {
        listen 443 ssl;
        server_name ai.globalshop.example.com;

        ssl_certificate     /etc/ssl/certs/server.crt;
        ssl_certificate_key /etc/ssl/certs/server.key;
        ssl_protocols       TLSv1.2 TLSv1.3;

        # SupportBot API
        location /v1/ {
            limit_req zone=api burst=20 nodelay;

            # API Key authentication
            if ($http_x_api_key = "") {
                return 401 '{"error": "API key required"}';
            }

            proxy_pass http://supportbot;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 5s;
            proxy_read_timeout 30s;
        }

        # Health check (no auth)
        location /health {
            proxy_pass http://supportbot/health;
        }
    }
}

Output:

TEXT 
// Execution successful

4. Performance Benchmarking and Optimization

💡 Tip: Always "warm up" before benchmarking — send 1-2 requests to load the model onto the GPU. The first request includes model loading time (cold start 5-30 seconds); subsequent requests reflect true inference latency. OLLAMA_KEEP_ALIVE controls model residence time.

(1) Key Performance Targets

Metric Target Test Method
TTFT < 500ms First-token latency measurement
P95 latency < 3s 100 requests, take P95
Throughput > 20 QPS Concurrent testing
Auto-resolution rate > 80% 100 real questions evaluation

▶ Example 3: Benchmark Script

PYTHON 
# benchmark.py
import asyncio
import aiohttp
import time
import json
import statistics

import os

API_URL = "http://localhost:8000/v1/chat"
API_KEY = os.environ.get("SUPPORTBOT_API_KEY", "your-api-key-here")

TEST_QUESTIONS = [
    "What is the return policy?",
    "How long does shipping take?",
    "Combien coûte la livraison?",
    "I received a damaged item, what do I do?",
    "Compare the wireless headphones models",
    "Where is order #88765?",
    "What is the return policy?",
    "My product broke after 2 weeks, I want a refund!",
    "Do you ship to Germany?",
    "What is the warranty for electronics?",
]

async def single_request(session: aiohttp.ClientSession, question: str) -> dict:
    start = time.time()
    try:
        async with session.post(
            API_URL,
            json={"message": question},
            headers={"X-Api-Key": API_KEY},
            timeout=aiohttp.ClientTimeout(total=30)
        ) as response:
            data = await response.json()
            elapsed = time.time() - start
            return {
                "question": question[:40],
                "latency_s": round(elapsed, 2),
                "intent": data.get("intent", "?"),
                "handler": data.get("handler", "?"),
                "language": data.get("language", "?"),
                "status": "success" if response.status == 200 else "error"
            }
    except Exception as e:
        return {"question": question[:40], "latency_s": round(time.time() - start, 2),
                "status": "error", "error": str(e)}

async def run_benchmark(concurrency: int = 1, total_requests: int = 50) -> dict:
    async with aiohttp.ClientSession() as session:
        # Warm up
        await single_request(session, "Hello")

        # Run benchmark
        start = time.time()
        tasks = []
        for i in range(total_requests):
            q = TEST_QUESTIONS[i % len(TEST_QUESTIONS)]
            tasks.append(single_request(session, q))
            if len(tasks) >= concurrency:
                results = await asyncio.gather(*tasks)
                tasks = []

        if tasks:
            results += await asyncio.gather(*tasks)

        total_time = time.time() - start

    latencies = [r["latency_s"] for r in results if r["status"] == "success"]
    errors = sum(1 for r in results if r["status"] == "error")

    return {
        "concurrency": concurrency,
        "total_requests": total_requests,
        "total_time_s": round(total_time, 1),
        "errors": errors,
        "error_rate": round(errors / total_requests * 100, 1),
        "avg_latency_s": round(statistics.mean(latencies), 2) if latencies else 0,
        "p50_latency_s": round(statistics.median(latencies), 2) if latencies else 0,
        "p95_latency_s": round(sorted(latencies)[int(len(latencies) * 0.95)], 2) if latencies else 0,
        "max_latency_s": round(max(latencies), 2) if latencies else 0,
        "throughput_rps": round(total_requests / total_time, 1)
    }

if __name__ == "__main__":
    print("=== SupportBot Benchmark ===")
    for c in [1, 4, 8]:
        result = asyncio.run(run_benchmark(concurrency=c, total_requests=20))
        print(f"\nConcurrency {c}:")
        for k, v in result.items():
            print(f"  {k}: {v}")

Output:

TEXT 
=== SupportBot Benchmark ===

5. Security Hardening

⚠️ Warning: Before going live, verify each security item individually — requests without API Key should return 401, rate-limited requests should return 429, port 11434 should not be externally accessible. Security configuration "looks correct" ≠ "actually works" — test with curl.

(1) Security Checklist

Check Item Configuration Status
API Key authentication Nginx X-Api-Key validation ✅
HTTPS encryption Nginx SSL configuration ✅
Rate limiting 60 req/min/IP ✅
Input filtering Prompt injection detection ✅
Output filtering Sensitive content filtering ✅
Data sanitization PII auto-removal ✅
Ollama internal network 127.0.0.1 binding ✅

▶ Example 4: Integrated Security Middleware

PYTHON 
# security.py - FastAPI security middleware
from fastapi import Request, HTTPException
from fastapi.responses import JSONResponse
import re
import time
from collections import defaultdict

class SecurityMiddleware:
    def __init__(self, api_key: str, rate_limit: int = 60):
        self.api_key = api_key
        self.rate_limit = rate_limit
        self.request_counts: dict = defaultdict(list)

        self.injection_patterns = [
            r"ignore\s+(previous|all)\s+instructions",
            r"you\s+are\s+now\s+DAN",
            r"show\s+(me\s+)?(your\s+)?system\s+prompt",
            r"reveal\s+(your|the)\s+(initial|system)",
        ]

        self.output_patterns = [
            r"system\s*prompt[:\s]",
            r"RETURN_POLICY_INTERNAL",
            r"INTERNAL_PRICING",
        ]

    def check_api_key(self, request: Request) -> bool:
        key = request.headers.get("X-Api-Key", "")
        return key == self.api_key

    def check_rate_limit(self, client_ip: str) -> bool:
        now = time.time()
        self.request_counts[client_ip] = [
            t for t in self.request_counts[client_ip]
            if now - t < 60
        ]
        if len(self.request_counts[client_ip]) >= self.rate_limit:
            return False
        self.request_counts[client_ip].append(now)
        return True

    def check_input(self, text: str) -> tuple[bool, str]:
        for pattern in self.injection_patterns:
            if re.search(pattern, text, re.IGNORECASE):
                return False, "Potential prompt injection detected"
        return True, ""

    def check_output(self, text: str) -> tuple[bool, str]:
        for pattern in self.output_patterns:
            if re.search(pattern, text, re.IGNORECASE):
                return False, "Sensitive content in response filtered"
        return True, ""

    def sanitize_pii(self, text: str) -> str:
        text = re.sub(r"[\w.-]+@[\w.-]+\.\w+", "[EMAIL_REDACTED]", text)
        text = re.sub(r"\b\d{3}[-.]?\d{3}[-.]?\d{4}\b", "[PHONE_REDACTED]", text)
        text = re.sub(r"\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b", "[CC_REDACTED]", text)
        return text

Output:

TEXT 
# Function defined successfully

6. Monitoring Integration

ℹ️ Info: GPU monitoring recommends using DCGM Exporter (NVIDIA official), which can collect GPU utilization, VRAM usage, temperature, and power consumption metrics. Combined with Grafana's GPU Dashboard template, you can quickly build a visualization panel to detect OOM and overheating risks in advance.

(1) Grafana Dashboard Metrics

Panel Metric Unit Alert
Request latency supportbot_request_duration_seconds s P95 > 3s
Request rate supportbot_requests_total req/min < 5 req/min
Error rate supportbot_errors_total % > 5%
GPU utilization DCGM_FI_DEV_GPU_UTIL % > 95%
VRAM usage DCGM_FI_DEV_FB_USED MB > 95%
Intent distribution supportbot_intent_count count —

▶ Example 5: Prometheus Configuration

YAML 
# monitoring/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "supportbot"
    static_configs:
      - targets: ["supportbot:8000"]
    metrics_path: /metrics

  - job_name: "ollama"
    static_configs:
      - targets: ["ollama:11434"]
    metrics_path: /metrics
    scheme: http

  - job_name: "node"
    static_configs:
      - targets: ["node-exporter:9100"]

rule_files:
  - "alert_rules.yml"

# alert_rules.yml
groups:
  - name: supportbot_alerts
    rules:
      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(supportbot_request_duration_seconds_bucket[5m])) > 3
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "SupportBot P95 latency above 3s"

      - alert: ServiceDown
        expr: up{job="supportbot"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "SupportBot service is down"

      - alert: HighErrorRate
        expr: rate(supportbot_errors_total[5m]) / rate(supportbot_requests_total[5m]) > 0.05
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "SupportBot error rate above 5%"

Output:

TEXT 
# Prometheus configuration loaded successfully
# Target status: 3 scrape targets all UP
# supportbot (UP) | ollama (UP) | node-exporter (UP)

7. Comprehensive Example: Go-Live Checklist and Operations Runbook

💡 Tip: An operations runbook's value becomes apparent when failures occur. Conduct a quarterly "fire drill" — deliberately shut down a service and follow the runbook to recover, verifying the document's accuracy and operability, and updating outdated steps.

PYTHON 
# ============================================
# Comprehensive: Production deployment checklist
# Final verification before going live
# ============================================

PRODUCTION_CHECKLIST = {
    "infrastructure": {
        "items": [
            "Docker Compose services all running (6/6)",
            "Ollama health check passing",
            "Chroma health check passing",
            "FastAPI health check passing",
            "Nginx SSL certificate valid (not expired)",
            "GPU acceleration verified (nvidia-smi inside container)",
        ],
        "commands": [
            "docker compose ps",
            "curl -f http://localhost:11434/api/tags",
            "curl -f http://localhost:8001/api/v1/heartbeat",
            "curl -f http://localhost:8000/health",
            "curl -f https://ai.globalshop.example.com/health",
        ]
    },
    "models": {
        "items": [
            "qwen2.5:7b pulled and verified",
            "llama3.2:3b pulled and verified",
            "nomic-embed-text pulled and verified",
            "supportbot Modelfile created and tested",
        ],
        "commands": [
            "docker exec supportbot-ollama ollama list",
            "docker exec supportbot-ollama ollama run supportbot 'Hello'",
        ]
    },
    "security": {
        "items": [
            "API Key authentication working (unauthorized request returns 401)",
            "Rate limiting working (burst above limit returns 429)",
            "Input injection filter tested",
            "Output content filter tested",
            "PII sanitization verified",
            "Ollama not accessible from internet (port 11434 blocked)",
        ]
    },
    "performance": {
        "items": [
            "Single request latency < 3s (P95)",
            "Concurrent 4 requests latency < 5s (P95)",
            "Throughput > 10 QPS",
            "RAG retrieval relevant (top-3 precision > 80%)",
            "Intent classification accuracy > 85%",
        ]
    },
    "monitoring": {
        "items": [
            "Prometheus scraping all services",
            "Grafana dashboards created and visible",
            "Alert rules configured and tested",
            "Log aggregation working",
        ]
    },
    "disaster_recovery": {
        "items": [
            "All configs in Git repository",
            "Chroma data backup scheduled (daily)",
            "Model files backed up on NFS/S3",
            "Recovery procedure documented and tested",
        ]
    }
}

def generate_runbook() -> str:
    """Generate operations runbook."""
    runbook = [
        "# SupportBot Operations Runbook\n",
        "## Service Overview",
        "- SupportBot API: https://ai.globalshop.example.com/v1/chat",
        "- Health Check: https://ai.globalshop.example.com/health",
        "- Grafana: http://localhost:3001 ⚠️ Change default admin password immediately!",
        "- Prometheus: http://localhost:9090\n",
        "## Common Operations\n",
        "### Restart a Service",
        "```bash",
        "docker compose restart supportbot  # Restart API",
        "docker compose restart ollama       # Restart Ollama",
        "```\n",
        "### Pull New Model",
        "```bash",
        "docker exec supportbot-ollama ollama pull model_name",
        "```\n",
        "### Rebuild Knowledge Base",
        "```bash",
        "docker exec supportbot-api python indexer.py /app/product_docs",
        "```\n",
        "### Check Logs",
        "```bash",
        "docker compose logs -f supportbot   # API logs",
        "docker compose logs -f ollama       # Ollama logs",
        "```\n",
        "### Emergency: Service Down",
        "1. Check: `docker compose ps`",
        "2. Restart: `docker compose restart <service>`",
        "3. Verify: `curl https://ai.globalshop.example.com/health`",
        "4. If Ollama OOM: restart with `OLLAMA_NUM_PARALLEL=2`",
        "5. Escalate: notify ops team\n",
        "## Performance Baseline",
        "| Metric | Target | Current |",
        "|:-------|:-------|:--------|",
        "| P95 Latency | < 3s | _fill after benchmark_ |",
        "| Throughput | > 10 QPS | _fill after benchmark_ |",
        "| Error Rate | < 1% | _fill after benchmark_ |",
        "| GPU Util | < 85% | _fill after benchmark_ |",
    ]
    return "\n".join(runbook)

# Generate and save
if __name__ == "__main__":
    print(generate_runbook())
    with open("RUNBOOK.md", "w") as f:
        f.write(generate_runbook())
    print("\nRunbook saved: RUNBOOK.md")

❓ FAQ

Q How long does the first startup take after deployment?
A Model pulling takes about 10-30 minutes (depending on network speed and model size). Subsequent restarts take about 30 seconds (models cached in Volume). Pull models in advance.
Q How do I obtain SSL certificates for production?
A Use certbot for free Let's Encrypt certificates, or Caddy for automatic HTTPS. Self-signed certificates work for internal networks.
Q How do I verify that security configuration is actually working?
A 1) Requests without API Key should return 401; 2) Rate-limited requests should return 429; 3) Port 11434 should not be externally accessible; 4) Injection attacks should be blocked. Test each one.
Q How long does it take to configure monitoring dashboards?
A Basic dashboards about 1 hour (import Node Exporter template + custom Ollama panel). Alert rules about 30 minutes. Total about 2 hours.
Q How do I continuously optimize SupportBot after going live?
A 1) Analyze logs for low-quality answers; 2) Add knowledge base documents; 3) Tune intent classification; 4) Expand Modelfile MESSAGE examples; 5) Regularly re-run benchmarks.
Q How do I roll back to a previous version?
A Git checkout configuration files → docker compose down → docker compose up -d. Model data in Volumes is unaffected. Chroma data can be restored from backup.

📖 Summary

📝 Exercises

  1. Basic (⭐): Deploy a minimal SupportBot version using Docker Compose (Ollama + FastAPI), and verify the /health endpoint returns OK.
  2. Intermediate (⭐⭐): Run the benchmark script, record P95 latency, throughput, and error rate, and compare against targets. If not meeting targets, adjust OLLAMA_NUM_PARALLEL or num_ctx and re-test.
  3. Advanced (⭐⭐⭐): Complete the full SupportBot production deployment — including Nginx security configuration + Prometheus/Grafana monitoring + alert rules + operations runbook — and execute the complete go-live Checklist verification.
Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站，几个人的小团队，精力有限。页面虽经检查，难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了，麻烦告诉我们，我们会在收到反馈后第一时间进行修复，再次感谢您的光临 🙏