English العربية Português 日本語

Ollama: Performance Tuning

Performance tuning is shifting gears for your local AI engine — from first-gear crawling to fifth-gear flying.

💡 Tip: OLLAMA_NUM_PARALLEL controls the number of parallel requests per model — setting it to 4 allows processing 4 requests simultaneously, increasing throughput. For 8GB VRAM, start testing from 2-4; for 16GB, try 4-8. Combine with OLLAMA_KEEP_ALIVE=30m to avoid frequent model unloading and reloading.

📋 Prerequisites: You should first master the following

1. What You Will Learn

2. A Real Story from a SaaS Entrepreneur

💡 Tip: OLLAMA_KEEP_ALIVE sets how long a model stays in memory. For high-concurrency scenarios, set it to 30m or longer to avoid frequent model unloading/reloading causing cold-start delays (5-30 seconds). For infrequent use, set 5m to release VRAM.

ℹ️ Info: num_ctx (context window size) directly affects KV Cache memory usage. An 8B model with num_ctx=2048 needs about 4GB VRAM, num_ctx=8192 needs about 6GB, and num_ctx=32768 needs about 12GB. Set it based on actual needs, don't blindly increase it.

(1) The Pain Point: Response Timeouts Under High Concurrency

Alice's SupportBot went live, and during peak hours, 10 concurrent requests caused response times to spike from 2 seconds to 30 seconds. Customers complained about "waiting too long," but Ollama only processes 1 concurrent request by default.

(2) The Solution: Concurrency and Context Optimization

Adjusting OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS, response time recovered to under 5 seconds:

BASH 
# Enable 4 parallel requests
export OLLAMA_NUM_PARALLEL=4
export OLLAMA_MAX_LOADED_MODELS=2

3. Key Performance Metrics

⚠️ Warning: Increasing OLLAMA_NUM_PARALLEL does not linearly increase throughput — each concurrent request shares GPU VRAM, and too many concurrent requests may cause OOM or a sharp drop in per-request speed. Start from 2 and test incrementally, observing VRAM usage and latency changes.

(1) Three Core Metrics

Metric Full Name Meaning Target Value
Tokens/s Tokens per second Generation speed GPU: 30+, CPU: 5+
TTFT Time to First Token First token latency < 500ms (GPU)
Memory VRAM/RAM usage Resource consumption < 90% capacity

(2) Ollama Performance Statistics

Performance data included in --verbose mode or non-streaming responses:

Field Meaning
total_duration Total time (nanoseconds)
load_duration Model loading time
prompt_eval_count Input token count
prompt_eval_duration Input processing time
eval_count Output token count
eval_duration Output generation time

▶ Example 1: Performance Metrics Collection

BASH 
# Collect performance metrics with verbose output
ollama run --verbose qwen2.5 "Explain AI in 50 words"

# Key output:
# total duration:       2500000000    # 2.5s total
# load duration:        500000000     # 0.5s model load
# prompt eval count:    15 token(s)
# prompt eval duration: 200000000     # 0.2s input processing
# prompt eval count:    15 token(s)   # = prompt_eval_speed: 75 tok/s
# eval count:           52 token(s)
# eval duration:        1800000000    # = eval_speed: 28.9 tok/s

Output:

TEXT 
I'm a helpful AI assistant running locally on your machine...
PYTHON 
import ollama
import time

def measure_performance(model: str, prompt: str) -> dict:
    start = time.time()
    response = ollama.chat(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        stream=False
    )
    total_time = time.time() - start
    content = response["message"]["content"]
    token_count = len(content) // 4  # Rough estimate

    return {
        "model": model,
        "total_time_s": round(total_time, 2),
        "estimated_tokens": token_count,
        "estimated_tok_per_s": round(token_count / total_time, 1) if total_time > 0 else 0
    }

result = measure_performance("qwen2.5", "Explain machine learning in 100 words")
print(result)

4. Concurrent Request Tuning

⚠️ Note: Excessive concurrency may cause OOM — OLLAMA_NUM_PARALLEL does not linearly increase throughput; each concurrent request shares GPU VRAM, and too many concurrent requests cause a sharp drop in per-request speed or even memory overflow. Start from 2 and test incrementally, observing VRAM usage and latency changes.

Variable Default Description Impact
OLLAMA_NUM_PARALLEL 1 Parallel requests per model ↑Throughput ↓Per-request speed
OLLAMA_MAX_LOADED_MODELS 1 Maximum simultaneously loaded models ↑Multi-model concurrency ↑VRAM
OLLAMA_KEEP_ALIVE 5m Model memory residence time ↑Avoids reloading ↓Memory reclamation

(2) Concurrency Configuration Decision

100% 
flowchart LR
    A[Concurrent Requests?] --> B{Peak QPS?}
    B -->|1-2| C[Default: NUM_PARALLEL=1]
    B -->|3-5| D[NUM_PARALLEL=4<br/>8GB VRAM minimum]
    B -->|6-10| E[NUM_PARALLEL=8<br/>16GB+ VRAM]
    B -->|10+| F[Multi-node<br/>Load Balancer]
Configuration VRAM Requirement Throughput Per-request Latency
NUM_PARALLEL=1 Lowest 1 req/s Shortest
NUM_PARALLEL=4 Medium 3-4 req/s Slight increase
NUM_PARALLEL=8 High 6-8 req/s Noticeable increase

▶ Example 2: Concurrency Configuration and Testing

BASH 
# Configure parallel processing
sudo systemctl edit ollama
# Add:
[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Output:

TEXT 
# Ollama command executed successfully
PYTHON 
import ollama
import asyncio
import time

async def concurrent_test(num_requests: int):
    """Test concurrent request handling."""
    client = ollama.AsyncClient()
    start = time.time()

    async def single_request(i: int):
        req_start = time.time()
        response = await client.chat(
            model="qwen2.5",
            messages=[{"role": "user", "content": f"Say hello {i}"}],
            stream=False,
            options={"temperature": 0.1}
        )
        return time.time() - req_start

    tasks = [single_request(i) for i in range(num_requests)]
    latencies = await asyncio.gather(*tasks)

    total = time.time() - start
    print(f"Concurrent: {num_requests} requests")
    print(f"Total time: {total:.2f}s")
    print(f"Avg latency: {sum(latencies)/len(latencies):.2f}s")
    print(f"Throughput: {num_requests/total:.1f} req/s")

asyncio.run(concurrent_test(4))

5. Context Window Management

(1) num_ctx Impact on Memory

num_ctx KV Cache (8B Q4_M) Total VRAM Needed Use Case
2048 ~1 GB ~6 GB Short conversations (default)
4096 ~2 GB ~7 GB Medium conversations
8192 ~4 GB ~9 GB RAG retrieval
32768 ~16 GB ~21 GB Long documents
⚠️ Note: KV Cache grows linearly with num_ctx. An 8B model with num_ctx=32768 cannot run on 8GB VRAM; it needs 21GB+ VRAM.

(2) num_ctx Optimization Strategies

Strategy Method Effect
Set on demand Chat uses 2048, RAG uses 8192 Reduce memory waste
Sliding window Keep only the last N conversation rounds Control input length
Summary compression Periodically compress early conversations Save context space
RAG curation Reduce top-k from 5 to 3 Reduce input tokens

▶ Example 3: num_ctx Comparison Test

PYTHON 
import ollama
import time

def test_context_sizes(model: str, prompt: str, context_sizes: list[int]):
    """Test performance with different context window sizes."""
    for ctx in context_sizes:
        start = time.time()
        response = ollama.chat(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            stream=False,
            options={"num_ctx": ctx, "temperature": 0.3}
        )
        elapsed = time.time() - start
        tokens = len(response["message"]["content"]) // 4
        speed = tokens / elapsed if elapsed > 0 else 0
        print(f"num_ctx={ctx}: {elapsed:.2f}s, ~{speed:.1f} tok/s")

test_context_sizes("qwen2.5", "Explain AI briefly", [2048, 4096, 8192])

Output:

TEXT 
# Function defined successfully

6. Batch Processing and Benchmarking

(1) Batch Processing Strategies

Strategy Description Use Case
Serial batch Process requests one by one Simple and reliable
Parallel batch asyncio concurrency High throughput
Queue scheduling FastAPI + queue Production environment

▶ Example 4: Batch Benchmark Script

PYTHON 
import ollama
import time
import asyncio
import json
from datetime import datetime

class BenchmarkRunner:
    def __init__(self, model: str = "qwen2.5"):
        self.model = model
        self.results = []

    def warmup(self, runs: int = 2):
        for _ in range(runs):
            ollama.chat(model=self.model,
                        messages=[{"role": "user", "content": "warmup"}],
                        stream=False)

    def single_benchmark(self, prompt: str) -> dict:
        start = time.time()
        response = ollama.chat(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            stream=False,
            options={"temperature": 0.3}
        )
        elapsed = time.time() - start
        content = response["message"]["content"]
        return {
            "prompt_length": len(prompt),
            "response_length": len(content),
            "estimated_tokens": len(content) // 4,
            "total_time_s": round(elapsed, 2),
            "tok_per_s": round(len(content) / 4 / elapsed, 1) if elapsed > 0 else 0
        }

    async def concurrent_benchmark(self, prompt: str, concurrency: int) -> dict:
        client = ollama.AsyncClient()
        start = time.time()
        tasks = [
            client.chat(model=self.model,
                       messages=[{"role": "user", "content": prompt}],
                       stream=False, options={"temperature": 0.3})
            for _ in range(concurrency)
        ]
        await asyncio.gather(*tasks)
        total = time.time() - start
        return {
            "concurrency": concurrency,
            "total_time_s": round(total, 2),
            "throughput_rps": round(concurrency / total, 1)
        }

    def run_full_benchmark(self):
        self.warmup()
        prompts = [
            "Hello",
            "Explain AI in 3 sentences",
            "Write a product description for wireless headphones"
        ]
        print("=== Single Request Benchmark ===")
        for p in prompts:
            result = self.single_benchmark(p)
            print(f"  {result['estimated_tokens']}tok, {result['tok_per_s']}tok/s, {result['total_time_s']}s")

        print("\n=== Concurrent Benchmark ===")
        for c in [1, 2, 4]:
            result = asyncio.run(self.concurrent_benchmark("Hello", c))
            print(f"  Concurrency {c}: {result['throughput_rps']} req/s")

# Run
runner = BenchmarkRunner("qwen2.5")
runner.run_full_benchmark()

Output:

TEXT 
=== Single Request Benchmark ===

=== Concurrent Benchmark ===

▶ Example 5: Performance Baseline Establishment

BASH 
#!/bin/bash
# Establish performance baseline

MODEL="qwen2.5"
DATE=$(date +%Y%m%d)
REPORT="baseline_${DATE}.txt"

echo "=== Performance Baseline ===" > "$REPORT"
echo "Date: $(date)" >> "$REPORT"
echo "Model: $MODEL" >> "$REPORT"
echo "GPU: $(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null || echo 'CPU')" >> "$REPORT"
echo "" >> "$REPORT"

# Single request benchmark
echo "## Single Request ##" >> "$REPORT"
for i in {1..5}; do
    start=$(date +%s%N)
    curl -s http://localhost:11434/api/chat -d "{
        \"model\": \"$MODEL\",
        \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],
        \"stream\": false
    }" > /dev/null
    end=$(date +%s%N)
    elapsed=$(( (end - start) / 1000000 ))
    echo "  Run $i: ${elapsed}ms" >> "$REPORT"
done

echo "" >> "$REPORT"
echo "## GPU Utilization ##" >> "$REPORT"
nvidia-smi --query-gpu=utilization.gpu,memory.used,memory.total --format=csv >> "$REPORT" 2>/dev/null || echo "CPU mode" >> "$REPORT"

echo "Baseline saved: $REPORT"

Output:

TEXT 
{"status":"ok","data":{}}

7. Comprehensive Example: SupportBot Performance Tuning Report

PYTHON 
# ============================================
# Comprehensive: Performance tuning report
# Full benchmark + optimization recommendations
# ============================================

import ollama
import time
import asyncio
import json

class PerformanceTuner:
    def __init__(self, model: str = "qwen2.5"):
        self.model = model

    def run_diagnostics(self) -> dict:
        """Run full performance diagnostics."""
        # Warm up
        ollama.chat(model=self.model,
                    messages=[{"role": "user", "content": "warmup"}],
                    stream=False)

        # Single request test
        start = time.time()
        resp = ollama.chat(
            model=self.model,
            messages=[{"role": "user", "content": "Hello"}],
            stream=False, options={"temperature": 0.3}
        )
        single_latency = time.time() - start

        # Context size test
        ctx_results = {}
        for ctx in [2048, 4096, 8192]:
            start = time.time()
            ollama.chat(
                model=self.model,
                messages=[{"role": "user", "content": "Hello"}],
                stream=False, options={"num_ctx": ctx, "temperature": 0.3}
            )
            ctx_results[ctx] = round(time.time() - start, 2)

        return {
            "model": self.model,
            "single_latency_s": round(single_latency, 2),
            "context_sizes": ctx_results,
            "recommendations": self._generate_recommendations(single_latency)
        }

    def _generate_recommendations(self, latency: float) -> list[str]:
        recs = []
        if latency > 5:
            recs.append("High latency detected. Enable GPU acceleration or use smaller model.")
        if latency > 2:
            recs.append("Set OLLAMA_NUM_PARALLEL=4 for concurrent requests.")
        recs.append("Use num_ctx=2048 for chat, num_ctx=8192 for RAG.")
        recs.append("Set OLLAMA_KEEP_ALIVE=30m to avoid model reloading.")
        return recs

    def generate_report(self) -> str:
        diag = self.run_diagnostics()
        report = [
            f"# Performance Tuning Report",
            f"Model: {diag['model']}",
            f"Single request latency: {diag['single_latency_s']}s",
            f"\n## Context Window Impact:",
        ]
        for ctx, latency in diag["context_sizes"].items():
            report.append(f"  num_ctx={ctx}: {latency}s")
        report.append("\n## Recommendations:")
        for r in diag["recommendations"]:
            report.append(f"  - {r}")
        return "\n".join(report)

tuner = PerformanceTuner("qwen2.5")
print(tuner.generate_report())

❓ FAQ

Q What should OLLAMA_NUM_PARALLEL be set to?
A 2-4 for 8GB VRAM, 4-8 for 16GB, 8 for 24GB. Too many parallel requests increase per-request latency and risk OOM. Start testing from 4.
Q What if TTFT is too high?
A 1) Ensure OLLAMA_KEEP_ALIVE is long enough (avoid cold starts); 2) Reduce num_ctx; 3) Use a smaller model; 4) Ensure GPU acceleration is working.
Q How do I reduce model loading time?
A Set OLLAMA_KEEP_ALIVE=30m or longer to keep the model resident in memory. First load is unavoidable; subsequent requests respond in seconds.
Q What if OOM occurs during concurrent requests?
A Reduce OLLAMA_NUM_PARALLEL; use smaller num_ctx; reduce OLLAMA_MAX_LOADED_MODELS; or add more VRAM.
Q What if benchmark results are unstable?
A 1) Warm up 2-3 times to eliminate cold starts; 2) Close other GPU programs; 3) Take 5+ runs and average; 4) Fix random seed.
Q How do I monitor production environment performance?
A Lesson 21 will detail the Prometheus + Grafana monitoring stack. For now, use --verbose and custom scripts to collect metrics.

📖 Summary

📝 Exercises

  1. Basic (⭐): Run --verbose mode, record Tokens/s, TTFT, and other metrics for 3 inference runs, and establish your performance baseline.
  2. Intermediate (⭐⭐): Configure OLLAMA_NUM_PARALLEL=4, run concurrent tests, and compare throughput and latency changes before and after concurrency.
  3. Advanced (⭐⭐⭐): Write a complete performance tuning report — including single-request benchmarks, concurrent tests, num_ctx comparison, and optimization recommendations — in Markdown format.
Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站，几个人的小团队，精力有限。页面虽经检查，难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了，麻烦告诉我们，我们会在收到反馈后第一时间进行修复，再次感谢您的光临 🙏