Ollama: Phase 3 Comprehensive Practice
Phase 3 Comprehensive Practice is the leap from "individual features" to "system-level" — full-stack deployment, pipeline construction, and model orchestration in one go.
💡 Tip: Advanced practice integrates skills from all 17 previous lessons — Docker deployment (L15) + RAG pipeline (L14) + Multi-model routing (L17) + Quantization selection (L16) + LangChain orchestration (L13). Follow the "deploy first → build pipeline → add routing" sequence for incremental integration, rather than trying to build everything at once.
📋 Prerequisites: You should first master the following
- Lesson 1: Local AI Concepts and Environment
- Lesson 2: Ollama Installation and Environment Setup
- Lesson 3: CLI Basics
- Lesson 4: Model Management
- Lesson 5: REST API Basics
- Lesson 6: Phase 1 Comprehensive Practice
- Lesson 7: Python SDK Integration
- Lesson 8: Modelfile Custom Models
- Lesson 9: GPU and CUDA Configuration
- Lesson 10: Multimodal Models
- Lesson 11: OpenAI-Compatible API
- Lesson 12: Phase 2 Comprehensive Practice
- Lesson 13: LangChain Integration
- Lesson 14: RAG Pipeline
- Lesson 15: Docker Containerized Deployment
- Lesson 16: Model Quantization and Optimization
- Lesson 17: Multi-Model Orchestration
1. What You Will Learn
- Docker Compose deployment of Ollama + Chroma + WebUI full stack
- Building a RAG pipeline and comparing Embedding models
- Implementing multi-model router for automatic traffic splitting
- Quantized model quality comparison experiments
- SupportBot final architecture planning
2. A Real Story from a SaaS Entrepreneur
💡 Tip: After finalizing the architecture, use a "capability matrix table" (model × scenario × metrics) to document the selection rationale and performance baseline for each component. This provides traceability for future optimization or model changes, and helps explain technical decisions to the team.
ℹ️ Info: Phase 3 Comprehensive Practice is a direct prerequisite for Phase 4 (security/monitoring/production deployment). After completing this practice, you will have a running SupportBot V4 (Docker deployment + RAG + multi-model routing). Phase 4 will harden and optimize it further.
(1) The Pain Point: Disconnected Capabilities Cannot Work Together
Alice has learned LangChain, RAG, Docker, Quantization, and Orchestration, but the skills are scattered. She needs to integrate them into a unified SupportBot architecture and determine the final technology selection and deployment plan.
(2) The Solution: Final Architecture Determination
flowchart TD
A[User Query] --> B[Router<br/>3B Classifier]
B -->|Simple| C[FAQ RAG<br/>3B + Chroma]
B -->|Complex| D[Deep Answer<br/>8B qwen2.5]
C --> E[Response]
D --> E
E --> F[Docker Deploy<br/>Ollama + FastAPI + Chroma]
3. Exercise 1: Docker Compose Full-Stack Deployment
⚠️ Note: Docker Compose full-stack deployment involves 4+ services and GPU passthrough, which can cause various issues on first startup. Test each service individually first (start with
docker run ollama, then add chroma, then add app), and troubleshoot incrementally, rather than running docker compose up all at once and facing a pile of errors.
⚠️ Warning: Docker Compose full-stack deployment involves 4+ services and GPU passthrough. First-time startup may encounter various issues. Test each service individually first (start with
docker run ollama, then add chroma, then add app), and troubleshoot incrementally rather than running docker compose up all at once.
(1) Full-Stack Architecture
| Service | Image | Port | Responsibility |
|---|---|---|---|
| ollama | ollama/ollama | 11434 | Model inference |
| chroma | chromadb/chroma | 8001 | Vector storage |
| webui | open-webui | 3000 | Web interface |
| app | custom fastapi | 8000 | Business logic |
▶ Example 1: Full-Stack Deployment and Verification
YAML
# docker-compose.phase3.yml
version: "3.8"
services:
ollama:
image: ollama/ollama
ports: ["11434:11434"]
volumes: ["ollama_data:/root/.ollama"]
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:11434/api/tags"]
interval: 30s
retries: 5
chroma:
image: chromadb/chroma
ports: ["8001:8000"]
volumes: ["chroma_data:/chroma/chroma"]
environment: [ANONYMIZED_TELEMETRY=FALSE]
webui:
image: ghcr.io/open-webui/open-webui:main
ports: ["3000:8080"]
environment: [OLLAMA_BASE_URL=http://ollama:11434]
volumes: ["webui_data:/app/backend/data"]
depends_on:
ollama: {condition: service_healthy}
volumes:
ollama_data:
chroma_data:
webui_data:
BASH
# Deploy and verify
docker compose -f docker-compose.phase3.yml up -d
# Pull models
docker exec ollama ollama pull qwen2.5
docker exec ollama ollama pull nomic-embed-text
docker exec ollama ollama pull llama3.2:3b
# Verify all services
docker compose -f docker-compose.phase3.yml ps
curl http://localhost:11434/api/tags # Ollama
curl http://localhost:8001/api/v1/heartbeat # Chroma
curl http://localhost:3000 # WebUI
Output:
TEXT
Full-stack deployment verified successfully, all services running normally
4. Exercise 2: RAG Pipeline and Embedding Comparison
(1) Embedding Model Comparison Dimensions
| Dimension | nomic-embed-text | mxbai-embed-large | all-minilm |
|---|---|---|---|
| Dimensions | 768 | 1024 | 384 |
| Size | 274 MB | 670 MB | 46 MB |
| English quality | High | Highest | Medium |
| Chinese quality | Medium | Medium-high | Low |
| Speed | Fast | Medium | Very fast |
▶ Example 2: Embedding Model Effect Comparison
PYTHON
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
import numpy as np
def compare_embeddings(query: str, docs: list[str],
models: list[str]) -> dict:
"""Compare retrieval results across embedding models."""
results = {}
for model_name in models:
embeddings = OllamaEmbeddings(model=model_name)
vectorstore = Chroma.from_texts(
texts=docs, embedding=embeddings,
collection_name=f"test_{model_name.replace('-','_')}"
)
retrieved = vectorstore.similarity_search_with_score(query, k=3)
results[model_name] = [
{"text": doc.page_content[:80], "score": round(score, 4)}
for doc, score in retrieved
]
return results
# Test documents
docs = [
"Return policy: 30 days from purchase, original condition required.",
"Shipping: Free for orders over $50, 3-5 business days delivery.",
"Warranty: 1-year manufacturer warranty for all electronics.",
"International shipping available to 50+ countries with import duties.",
"Refund process: 5-7 business days after return received.",
"Product exchange: Available within 14 days for different size/color."
]
# Compare
results = compare_embeddings(
query="How do I get my money back?",
docs=docs,
models=["nomic-embed-text", "all-minilm"]
)
for model, hits in results.items():
print(f"\n{model}:")
for hit in hits:
print(f" [{hit['score']}] {hit['text']}")
Output:
TEXT
# Function defined successfully
5. Exercise 3: Multi-Model Router
(1) Complete Router Implementation
flowchart TD
A[Customer Question] --> B[Intent Classifier<br/>llama3.2:3b]
B --> C{Intent Category}
C -->|FAQ/Shipping/Order| D[FAQ RAG Pipeline<br/>3B + Chroma]
C -->|Product/Comparison| E[Deep Answer<br/>8B qwen2.5]
C -->|Complaint| F[Empathetic Response<br/>8B qwen2.5 + special prompt]
D --> G[Response]
E --> G
F --> G
▶ Example 3: Complete Router + RAG
PYTHON
import ollama
import json
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
class SmartRouter:
def __init__(self, chroma_path: str = "./chroma_kb"):
self.classifier = ChatOllama(model="llama3.2:3b", temperature=0)
self.small_model = ChatOllama(model="llama3.2:3b", temperature=0.3)
self.large_model = ChatOllama(model="qwen2.5", temperature=0.4)
self.embeddings = OllamaEmbeddings(model="nomic-embed-text")
try:
self.vectorstore = Chroma(
persist_directory=chroma_path,
embedding_function=self.embeddings
)
self.retriever = self.vectorstore.as_retriever(search_kwargs={"k": 3})
except Exception:
self.retriever = None
def classify(self, question: str) -> str:
response = self.classifier.invoke(
f"Classify as one word: faq, product, comparison, complaint, other.\nQuestion: {question}"
)
return response.content.strip().lower()
def answer(self, question: str) -> dict:
intent = self.classify(question)
if intent in ("faq", "other") and self.retriever:
# RAG pipeline for FAQ
docs = self.retriever.invoke(question)
context = "\n\n".join(d.page_content for d in docs)
response = self.small_model.invoke(
f"Context:\n{context}\n\nAnswer based on context: {question}"
)
model_used = "3B + RAG"
elif intent == "complaint":
response = self.large_model.invoke(
f"You are an empathetic customer service agent. "
f"Show understanding and offer solutions:\n{question}"
)
model_used = "8B (empathetic)"
else:
response = self.large_model.invoke(question)
model_used = "8B"
return {
"intent": intent,
"model": model_used,
"answer": response.content
}
# Usage
router = SmartRouter()
for q in ["What is the return policy?", "Compare 3 headphone models", "My order arrived damaged!"]:
result = router.answer(q)
print(f"[{result['intent']}|{result['model']}] {result['answer'][:80]}...")
Output:
TEXT
# Function defined successfully
6. Quantization Experiment
(1) Experiment Design
| Group | Model | Quantization | Size | Expected Quality |
|---|---|---|---|---|
| A | llama3.2:3b | Q4_K_M | ~2 GB | Baseline |
| B | qwen2.5 | Q4_K_M | ~5 GB | Higher |
| C | qwen2.5 | Q8_0 | ~8 GB | Highest |
▶ Example 4: Quantization Comparison Test
PYTHON
import ollama
import time
def quantization_benchmark(test_questions: list[str], models: list[str]):
"""Compare response quality across quantizations."""
results = {}
for model in models:
model_results = []
# Warm up
ollama.chat(model=model, messages=[{"role": "user", "content": "hi"}],
stream=False)
for q in test_questions:
start = time.time()
resp = ollama.chat(
model=model,
messages=[{"role": "user", "content": q}],
stream=False,
options={"temperature": 0.3}
)
elapsed = time.time() - start
model_results.append({
"question": q,
"answer": resp["message"]["content"][:200],
"latency_s": round(elapsed, 2)
})
results[model] = model_results
return results
# Run if models are available
# questions = [
# "What is the return policy for electronics?",
# "Translate to French: Free shipping on orders over $50",
# "Write a SQL query for top 5 customers"
# ]
# results = quantization_benchmark(questions, ["llama3.2:3b", "qwen2.5"])
Output:
TEXT
# Function defined successfully
7. Comprehensive Example: SupportBot Final Architecture Planning
PYTHON
# ============================================
# Comprehensive: SupportBot final architecture
# Integrates all Phase 3 capabilities
# ============================================
from dataclasses import dataclass, field
from typing import Optional
import json
@dataclass
class SupportBotArchitecture:
"""Final SupportBot architecture specification."""
name: str = "SupportBot V4"
version: str = "4.0"
components: dict = field(default_factory=lambda: {
"intent_classifier": {
"model": "llama3.2:3b",
"quantization": "Q4_K_M",
"purpose": "Classify customer intent in <1s",
"vram": "~2GB"
},
"faq_rag": {
"model": "llama3.2:3b",
"embedding": "nomic-embed-text",
"vector_db": "Chroma",
"purpose": "Answer FAQ from knowledge base",
"vram": "~2.3GB shared"
},
"deep_answer": {
"model": "qwen2.5:7b",
"quantization": "Q4_K_M",
"purpose": "Complex product questions and comparisons",
"vram": "~5GB"
},
"complaint_handler": {
"model": "qwen2.5:7b",
"system_prompt": "empathetic mode",
"purpose": "Handle complaints with empathy",
"vram": "shared with deep_answer"
},
"multimodal": {
"model": "llava",
"purpose": "Damage report image analysis",
"vram": "~5GB (on-demand)"
}
})
deployment: dict = field(default_factory=lambda: {
"platform": "Docker Compose",
"services": ["ollama", "chroma", "fastapi", "nginx"],
"vram_total": "8GB minimum, 12GB recommended",
"persistence": "Named volumes for models and Chroma"
})
routing_rules: dict = field(default_factory=lambda: {
"faq": "faq_rag (3B + RAG)",
"order_status": "faq_rag (3B + RAG)",
"shipping": "faq_rag (3B + RAG)",
"product_info": "deep_answer (8B)",
"comparison": "deep_answer (8B)",
"complaint": "complaint_handler (8B empathetic)",
"damage_report": "multimodal (llava)"
})
cost_analysis: dict = field(default_factory=lambda: {
"before_gpt4": "2,000 USD/month",
"after_local": "98 USD/month (hardware amortization + electricity)",
"saving": "1,902 USD/month (95% reduction)",
"payback_months": 1.1
})
def to_report(self) -> str:
"""Generate architecture report."""
report = [f"# {self.name} v{self.version} Architecture\n"]
report.append("## Components")
for name, spec in self.components.items():
report.append(f"- {name}: {spec['model']} - {spec['purpose']}")
report.append("\n## Deployment")
report.append(f"- Platform: {self.deployment['platform']}")
report.append(f"- Services: {', '.join(self.deployment['services'])}")
report.append(f"- VRAM: {self.deployment['vram_total']}")
report.append("\n## Routing Rules")
for intent, target in self.routing_rules.items():
report.append(f"- {intent} → {target}")
report.append("\n## Cost Analysis")
for key, value in self.cost_analysis.items():
report.append(f"- {key}: {value}")
return "\n".join(report)
# Generate report
arch = SupportBotArchitecture()
print(arch.to_report())
❓ FAQ
Q What are the hardware requirements for full-stack Docker deployment?
A Minimum 8GB VRAM + 16GB RAM + 50GB disk. Recommended 12GB VRAM + 32GB RAM. CPU mode requires 32GB RAM.
Q How do I improve RAG pipeline retrieval quality?
A 1) Better chunking strategy (preserve semantic boundaries); 2) Better Embedding model (mxbai-embed-large); 3) More document coverage; 4) MMR search for diversity.
Q What if the router classification accuracy is not high enough?
A Add keyword pre-filtering rules to reduce model burden. Use Few-shot examples to improve classification consistency. Consider upgrading the classifier to an 8B model.
Q How long does the quantization comparison experiment take?
A 3 models × 3 questions × 3 runs ≈ about 15 minutes. Make sure to warm up first to avoid cold-start bias.
Q What if the SupportBot final architecture needs changes?
A Architecture is living. Phase 4 will further adjust based on performance and security requirements. Phase 5's hands-on project will validate and refine the architecture.
Q What prerequisite knowledge is needed for Phase 4?
A Linux server operations basics, Nginx reverse proxy, Prometheus/Grafana monitoring concepts. Lessons 19-22 will cover each in detail.
📖 Summary
- Docker Compose full-stack deployment: Ollama + Chroma + WebUI + FastAPI
- Embedding comparison: nomic-embed-text for best general value, mxbai-embed-large for high precision
- Router + RAG combination: FAQ goes through RAG fast lane, complex questions go to 8B deep answers
- Quantization experiment validates: Q4_K_M quality loss < 3% in customer service scenarios
- SupportBot final architecture: Classifier + FAQ RAG + Deep Answer + Sentiment Processing + Multimodal
- All Phase 3 skills integrated into a unified architecture; Phase 4 enters the operations optimization phase
📝 Exercises
- Basic (⭐): Deploy an Ollama + Chroma + WebUI full stack with Docker Compose, and verify all three services are running normally.
- Intermediate (⭐⭐): Build a RAG pipeline, compare retrieval quality with two Embedding models, and record Top-3 hit rates.
- Advanced (⭐⭐⭐): Implement a complete SupportBot architecture — router + RAG + deep answering, and output an architecture design document and performance test report.
