Ollama: Multi-Model Orchestration
Multi-model orchestration is AI teamwork — small models lead the charge, large models provide backbone, each playing its role.
💡 Tip: The core of multi-model routing strategy is "assign by task" — simple FAQs use a 3B small model (1 second response, low GPU consumption), complex reasoning uses an 8B large model (5 seconds response, high precision). You can also "assign by load" — use small models during peak hours to handle more requests, and switch to large models during low-traffic periods for higher quality. Routing 80% of requests to small models can reduce average latency by 50%.
📋 Prerequisites: You should first master the following
- Lesson 13: LangChain Integration
1. What You Will Learn
- Multi-model architecture patterns: Router / Cascade / Parallel Voting
- Router pattern: small model classifies → large model answers in depth
- Cascade pattern: rough draft → refine → format pipeline
- Python asyncio for concurrent calls to multiple Ollama instances
- Alice's SupportBot V4: Router-based traffic splitting
2. A Real Story from a SaaS Entrepreneur
ℹ️ Info: In multi-model orchestration, each model needs to be independently loaded into VRAM. Two models (3B + 8B) residing simultaneously require about 10GB VRAM. If VRAM is insufficient, Ollama automatically unloads inactive models, but switching incurs a 5-30 second reload delay.
(1) The Pain Point: Large Models Waste Resources on Simple Questions
Alice found that 80% of SupportBot questions ("What is the return policy?" "How do I check shipping?") could be answered by a small model, but routing everything through a large model wastes GPU resources and is slower.
(2) The Solution: Router Pattern for Traffic Splitting
A small model (3B) classifies intent in 1 second; simple questions get direct answers, complex questions are routed to a large model (8B):
PYTHON
# Router pattern: small model classifies, large model handles complex
intent = small_model.classify(question) # 1 second
if intent == "simple":
answer = small_model.answer(question) # 2 seconds
else:
answer = large_model.answer(question) # 5 seconds
3. Three Orchestration Patterns
⚠️ Note: The Parallel Voting pattern has the highest reliability but also the highest cost — each request requires calling 3-5 models, multiplying GPU computation. Only use it for critical decision scenarios (e.g., medical advice, legal judgment); the Router pattern is sufficient for everyday customer service.
💡 Tip: The Router pattern is the most cost-effective orchestration method — 80% of simple requests use a 3B model (1 second response), and only 20% of complex requests are routed to an 8B model. Overall average response time drops from 5 seconds to 2 seconds, with 60% lower GPU utilization.
(1) Pattern Comparison
| Pattern | Flow | Use Case | Latency | Cost |
|---|---|---|---|---|
| Router | Classify → Dispatch | Customer service routing, task classification | Low | Low |
| Cascade | Draft → Refine → Format | Code generation, content creation | Medium | Medium |
| Parallel Voting | Multi-model → Vote/Merge | High-reliability decisions | High | High |
flowchart TD
subgraph Router
R1[Input] --> R2[Classifier<br/>3B Model]
R2 -->|Simple| R3[Small Model Answer]
R2 -->|Complex| R4[Large Model Answer]
end
subgraph Cascade
C1[Input] --> C2[Draft<br/>3B Model]
C2 --> C3[Refine<br/>8B Model]
C3 --> C4[Format<br/>3B Model]
end
subgraph Voting
V1[Input] --> V2[Model A]
V1 --> V3[Model B]
V1 --> V4[Model C]
V2 --> V5[Majority Vote]
V3 --> V5
V4 --> V5
end
4. Router Pattern
(1) Router Architecture
sequenceDiagram
participant U as User
participant C as Classifier (3B)
participant S as Small Model (3B)
participant L as Large Model (8B)
U->>C: "I want a refund"
C-->>C: Intent: refund (simple)
C->>S: Route to small model
S-->>U: Refund policy answer
U->>C: "Compare warranty terms for 3 products"
C-->>C: Intent: comparison (complex)
C->>L: Route to large model
L-->>U: Detailed comparison
(2) Intent Classification Label Design
| Label | Complexity | Routing Target | Example Question |
|---|---|---|---|
| faq | Simple | Small model | "What is the return policy?" |
| order_status | Simple | Small model | "Where is order #12345?" |
| product_info | Medium | Medium model | "Does this headphone support Bluetooth 5.3?" |
| comparison | Complex | Large model | "Compare the 3 headphone models" |
| complaint | Complex | Large model | "The item I received is completely different from the description" |
▶ Example 1: Router Pattern Implementation
PYTHON
import ollama
import json
from dataclasses import dataclass
from typing import Optional
@dataclass
class ModelRouter:
classifier_model: str = "llama3.2:3b"
small_model: str = "llama3.2:3b"
large_model: str = "qwen2.5"
INTENTS = {
"faq": "simple",
"order_status": "simple",
"product_info": "medium",
"comparison": "complex",
"complaint": "complex"
}
def classify(self, question: str) -> dict:
response = ollama.chat(
model=self.classifier_model,
messages=[{
"role": "user",
"content": f"""Classify this customer question.
Return JSON: {{"intent": "faq|order_status|product_info|comparison|complaint", "complexity": "simple|medium|complex"}}
Question: {question}"""
}],
format="json",
stream=False,
options={"temperature": 0.1}
)
return json.loads(response["message"]["content"])
def route(self, question: str, system: str = "") -> str:
intent_data = self.classify(question)
complexity = intent_data.get("complexity", "simple")
model = self.small_model if complexity == "simple" else self.large_model
messages = []
if system:
messages.append({"role": "system", "content": system})
messages.append({"role": "user", "content": question})
response = ollama.chat(model=model, messages=messages,
stream=False, options={"temperature": 0.3})
return response["message"]["content"], model, intent_data
# Usage
router = ModelRouter()
questions = [
"What is the return policy?",
"Compare the 3 wireless headphones you sell",
"Where is order #88765?"
]
for q in questions:
answer, model_used, intent = router.route(q, system="You are SupportBot.")
print(f"Q: {q}")
print(f"Intent: {intent}, Model: {model_used}")
print(f"A: {answer[:100]}...\n")
Output:
TEXT
# Function defined successfully
5. Cascade Pattern
(1) Cascade Architecture Details
| Stage | Model | Task | Parameters |
|---|---|---|---|
| Draft | 3B | Quick draft | temperature=0.5 |
| Refine | 8B | Deep refinement | temperature=0.3 |
| Format | 3B | Format output | temperature=0.1 |
▶ Example 2: Cascade Content Generation
PYTHON
import ollama
class CascadePipeline:
def __init__(self):
self.draft_model = "llama3.2:3b"
self.refine_model = "qwen2.5"
self.format_model = "llama3.2:3b"
def generate(self, topic: str) -> dict:
# Stage 1: Draft
draft = ollama.chat(
model=self.draft_model,
messages=[{"role": "user",
"content": f"Write a brief draft about: {topic}"}],
stream=False, options={"temperature": 0.5}
)["message"]["content"]
# Stage 2: Refine
refined = ollama.chat(
model=self.refine_model,
messages=[
{"role": "system", "content": "Improve the following text. Add details and fix errors."},
{"role": "user", "content": draft}
],
stream=False, options={"temperature": 0.3}
)["message"]["content"]
# Stage 3: Format
formatted = ollama.chat(
model=self.format_model,
messages=[
{"role": "system", "content": "Format this text as a professional product description with bullet points."},
{"role": "user", "content": refined}
],
stream=False, options={"temperature": 0.1}
)["message"]["content"]
return {"draft": draft, "refined": refined, "formatted": formatted}
pipeline = CascadePipeline()
result = pipeline.generate("wireless noise-cancelling headphones")
print("=== Final Output ===")
print(result["formatted"])
Output:
TEXT
=== Final Output ===
6. Parallel Voting and asyncio Concurrency
(1) Parallel Voting Architecture
| Pattern | Model Count | Decision Method | Use Case |
|---|---|---|---|
| Majority vote | 3+ | Take majority-consensus answer | Factual judgment |
| Weighted average | 2+ | Weight by model quality | Scoring tasks |
| Best selection | 2+ | Choose most detailed/credible | Open-ended questions |
▶ Example 3: asyncio Concurrent Invocation
PYTHON
import asyncio
import ollama
from dataclasses import dataclass
@dataclass
class ParallelVoter:
models: list[str] = None
def __post_init__(self):
if self.models is None:
self.models = ["llama3.2:3b", "qwen2.5", "mistral"]
async def query_model(self, model: str, question: str) -> dict:
client = ollama.AsyncClient()
response = await client.chat(
model=model,
messages=[{"role": "user", "content": question}],
stream=False,
options={"temperature": 0.3}
)
return {"model": model, "answer": response["message"]["content"]}
async def vote(self, question: str) -> dict:
tasks = [self.query_model(m, question) for m in self.models]
results = await asyncio.gather(*tasks)
# Simple voting: check for consensus
answers = [r["answer"] for r in results]
return {
"question": question,
"results": results,
"consensus": len(set(a[:50] for a in answers)) == 1
}
# Usage
async def main():
voter = ParallelVoter()
result = await voter.vote("Is 2+2 equal to 4?")
for r in result["results"]:
print(f"{r['model']}: {r['answer'][:80]}")
asyncio.run(main())
Output:
TEXT
# Function defined successfully
▶ Example 4: Weighted Ensemble
PYTHON
import ollama
def weighted_ensemble(question: str, models: list[dict]) -> str:
"""Query multiple models and select the best response."""
responses = []
for m in models:
resp = ollama.chat(
model=m["name"],
messages=[{"role": "user", "content": question}],
stream=False,
options={"temperature": m.get("temperature", 0.3)}
)
responses.append({
"model": m["name"],
"answer": resp["message"]["content"],
"weight": m.get("weight", 1.0),
"length": len(resp["message"]["content"])
})
# Select longest response from highest-weight model
responses.sort(key=lambda x: x["weight"] * x["length"], reverse=True)
return responses[0]["answer"]
models = [
{"name": "qwen2.5", "weight": 1.5, "temperature": 0.3},
{"name": "llama3.2:3b", "weight": 0.8, "temperature": 0.5},
]
result = weighted_ensemble("Explain the benefits of local AI", models)
print(result)
Output:
TEXT
# Function defined successfully
7. Comprehensive Example: SupportBot V4 Router System
PYTHON
# ============================================
# Comprehensive: SupportBot V4 with Router
# Multi-model routing for e-commerce support
# ============================================
import ollama
import json
import time
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class SupportBotV4:
classifier: str = "llama3.2:3b"
simple_model: str = "llama3.2:3b"
complex_model: str = "qwen2.5"
system_prompt: str = "You are SupportBot for GlobalShop e-commerce. Be polite and concise."
INTENT_MAP: dict = field(default_factory=lambda: {
"faq": "simple",
"order_status": "simple",
"shipping": "simple",
"product_info": "complex",
"comparison": "complex",
"complaint": "complex",
"refund": "complex"
})
def classify_intent(self, question: str) -> tuple[str, str]:
"""Classify intent and determine complexity."""
response = ollama.chat(
model=self.classifier,
messages=[{
"role": "user",
"content": f"""Classify intent (one word): {question}
Categories: faq, order_status, shipping, product_info, comparison, complaint, refund
Reply with just the category word."""
}],
stream=False,
options={"temperature": 0.0}
)
intent = response["message"]["content"].strip().lower()
for key in self.INTENT_MAP:
if key in intent:
return key, self.INTENT_MAP[key]
return "faq", "simple"
def answer(self, question: str) -> dict:
"""Route question and generate answer."""
start = time.time()
intent, complexity = self.classify_intent(question)
model = self.simple_model if complexity == "simple" else self.complex_model
response = ollama.chat(
model=model,
messages=[
{"role": "system", "content": self.system_prompt},
{"role": "user", "content": question}
],
stream=False,
options={"temperature": 0.3 if complexity == "simple" else 0.4}
)
elapsed = time.time() - start
return {
"question": question,
"intent": intent,
"complexity": complexity,
"model_used": model,
"answer": response["message"]["content"],
"latency_s": round(elapsed, 2)
}
def batch_process(self, questions: list[str]) -> list[dict]:
"""Process multiple questions and show routing stats."""
results = [self.answer(q) for q in questions]
simple_count = sum(1 for r in results if r["complexity"] == "simple")
avg_simple = sum(r["latency_s"] for r in results if r["complexity"] == "simple") / max(simple_count, 1)
complex_count = len(results) - simple_count
avg_complex = sum(r["latency_s"] for r in results if r["complexity"] == "complex") / max(complex_count, 1)
print(f"=== Routing Stats ===")
print(f"Simple: {simple_count}/{len(results)} (avg {avg_simple:.1f}s)")
print(f"Complex: {complex_count}/{len(results)} (avg {avg_complex:.1f}s)")
print(f"Cost saving: ~{simple_count * 60}% of queries use small model")
return results
# Usage
if __name__ == "__main__":
bot = SupportBotV4()
questions = [
"What is the return policy?",
"Where is my order #88765?",
"Do you ship internationally?",
"Compare the 3 headphone models in detail",
"I received a completely wrong item, this is unacceptable!",
"What are the warranty terms for electronics?",
"Analyze which laptop is best for a graphic designer under $1000",
]
results = bot.batch_process(questions)
for r in results:
print(f"\n[{r['complexity'].upper()}|{r['model_used']}] {r['intent']}")
print(f" Q: {r['question']}")
print(f" A: {r['answer'][:100]}... ({r['latency_s']}s)")
❓ FAQ
Q What if the router classification is inaccurate?
A Use a better classifier. Small models (3B) have about 85% classification accuracy, 8B models about 92%. You can also pre-filter with keyword matching, letting the model handle only ambiguous cases.
Q Is the cascade pattern better than using a large model directly?
A Depends on the scenario. Cascade total latency = sum of all stages, but each stage is faster. Suitable for scenarios needing draft + refinement (writing, code). Simple Q&A doesn't need cascading.
Q Does parallel voting consume 3x GPU resources?
A Yes. 3 models running inference simultaneously need 3x VRAM. You can call 3 models sequentially to avoid OOM, but lose the parallel advantage.
Q Is there a big speed difference between asyncio concurrency and sequential calls?
A Depends on GPU. With a single GPU, Ollama queues internally, so concurrency doesn't help. With multiple GPUs or CPU+GPU mix, asyncio can run different models concurrently.
Q How do I evaluate the router's traffic-splitting effectiveness?
A Track routing ratio, per-category latency, and user satisfaction. Target: 80% of requests to small models, 50%+ average latency reduction, no drop in satisfaction.
Q Should small and large models use the same System Prompt?
A Recommended to keep them consistent for a unified user experience. But you can fine-tune for different complexity levels — the large model's Prompt can be more detailed.
📖 Summary
- Three orchestration patterns: Router (traffic splitting), Cascade (pipeline), Parallel Voting (high reliability)
- Router pattern: 3B classification + 8B deep answers, 80% of requests to small model, average latency reduced by 50%
- Cascade pattern: Draft(3B) → Refine(8B) → Format(3B), suitable for content creation
- asyncio for concurrent multi-model calls; note queuing on single GPU
- SupportBot V4 uses router-based splitting: 2s for simple questions, 5s for complex
- Evaluate router effectiveness by routing ratio, latency improvement, and user satisfaction
📝 Exercises
- Basic (⭐): Implement a simple two-way router — FAQ goes to small model, everything else to large model. Test with 5 questions.
- Intermediate (⭐⭐): Implement a three-stage cascade pipeline — draft → refine → format. Compare quality with single-model direct output.
- Advanced (⭐⭐⭐): Implement a complete router system for SupportBot V4, including intent classification, routing statistics, latency monitoring, and output a routing effectiveness report.
