Ollama: RAG Pipeline
RAG gives AI a knowledge vault — instead of guessing answers, it retrieves truth from documents.
💡 Tip: The core principle of RAG is "retrieve first, then generate" — the model answers only based on retrieved document fragments, not from memory. This greatly reduces "hallucinations" (fabricating information), but retrieval quality directly determines answer quality. Chunking strategy and Embedding model selection are key.
📋 Prerequisites: You should first master the following
- Lesson 13: LangChain Integration
1. What You Will Learn
- RAG architecture: Document loading → Chunking → Embedding → Vector storage → Retrieval → Generation
- Using Ollama Embeddings models
- Vector database selection: Chroma / FAISS / Qdrant
- Document chunking strategies and parameter tuning
- Alice's SupportBot V3: Product knowledge base RAG
2. A Real Story from a SaaS Entrepreneur
⚠️ Warning: RAG cannot eliminate hallucinations — it can only significantly reduce their probability. If retrieved document fragments are irrelevant or insufficient, the model may still "fabricate" answers. Key strategy: constrain the model to answer only within the scope of retrieval results, and add to the System Prompt: "If the relevant information is not in the documents, respond with 'I cannot confirm that'."
(1) The Pain Point: The Model Doesn't Know Product Details
Alice's SupportBot can handle general customer service, but doesn't know specific product return policies, warranty terms, or pricing details. The model frequently "fabricates" information, leading to constant customer complaints.
(2) The Solution: RAG Enables the Model to Answer Based on Real Documents
Import product manuals, FAQs, and policy documents into a vector database, retrieve relevant fragments, and let the model generate answers based on facts:
PYTHON
# RAG: Retrieve relevant docs, then generate answer
docs = retriever.invoke("What is the warranty for headphones?")
answer = llm.invoke(f"Based on: {docs}\nAnswer: What is the warranty?")
3. RAG Architecture in Detail
💡 Tip: The core principle of RAG is "retrieve first, then generate" — the model answers only based on retrieved document fragments, not from memory. This greatly reduces "hallucinations" (fabricating information), but retrieval quality directly determines answer quality. Chunking strategy and Embedding model selection are key.
(1) End-to-End Pipeline Flow
flowchart LR
A[Documents] --> B[Loader]
B --> C[Splitter]
C --> D[Embedding Model<br/>nomic-embed-text]
D --> E[Vector Store<br/>Chroma/FAISS]
F[User Query] --> G[Query Embedding]
G --> E
E --> H[Top-K Chunks]
H --> I[LLM Generation<br/>qwen2.5]
I --> J[Answer]
(2) Stage-by-Stage Breakdown
| Stage | Tool | Input | Output |
|---|---|---|---|
| Load | PyPDFLoader / TextLoader | PDF/MD/TXT | Document list |
| Chunk | RecursiveCharacterTextSplitter | Document | Chunk list |
| Embed | OllamaEmbeddings | Chunk text | Vector (768d) |
| Store | Chroma / FAISS | Vector + text | Index |
| Retrieve | Similarity Search | Query vector | Top-K Chunks |
| Generate | ChatOllama | Query + Chunks | Answer |
4. Ollama Embeddings
⚠️ Note: Embedding model selection directly affects retrieval quality — nomic-embed-text (768 dimensions) performs well in English but is average in Chinese, mxbai-embed-large (1024 dimensions) has higher precision but is double the size. For Chinese scenarios, test multiple models and pick the best — don't assume "bigger is always better."
(1) Embedding Model Comparison
| Model | Dimensions | Size | Speed | Use Case |
|---|---|---|---|---|
| nomic-embed-text | 768 | 274 MB | Fast | General English |
| mxbai-embed-large | 1024 | 670 MB | Medium | High precision |
| all-minilm | 384 | 46 MB | Very fast | Lightweight |
(2) Embedding Model Selection Guide
| Scenario | Recommended Model | Reason |
|---|---|---|
| General RAG | nomic-embed-text | Best value |
| High-precision retrieval | mxbai-embed-large | Higher dimensions, better discrimination |
| Resource-constrained | all-minilm | Smallest and fastest |
| Chinese-dominant | nomic-embed-text | Acceptable Chinese performance |
▶ Example 1: Ollama Embeddings Basics
PYTHON
from langchain_ollama import OllamaEmbeddings
# Initialize embeddings model
embeddings = OllamaEmbeddings(model="nomic-embed-text")
# Embed a single text
vector = embeddings.embed_query("What is the return policy?")
print(f"Dimension: {len(vector)}") # 768
# Embed multiple texts
vectors = embeddings.embed_documents([
"Free shipping on orders over $50",
"30-day return policy for all products",
"Warranty covers manufacturing defects for 1 year"
])
print(f"Embedded {len(vectors)} documents")
Output:
TEXT
# Execution successful
5. Document Loading and Chunking
(1) Document Loaders
| Loader | Format | Installation |
|---|---|---|
| PyPDFLoader | pip install pypdf | |
| TextLoader | TXT | Built-in |
| UnstructuredMarkdownLoader | Markdown | pip install unstructured |
| CSVLoader | CSV | Built-in |
| UnstructuredHTMLLoader | HTML/URL | pip install unstructured |
(2) Chunking Strategy Comparison
| Strategy | Parameters | Suitable Documents | Pros/Cons |
|---|---|---|---|
| Fixed size | chunk_size=500, overlap=50 | General | Simple but may break semantics |
| Recursive character | chunk_size=500, overlap=50, separators | Structured documents | Preserves paragraph boundaries |
| Semantic chunking | similarity_threshold | Long texts | Best quality but computationally expensive |
💡 Tip: RecursiveCharacterTextSplitter is the preferred choice. chunk_size=500-1000 and overlap=50-100 are empirically optimal values.
▶ Example 2: Document Loading and Chunking
PYTHON
from langchain_community.document_loaders import TextLoader, PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Load text document
loader = TextLoader("product_faq.txt")
docs = loader.load()
print(f"Loaded {len(docs)} document(s)")
# Split into chunks
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")
print(f"First chunk preview: {chunks[0].page_content[:100]}...")
Output:
TEXT
# Execution successful
6. Vector Database Selection
(1) Three Major Vector Stores Compared
| Dimension | Chroma | FAISS | Qdrant |
|---|---|---|---|
| Type | Embedded | In-memory | Client/Server |
| Persistence | ✅ Local files | ❌ Memory only | ✅ Disk/Cloud |
| Installation | pip install | pip install + faiss-cpu | Docker / pip |
| Best for | Dev/small scale | High-speed retrieval | Production |
| Filtering | ✅ Metadata filtering | ❌ | ✅ Advanced filtering |
| Distributed | ❌ | ❌ | ✅ |
▶ Example 3: Chroma Vector Storage and Retrieval
PYTHON
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
# Prepare documents
documents = [
"Return policy: 30 days from purchase, items must be in original condition.",
"Shipping: Free for orders over $50. Standard delivery takes 3-5 business days.",
"Warranty: All electronics have a 1-year manufacturer warranty.",
"International shipping: Available to 50+ countries. Import duties may apply.",
"Refund process: Full refund within 5-7 business days after we receive the return."
]
# Split documents
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
chunks = splitter.create_documents(documents)
# Create vector store
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory="./chroma_db"
)
# Similarity search
results = vectorstore.similarity_search("How do I get a refund?", k=2)
for doc in results:
print(f"- {doc.page_content}")
Output:
TEXT
# Execution successful
▶ Example 4: FAISS Vector Storage
PYTHON
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS
embeddings = OllamaEmbeddings(model="nomic-embed-text")
# Create FAISS index
texts = [
"Product A: Wireless headphones, $89.99, Bluetooth 5.3",
"Product B: Laptop stand, $34.99, Adjustable height",
"Product C: USB-C cable, $12.99, 2-meter length"
]
vectorstore = FAISS.from_texts(texts, embeddings)
# Save and load
vectorstore.save_local("./faiss_index")
loaded_vs = FAISS.load_local("./faiss_index", embeddings,
allow_dangerous_deserialization=True)
# Search
results = loaded_vs.similarity_search("headphones", k=2)
for doc in results:
print(doc.page_content)
Output:
TEXT
# Execution successful
7. Complete RAG Pipeline and SupportBot V3
▶ Example 5: Complete RAG Pipeline
PYTHON
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
# Build retriever
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})
# Build RAG chain
llm = ChatOllama(model="qwen2.5", temperature=0.3)
prompt = ChatPromptTemplate.from_messages([
("system", "You are SupportBot. Answer based ONLY on the following context. "
"If the answer is not in the context, say 'I don't have that information. "
"Let me connect you with a human agent.'\n\nContext:\n{context}"),
("human", "{question}")
])
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
# Ask questions
answer = rag_chain.invoke("What is the return policy?")
print(answer)
Output:
TEXT
# Function defined successfully
8. Comprehensive Example: SupportBot V3 RAG Customer Service System
PYTHON
# ============================================
# Comprehensive: SupportBot V3 with RAG
# Product knowledge base powered customer service
# ============================================
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from pathlib import Path
class SupportBotV3:
def __init__(self, kb_path: str = "./knowledge_base",
model: str = "qwen2.5",
embed_model: str = "nomic-embed-text"):
self.llm = ChatOllama(model=model, temperature=0.3)
self.embeddings = OllamaEmbeddings(model=embed_model)
self.kb_path = kb_path
self.vectorstore = None
self.rag_chain = None
def build_knowledge_base(self, doc_dir: str):
"""Load documents and build vector store."""
docs = []
for f in Path(doc_dir).glob("*.txt"):
loader = TextLoader(str(f))
docs.extend(loader.load())
for f in Path(doc_dir).glob("*.md"):
loader = TextLoader(str(f))
docs.extend(loader.load())
splitter = RecursiveCharacterTextSplitter(
chunk_size=500, chunk_overlap=50
)
chunks = splitter.split_documents(docs)
self.vectorstore = Chroma.from_documents(
documents=chunks,
embedding=self.embeddings,
persist_directory=self.kb_path
)
print(f"Indexed {len(chunks)} chunks from {len(docs)} documents")
def setup_rag_chain(self):
"""Build the RAG chain."""
if not self.vectorstore:
self.vectorstore = Chroma(
persist_directory=self.kb_path,
embedding_function=self.embeddings
)
retriever = self.vectorstore.as_retriever(search_kwargs={"k": 3})
prompt = ChatPromptTemplate.from_messages([
("system", (
"You are SupportBot for GlobalShop e-commerce. "
"Answer based ONLY on the provided context. "
"If not in context, say: 'Let me connect you with a human agent.' "
"Be concise (2-3 sentences).\n\nContext:\n{context}"
)),
("human", "{question}")
])
def format_docs(docs):
return "\n\n".join(d.page_content for d in docs)
self.rag_chain = (
{"context": retriever | format_docs,
"question": RunnablePassthrough()}
| prompt | self.llm | StrOutputParser()
)
def ask(self, question: str) -> str:
"""Ask a question using RAG."""
if not self.rag_chain:
self.setup_rag_chain()
return self.rag_chain.invoke(question)
# Usage
if __name__ == "__main__":
bot = SupportBotV3()
# First time: build knowledge base
# bot.build_knowledge_base("./docs")
# Ask questions
questions = [
"What is the return policy for electronics?",
"How long does international shipping take?",
"Do you offer free returns?",
]
for q in questions:
answer = bot.ask(q)
print(f"Q: {q}")
print(f"A: {answer}\n")
❓ FAQ
Q Should I choose RAG or Fine-tuning?
A RAG is suitable for scenarios where knowledge is frequently updated (product FAQs, policy documents) without retraining. Fine-tuning is for changing model behavior patterns. 90% of enterprise scenarios are better served by RAG.
Q How do I choose chunk_size?
A 500-1000 characters is a good starting point. Short documents (FAQs) use 200-500; long documents (manuals) use 500-1000. Set overlap to 10-20% of chunk_size.
Q What should I do if retrieval results are inaccurate?
A Try: 1) Switch to a higher-dimensional embedding model (mxbai-embed-large); 2) Increase k value (3→5); 3) Use MMR search for more diversity; 4) Improve document chunking quality.
Q How do I choose between Chroma and FAISS?
A Use Chroma during development (auto-persistence, filtering support). Use FAISS when you need extreme speed and all data fits in memory. Use Qdrant for production (distributed, advanced filtering).
Q How do I estimate RAG token consumption?
A Each query consumes = query tokens + top-k chunks tokens + generated answer tokens. With k=3 and chunk_size=500, input is approximately 1500-2000 tokens. Local inference has no token cost.
Q How do I evaluate RAG quality?
A Use the RAGAS framework to evaluate 4 metrics: Faithfulness (is the answer faithful to the context), Answer Relevancy (is the answer on-topic), Context Precision (retrieval precision), and Context Recall.
📖 Summary
- RAG six-step pipeline: Load → Chunk → Embed → Store → Retrieve → Generate
- nomic-embed-text is the default Ollama embedding model, 768 dimensions, best value
- RecursiveCharacterTextSplitter is the preferred chunking tool; chunk_size=500 is a good starting point
- Chroma for development, FAISS for high-speed retrieval, Qdrant for production
- RAG chain composed with LCEL: retriever | format | prompt | llm | parser
- SupportBot V3 uses RAG to ground answers in evidence, eliminating "fabrication" issues
📝 Exercises
- Basic (⭐): Use OllamaEmbeddings to store 5 product FAQ texts in Chroma, and perform one similarity search.
- Intermediate (⭐⭐): Build a complete RAG pipeline — from text file loading, chunking, embedding, storage to retrieval-generation, and test with 3 questions.
- Advanced (⭐⭐⭐): Build a knowledge base for SupportBot V3 — prepare 10+ product documents, build a RAG system, and compare answer quality with and without RAG for the same questions.