Ollama: RAG Pipeline

RAG gives AI a knowledge vault — instead of guessing answers, it retrieves truth from documents.

💡 Tip: The core principle of RAG is "retrieve first, then generate" — the model answers only based on retrieved document fragments, not from memory. This greatly reduces "hallucinations" (fabricating information), but retrieval quality directly determines answer quality. Chunking strategy and Embedding model selection are key.

📋 Prerequisites: You should first master the following

1. What You Will Learn


2. A Real Story from a SaaS Entrepreneur

⚠️ Warning: RAG cannot eliminate hallucinations — it can only significantly reduce their probability. If retrieved document fragments are irrelevant or insufficient, the model may still "fabricate" answers. Key strategy: constrain the model to answer only within the scope of retrieval results, and add to the System Prompt: "If the relevant information is not in the documents, respond with 'I cannot confirm that'."

(1) The Pain Point: The Model Doesn't Know Product Details

Alice's SupportBot can handle general customer service, but doesn't know specific product return policies, warranty terms, or pricing details. The model frequently "fabricates" information, leading to constant customer complaints.

(2) The Solution: RAG Enables the Model to Answer Based on Real Documents

Import product manuals, FAQs, and policy documents into a vector database, retrieve relevant fragments, and let the model generate answers based on facts:

PYTHON
# RAG: Retrieve relevant docs, then generate answer
docs = retriever.invoke("What is the warranty for headphones?")
answer = llm.invoke(f"Based on: {docs}\nAnswer: What is the warranty?")

3. RAG Architecture in Detail

💡 Tip: The core principle of RAG is "retrieve first, then generate" — the model answers only based on retrieved document fragments, not from memory. This greatly reduces "hallucinations" (fabricating information), but retrieval quality directly determines answer quality. Chunking strategy and Embedding model selection are key.

(1) End-to-End Pipeline Flow

100%
flowchart LR
    A[Documents] --> B[Loader]
    B --> C[Splitter]
    C --> D[Embedding Model<br/>nomic-embed-text]
    D --> E[Vector Store<br/>Chroma/FAISS]
    F[User Query] --> G[Query Embedding]
    G --> E
    E --> H[Top-K Chunks]
    H --> I[LLM Generation<br/>qwen2.5]
    I --> J[Answer]

(2) Stage-by-Stage Breakdown

Stage Tool Input Output
Load PyPDFLoader / TextLoader PDF/MD/TXT Document list
Chunk RecursiveCharacterTextSplitter Document Chunk list
Embed OllamaEmbeddings Chunk text Vector (768d)
Store Chroma / FAISS Vector + text Index
Retrieve Similarity Search Query vector Top-K Chunks
Generate ChatOllama Query + Chunks Answer

4. Ollama Embeddings

⚠️ Note: Embedding model selection directly affects retrieval quality — nomic-embed-text (768 dimensions) performs well in English but is average in Chinese, mxbai-embed-large (1024 dimensions) has higher precision but is double the size. For Chinese scenarios, test multiple models and pick the best — don't assume "bigger is always better."

(1) Embedding Model Comparison

Model Dimensions Size Speed Use Case
nomic-embed-text 768 274 MB Fast General English
mxbai-embed-large 1024 670 MB Medium High precision
all-minilm 384 46 MB Very fast Lightweight

(2) Embedding Model Selection Guide

Scenario Recommended Model Reason
General RAG nomic-embed-text Best value
High-precision retrieval mxbai-embed-large Higher dimensions, better discrimination
Resource-constrained all-minilm Smallest and fastest
Chinese-dominant nomic-embed-text Acceptable Chinese performance

▶ Example 1: Ollama Embeddings Basics

PYTHON
from langchain_ollama import OllamaEmbeddings

# Initialize embeddings model
embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Embed a single text
vector = embeddings.embed_query("What is the return policy?")
print(f"Dimension: {len(vector)}")  # 768

# Embed multiple texts
vectors = embeddings.embed_documents([
    "Free shipping on orders over $50",
    "30-day return policy for all products",
    "Warranty covers manufacturing defects for 1 year"
])
print(f"Embedded {len(vectors)} documents")

Output:

TEXT
# Execution successful

5. Document Loading and Chunking

(1) Document Loaders

Loader Format Installation
PyPDFLoader PDF pip install pypdf
TextLoader TXT Built-in
UnstructuredMarkdownLoader Markdown pip install unstructured
CSVLoader CSV Built-in
UnstructuredHTMLLoader HTML/URL pip install unstructured

(2) Chunking Strategy Comparison

Strategy Parameters Suitable Documents Pros/Cons
Fixed size chunk_size=500, overlap=50 General Simple but may break semantics
Recursive character chunk_size=500, overlap=50, separators Structured documents Preserves paragraph boundaries
Semantic chunking similarity_threshold Long texts Best quality but computationally expensive
💡 Tip: RecursiveCharacterTextSplitter is the preferred choice. chunk_size=500-1000 and overlap=50-100 are empirically optimal values.

▶ Example 2: Document Loading and Chunking

PYTHON
from langchain_community.document_loaders import TextLoader, PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Load text document
loader = TextLoader("product_faq.txt")
docs = loader.load()
print(f"Loaded {len(docs)} document(s)")

# Split into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")
print(f"First chunk preview: {chunks[0].page_content[:100]}...")

Output:

TEXT
# Execution successful

6. Vector Database Selection

(1) Three Major Vector Stores Compared

Dimension Chroma FAISS Qdrant
Type Embedded In-memory Client/Server
Persistence ✅ Local files ❌ Memory only ✅ Disk/Cloud
Installation pip install pip install + faiss-cpu Docker / pip
Best for Dev/small scale High-speed retrieval Production
Filtering ✅ Metadata filtering ✅ Advanced filtering
Distributed

▶ Example 3: Chroma Vector Storage and Retrieval

PYTHON
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Prepare documents
documents = [
    "Return policy: 30 days from purchase, items must be in original condition.",
    "Shipping: Free for orders over $50. Standard delivery takes 3-5 business days.",
    "Warranty: All electronics have a 1-year manufacturer warranty.",
    "International shipping: Available to 50+ countries. Import duties may apply.",
    "Refund process: Full refund within 5-7 business days after we receive the return."
]

# Split documents
splitter = RecursiveCharacterTextSplitter(chunk_size=200, chunk_overlap=20)
chunks = splitter.create_documents(documents)

# Create vector store
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# Similarity search
results = vectorstore.similarity_search("How do I get a refund?", k=2)
for doc in results:
    print(f"- {doc.page_content}")

Output:

TEXT
# Execution successful

▶ Example 4: FAISS Vector Storage

PYTHON
from langchain_ollama import OllamaEmbeddings
from langchain_community.vectorstores import FAISS

embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Create FAISS index
texts = [
    "Product A: Wireless headphones, $89.99, Bluetooth 5.3",
    "Product B: Laptop stand, $34.99, Adjustable height",
    "Product C: USB-C cable, $12.99, 2-meter length"
]
vectorstore = FAISS.from_texts(texts, embeddings)

# Save and load
vectorstore.save_local("./faiss_index")
loaded_vs = FAISS.load_local("./faiss_index", embeddings,
                              allow_dangerous_deserialization=True)

# Search
results = loaded_vs.similarity_search("headphones", k=2)
for doc in results:
    print(doc.page_content)

Output:

TEXT
# Execution successful

7. Complete RAG Pipeline and SupportBot V3

▶ Example 5: Complete RAG Pipeline

PYTHON
from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

# Build retriever
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# Build RAG chain
llm = ChatOllama(model="qwen2.5", temperature=0.3)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are SupportBot. Answer based ONLY on the following context. "
               "If the answer is not in the context, say 'I don't have that information. "
               "Let me connect you with a human agent.'\n\nContext:\n{context}"),
    ("human", "{question}")
])

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

# Ask questions
answer = rag_chain.invoke("What is the return policy?")
print(answer)

Output:

TEXT
# Function defined successfully

8. Comprehensive Example: SupportBot V3 RAG Customer Service System

PYTHON
# ============================================
# Comprehensive: SupportBot V3 with RAG
# Product knowledge base powered customer service
# ============================================

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import Chroma
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from pathlib import Path

class SupportBotV3:
    def __init__(self, kb_path: str = "./knowledge_base",
                 model: str = "qwen2.5",
                 embed_model: str = "nomic-embed-text"):
        self.llm = ChatOllama(model=model, temperature=0.3)
        self.embeddings = OllamaEmbeddings(model=embed_model)
        self.kb_path = kb_path
        self.vectorstore = None
        self.rag_chain = None

    def build_knowledge_base(self, doc_dir: str):
        """Load documents and build vector store."""
        docs = []
        for f in Path(doc_dir).glob("*.txt"):
            loader = TextLoader(str(f))
            docs.extend(loader.load())
        for f in Path(doc_dir).glob("*.md"):
            loader = TextLoader(str(f))
            docs.extend(loader.load())

        splitter = RecursiveCharacterTextSplitter(
            chunk_size=500, chunk_overlap=50
        )
        chunks = splitter.split_documents(docs)

        self.vectorstore = Chroma.from_documents(
            documents=chunks,
            embedding=self.embeddings,
            persist_directory=self.kb_path
        )
        print(f"Indexed {len(chunks)} chunks from {len(docs)} documents")

    def setup_rag_chain(self):
        """Build the RAG chain."""
        if not self.vectorstore:
            self.vectorstore = Chroma(
                persist_directory=self.kb_path,
                embedding_function=self.embeddings
            )

        retriever = self.vectorstore.as_retriever(search_kwargs={"k": 3})

        prompt = ChatPromptTemplate.from_messages([
            ("system", (
                "You are SupportBot for GlobalShop e-commerce. "
                "Answer based ONLY on the provided context. "
                "If not in context, say: 'Let me connect you with a human agent.' "
                "Be concise (2-3 sentences).\n\nContext:\n{context}"
            )),
            ("human", "{question}")
        ])

        def format_docs(docs):
            return "\n\n".join(d.page_content for d in docs)

        self.rag_chain = (
            {"context": retriever | format_docs,
             "question": RunnablePassthrough()}
            | prompt | self.llm | StrOutputParser()
        )

    def ask(self, question: str) -> str:
        """Ask a question using RAG."""
        if not self.rag_chain:
            self.setup_rag_chain()
        return self.rag_chain.invoke(question)

# Usage
if __name__ == "__main__":
    bot = SupportBotV3()

    # First time: build knowledge base
    # bot.build_knowledge_base("./docs")

    # Ask questions
    questions = [
        "What is the return policy for electronics?",
        "How long does international shipping take?",
        "Do you offer free returns?",
    ]
    for q in questions:
        answer = bot.ask(q)
        print(f"Q: {q}")
        print(f"A: {answer}\n")

❓ FAQ

Q Should I choose RAG or Fine-tuning?
A RAG is suitable for scenarios where knowledge is frequently updated (product FAQs, policy documents) without retraining. Fine-tuning is for changing model behavior patterns. 90% of enterprise scenarios are better served by RAG.
Q How do I choose chunk_size?
A 500-1000 characters is a good starting point. Short documents (FAQs) use 200-500; long documents (manuals) use 500-1000. Set overlap to 10-20% of chunk_size.
Q What should I do if retrieval results are inaccurate?
A Try: 1) Switch to a higher-dimensional embedding model (mxbai-embed-large); 2) Increase k value (3→5); 3) Use MMR search for more diversity; 4) Improve document chunking quality.
Q How do I choose between Chroma and FAISS?
A Use Chroma during development (auto-persistence, filtering support). Use FAISS when you need extreme speed and all data fits in memory. Use Qdrant for production (distributed, advanced filtering).
Q How do I estimate RAG token consumption?
A Each query consumes = query tokens + top-k chunks tokens + generated answer tokens. With k=3 and chunk_size=500, input is approximately 1500-2000 tokens. Local inference has no token cost.
Q How do I evaluate RAG quality?
A Use the RAGAS framework to evaluate 4 metrics: Faithfulness (is the answer faithful to the context), Answer Relevancy (is the answer on-topic), Context Precision (retrieval precision), and Context Recall.

📖 Summary


📝 Exercises

  1. Basic (⭐): Use OllamaEmbeddings to store 5 product FAQ texts in Chroma, and perform one similarity search.
  2. Intermediate (⭐⭐): Build a complete RAG pipeline — from text file loading, chunking, embedding, storage to retrieval-generation, and test with 3 questions.
  3. Advanced (⭐⭐⭐): Build a knowledge base for SupportBot V3 — prepare 10+ product documents, build a RAG system, and compare answer quality with and without RAG for the same questions.
Web-Tutorial.com

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

100%

🙏 帮我们做得更好

我们是刚上线的编程教程站,几个人的小团队,精力有限。页面虽经检查,难免还有疏漏——链接失效、排版错乱、内容有误、语言生硬……

如果您发现了,麻烦告诉我们,我们会在收到反馈后第一时间进行修复,再次感谢您的光临 🙏