RAG (Retrieval-Augmented Generation)
1. What You'll Learn
❶ What Problems Does RAG Solve—Knowledge Cutoff and Hallucinations in LLMs
❷ An Intuitive Understanding of Vector Embeddings
❸ Similarity Search—Cosine Similarity
❹ RAG Process: Retrieval → Augmentation → Generation
❺ Implementing a Simple RAG System in Python
2. The Story
Alice is in charge of internal knowledge management at a medium-sized company. The company has accumulated more than 1,000 PDF documents—ranging from expense reimbursement policies to safety procedures. Whenever a new employee asks a question, Alice has to manually search through the documents, which is very inefficient.
She tried using ChatGPT directly to answer internal questions, but ChatGPT knew nothing about company documents and often “made up” policies that didn’t exist—for example, claiming that “the travel allowance is $80 per day,” when the actual policy was $45.
Charlie proposed a solution: "First, find paragraphs related to the question in the knowledge base, then have the LLM answer based on those paragraphs—that way, it won’t make things up." This is RAG (Retrieval-Augmented Generation).
Alice followed these steps: she divided the document into sections, generated vectors, and stored them in a vector database; when a user asked a question, she first retrieved the most relevant paragraphs and then incorporated them into the prompt for the LLM to answer. From then on, the answers were “evidence-based,” hallucinations were significantly reduced, and new employees were able to quickly obtain accurate answers.
3. Knowledge Cutoff and Hallucination Issues in LLMs
(1) Knowledge Cutoff
An LLM’s knowledge comes from its training data, and once training is complete, it is “frozen.” GPT-4’s training data is current as of 2023, so it knows nothing about events that have occurred since then. More importantly, your private data (company documents, personal notes) has never been included in the training set, so the LLM is inherently unable to answer questions about it.
(2) Hallucinations
When an LLM encounters a question it doesn’t know the answer to, it doesn’t say, “I don’t know,” but instead confidently fabricates an answer that seems plausible—this is hallucination. Hallucinations are particularly dangerous in factual question-answering tasks.
(3) The Core Concept of RAG
Rather than having an LLM "memorize" all knowledge, it’s better to dynamically provide relevant information when asking a question, allowing the LLM to respond based on the given context. This is RAG:
- Retrieval: Retrieving relevant documents from the knowledge base
- Augmented: Incorporate the search results into the prompt as context
- Generation: The LLM generates a response based on the enhanced prompt
| Feature | Pure LLM | RAG-enhanced |
|---|---|---|
| Knowledge Source | Training Data Only | Training Data + External Documentation |
| Private Data | Inaccessible | Retrievable |
| Risk of Hallucinations | High (Prone to Fabrication) | Low (Contextually Constrained) |
| Knowledge Update | Requires Retraining | Just Update the Documentation |
| Cost | Inference Cost | Inference + Retrieval Cost |
| Traceability | None | Citable Source Paragraph |
- Vector Embedding
(1) How is text converted into a vector?
Embedding is the process of mapping text to high-dimensional vectors. For example, after passing through an embedding model, a sentence is converted into a 1,536-dimensional array of floating-point numbers. Text with similar meanings has similar vectors, while text with different meanings has vectors that are far apart.
You can think of embeddings as a "semantic coordinate system": each text has a position in this system, and texts that are closer together are semantically more similar.
(2) Common Embedding Models
| Model | Dimensions | Provider | Features |
|---|---|---|---|
| text-embedding-3-small | 1536 | OpenAI | Great value for the price, fast |
| text-embedding-3-large | 3072 | OpenAI | Higher accuracy, higher cost |
| text-embedding-ada-002 | 1536 | OpenAI | Previous generation, still widely used |
| bge-large-en-v1.5 | 1024 | BAAI | Open source, excellent performance in English |
| bge-large-zh-v1.5 | 1024 | BAAI | Open source, excellent performance in Chinese |
| m3e-base | 768 | Moka AI | Open source, supports both Chinese and English |
▶ Example: Generating Text Embeddings Using the OpenAI API (Difficulty: ⭐)
PYTHON
from openai import OpenAI
client = OpenAI()
def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
response = client.embeddings.create(input=text, model=model)
return response.data[0].embedding
text = "The company travel allowance is $45 per day."
vector = get_embedding(text)
print(f"Dimension: {len(vector)}")
print(f"First 5 values: {vector[:5]}")
TEXT
Dimension: 1536
First 5 values: [0.0123, -0.0045, 0.0378, -0.0211, 0.0056]
5. Similarity Search and Cosine Similarity
(1) Cosine Similarity
Cosine similarity measures the degree of similarity between the directions of two vectors, with values ranging from [-1, 1]:
$$\text{cosine_similarity}(A, B) = \frac{A \cdot B}{|A| \times |B|}$$
- 1: Exactly the same direction (most similar in meaning)
- 0: Orthogonal (uncorrelated)
- -1: Opposite direction (opposite meaning)
(2) Vector Databases
A vector database is a system specifically designed to store and retrieve vectors, supporting efficient approximate nearest neighbor (ANN) searches.
| Database | Type | Features |
|---|---|---|
| Chroma | Open source, embedded | Native Python, suitable for prototyping |
| FAISS | Open source, library | Powered by Meta, extremely fast, fully local |
| Pinecone | Cloud Services | Fully Managed, Scalable, Pay-as-You-Go |
| Milvus | Open-source, distributed | Suitable for large-scale production environments |
| Qdrant | Open Source | Implemented in Rust, with excellent performance |
▶ Example: Calculate the cosine similarity between two sentences (Difficulty: ⭐)
PYTHON
import numpy as np
def cosine_similarity(a: list[float], b: list[float]) -> float:
a_arr = np.array(a)
b_arr = np.array(b)
dot = np.dot(a_arr, b_arr)
norm_a = np.linalg.norm(a_arr)
norm_b = np.linalg.norm(b_arr)
return float(dot / (norm_a * norm_b))
sentences = {
"travel_allowance": get_embedding("The company travel allowance is $45 per day."),
"reimbursement": get_embedding("How do I claim travel reimbursement?"),
"unrelated": get_embedding("The weather is sunny today."),
}
sim1 = cosine_similarity(sentences["travel_allowance"], sentences["reimbursement"])
sim2 = cosine_similarity(sentences["travel_allowance"], sentences["unrelated"])
print(f"Related sentences: {sim1:.4f}")
print(f"Unrelated sentences: {sim2:.4f}")
TEXT
Related sentences: 0.8234
Unrelated sentences: 0.3012
▶ Example: Simple Vector Retrieval—Finding the Most Similar Document (Difficulty: ⭐⭐)
PYTHON
def top_k_search(
query_embedding: list[float],
doc_embeddings: list[list[float]],
doc_texts: list[str],
k: int = 3,
) -> list[tuple[str, float]]:
similarities = [
(doc_texts[i], cosine_similarity(query_embedding, doc_embeddings[i]))
for i in range(len(doc_texts))
]
similarities.sort(key=lambda x: x[1], reverse=True)
return similarities[:k]
documents = [
"Travel allowance is $45 per day for domestic trips.",
"International travel requires VP approval in advance.",
"Employees must submit receipts within 30 days.",
"Remote work policy allows up to 2 days per week at home.",
"The office kitchen is stocked with free coffee and snacks.",
]
doc_embeddings = [get_embedding(doc) for doc in documents]
query = "How much can I spend on travel per day?"
query_embedding = get_embedding(query)
results = top_k_search(query_embedding, doc_embeddings, documents, k=3)
for text, score in results:
print(f"[{score:.4f}] {text}")
TEXT
[0.8912] Travel allowance is $45 per day for domestic trips.
[0.7634] Employees must submit receipts within 30 days.
[0.7102] International travel requires VP approval in advance.
6. RAG Architecture: Retrieval → Augmentation → Generation
(1) The Complete RAG Process
RAG combines "retrieval" and "generation" to form a complete workflow:
- User Question: The user enters a question in natural language
- Query Embedding: Convert the query into a vector
- Vector Retrieval: Searching for the most similar document fragments (Top-K) in a vector database
- Context Enhancement: Incorporate the retrieved documents into the prompt
- LLM Generation: The LLM answers questions based on the enhanced prompt
(2) Mermaid Flowchart
graph TB
A["User Question"] --> B["Query Embedding"]
B --> C["Vector Search"]
D["Document Chunks<br/>+ Embeddings"] --> C
C --> E["Top-K Results"]
E --> F["Augmented Prompt<br/>Context + Question"]
F --> G["LLM Generation"]
G --> H["Answer"]
(3) Key Design Decisions
- Top-K Selection: If K is too small, relevant information may be omitted; if K is too large, noise may be introduced. Typically, K = 3–5.
- Prompt Template: Explicitly instruct the LLM to "answer based solely on the provided context" to reduce hallucinations.
- Reranking: An optional step in which a cross-encoder is used to reorder the search results to improve accuracy.
▶ Example: Complete RAG Call Flow (Difficulty: ⭐⭐)
PYTHON
from openai import OpenAI
client = OpenAI()
def rag_query(question: str, documents: list[str], k: int = 3) -> str:
query_emb = get_embedding(question)
doc_embs = [get_embedding(doc) for doc in documents]
top_docs = top_k_search(query_emb, doc_embs, documents, k=k)
context = "\n\n".join([f"[Doc {i+1}] {text}" for i, (text, _) in enumerate(top_docs)])
prompt = f"""Answer the question based ONLY on the following context.
If the context does not contain the answer, say "I don't know."
Context:
{context}
Question: {question}
Answer:"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0,
)
return response.choices[0].message.content
docs = [
"Travel allowance is $45 per day for domestic trips.",
"International travel requires VP approval at least 2 weeks in advance.",
"Employees must submit expense reports within 30 days of the trip.",
"The maximum hotel reimbursement is $200 per night.",
"Flight bookings must use the company travel portal.",
]
answer = rag_query("What is the daily travel allowance?", docs)
print(answer)
TEXT
The daily travel allowance is $45 for domestic trips.
▶ Example: Comparing Answer Quality with and without RAG (Difficulty: ⭐⭐)
PYTHON
def ask_without_rag(question: str) -> str:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": question}],
temperature=0,
)
return response.choices[0].message.content
question = "What is Acme Corp's daily travel allowance?"
print("=== Without RAG ===")
print(ask_without_rag(question))
print("\n=== With RAG ===")
print(rag_query(question, docs))
TEXT
=== Without RAG ===
I don't have specific information about Acme Corp's daily travel allowance.
Company travel allowances vary, but a common range is $50-$100 per day.
Please check your company's travel policy for the exact amount.
=== With RAG ===
The daily travel allowance is $45 for domestic trips.
Without RAG, LLMs either admit they don’t know or make up a generic answer; with RAG, their answers are precise and well-founded.
7. Chunking Strategy
(1) Why is chunking necessary?
Documents are often too long to be embedded directly. Long documents need to be divided into smaller chunks, and embeddings generated for each chunk separately. The chunking strategy directly affects retrieval quality.
(2) Common Chunking Strategies
| Strategy | Principle | Advantages | Disadvantages |
|---|---|---|---|
| Fixed-size chunks | Split into segments every N tokens | Simple to implement | May break up the meaning |
| Sentence-level segmentation | Split by periods/line breaks | Semantically complete | Segments may be too small or too large |
| Paragraph-level segmentation | Segmented by paragraphs | Semantic coherence | Uneven length |
| Semantic Chunking | Detects semantic boundaries based on embedding similarity | Most complete semantics | High computational cost |
| Recursive character chunking | Recursive splitting by delimiter hierarchy | Balancing semantics and length | Requires parameter tuning |
(3) Recommendations for Chunking Parameters
- Chunk Size: Typically 256–1024 tokens, depending on the model and use case
- Chunk Overlap: Adjacent chunks overlap by 10%–20% to avoid semantic discontinuity
- Rule of thumb: Use smaller blocks (256–512) for question-answering scenarios and larger blocks (512–1024) for summarization scenarios.
(4) Recursive chunking using LangChain
PYTHON
from langchain.text_splitter import RecursiveCharacterTextSplitter
text = """Acme Corp Travel Policy
1. Domestic Travel
The daily travel allowance is $45. This covers meals and incidental expenses.
Hotel reimbursement is capped at $200 per night.
2. International Travel
All international trips require VP approval at least 2 weeks in advance.
The daily allowance for international travel is $75.
Hotel reimbursement is capped at $300 per night.
3. Expense Reporting
All expense reports must be submitted within 30 days of trip completion.
Receipts are required for all expenses over $25."""
splitter = RecursiveCharacterTextSplitter(
chunk_size=200,
chunk_overlap=50,
separators=["\n\n", "\n", ". ", " "],
)
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
print(f"--- Chunk {i+1} ---")
print(chunk)
print()
TEXT
--- Chunk 1 ---
Acme Corp Travel Policy
1. Domestic Travel
The daily travel allowance is $45. This covers meals and incidental expenses.
Hotel reimbursement is capped at $200 per night.
--- Chunk 2 ---
2. International Travel
All international trips require VP approval at least 2 weeks in advance.
The daily allowance for international travel is $75.
--- Chunk 3 ---
international travel is $75.
Hotel reimbursement is capped at $300 per night.
3. Expense Reporting
All expense reports must be submitted within 30 days of trip completion.
8. Comprehensive Example: Mini Knowledge Base Q&A System
Building a complete RAG pipeline: 3 documents → chunking → embedding → storage → user query → retrieval → prompt concatenation → LLM response.
▶ Example: Mini Knowledge Base Q&A System (Difficulty: ⭐⭐⭐)
PYTHON
import numpy as np
from openai import OpenAI
client = OpenAI()
# --- Step 1: Prepare documents ---
documents = [
{
"title": "Travel Policy",
"content": (
"Acme Corp Travel Policy\n\n"
"1. Domestic Travel\n"
"The daily travel allowance is $45 for domestic trips. "
"This covers meals and incidental expenses. "
"Hotel reimbursement is capped at $200 per night.\n\n"
"2. International Travel\n"
"All international trips require VP approval at least 2 weeks in advance. "
"The daily allowance for international travel is $75. "
"Hotel reimbursement is capped at $300 per night."
),
},
{
"title": "Leave Policy",
"content": (
"Acme Corp Leave Policy\n\n"
"1. Annual Leave\n"
"Full-time employees receive 15 days of paid annual leave per year. "
"Unused leave can be carried over to the next year, up to a maximum of 5 days.\n\n"
"2. Sick Leave\n"
"Employees receive 10 days of paid sick leave per year. "
"A doctor's note is required for absences exceeding 3 consecutive days."
),
},
{
"title": "IT Security Policy",
"content": (
"Acme Corp IT Security Policy\n\n"
"1. Password Requirements\n"
"All passwords must be at least 12 characters long and include uppercase, "
"lowercase, numbers, and special characters. "
"Passwords must be changed every 90 days.\n\n"
"2. Device Policy\n"
"Personal devices may not connect to the corporate network. "
"All company devices must have approved antivirus software installed."
),
},
]
# --- Step 2: Chunk documents ---
def chunk_text(text: str, chunk_size: int = 200, overlap: int = 50) -> list[str]:
words = text.split()
chunks = []
start = 0
while start < len(words):
end = start + chunk_size
chunks.append(" ".join(words[start:end]))
start += chunk_size - overlap
return chunks
all_chunks: list[str] = []
chunk_sources: list[str] = []
for doc in documents:
chunks = chunk_text(doc["content"])
for chunk in chunks:
all_chunks.append(chunk)
chunk_sources.append(doc["title"])
print(f"Total chunks: {len(all_chunks)}")
# --- Step 3: Generate embeddings ---
def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
response = client.embeddings.create(input=text, model=model)
return response.data[0].embedding
chunk_embeddings = [get_embedding(chunk) for chunk in all_chunks]
print(f"Embedding dimension: {len(chunk_embeddings[0])}")
# --- Step 4: Vector search ---
def cosine_similarity(a: list[float], b: list[float]) -> float:
a_arr = np.array(a)
b_arr = np.array(b)
return float(np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr)))
def search(query: str, k: int = 3) -> list[tuple[str, str, float]]:
query_emb = get_embedding(query)
scores = [
(all_chunks[i], chunk_sources[i], cosine_similarity(query_emb, chunk_embeddings[i]))
for i in range(len(all_chunks))
]
scores.sort(key=lambda x: x[2], reverse=True)
return scores[:k]
# --- Step 5: RAG query ---
def rag_answer(question: str, k: int = 3) -> str:
results = search(question, k=k)
context_parts = []
for i, (chunk, source, score) in enumerate(results):
context_parts.append(f"[Source: {source}, Relevance: {score:.2f}]\n{chunk}")
context = "\n\n---\n\n".join(context_parts)
prompt = f"""Answer the question based ONLY on the following context.
If the context does not contain enough information, say "I don't have enough information."
Always cite which source document your answer comes from.
Context:
{context}
Question: {question}
Answer:"""
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0,
)
return response.choices[0].message.content
# --- Step 6: Test ---
questions = [
"What is the daily travel allowance for domestic trips?",
"How many days of annual leave do employees get?",
"What are the password requirements?",
"What is the company's remote work policy?",
]
for q in questions:
print(f"Q: {q}")
print(f"A: {rag_answer(q)}")
print()
TEXT
Total chunks: 9
Embedding dimension: 1536
Q: What is the daily travel allowance for domestic trips?
A: The daily travel allowance for domestic trips is $45, according to the Travel Policy.
Q: How many days of annual leave do employees get?
A: Full-time employees receive 15 days of paid annual leave per year, according to the Leave Policy.
Q: What are the password requirements?
A: Passwords must be at least 12 characters long and include uppercase, lowercase, numbers, and special characters. They must be changed every 90 days, according to the IT Security Policy.
Q: What is the company's remote work policy?
A: I don't have enough information. The provided context does not contain any remote work policy.
Note the last question: RAG correctly responds with "insufficient information" rather than fabricating an answer—and that is precisely the value of RAG.
❓ FAQ
Q Which is better, RAG or fine-tuning?
A They address different problems. RAG is suitable for knowledge-intensive tasks (enabling LLMs to access external knowledge) without the need for retraining; fine-tuning is suitable for style/format adaptation (teaching LLMs specific output formats). In many scenarios, the two approaches complement each other—first using RAG to provide knowledge, then using fine-tuning to optimize the style. RAG is less expensive and can be updated more quickly, so it is usually the preferred choice.
Q What is a vector database?
A A vector database is a system specifically designed to store high-dimensional vectors and support fast similarity searches. Traditional databases perform queries based on exact matches, while vector databases perform queries based on "semantic similarity." Common options include: Chroma (embedded, suitable for development), FAISS (standalone library, fast), and Pinecone (fully managed cloud service).
Q Is a higher embedding dimension always better?
A Not necessarily. Higher dimensions theoretically allow for encoding more semantic information, but they also entail greater storage and computational overhead. Actual performance depends on the model quality and training data. For example, text-embedding-3-small (1,536 dimensions) is sufficient for many tasks, while text-embedding-3-large (3,072 dimensions) only provides significant improvements in specific scenarios. When making a choice, balance accuracy and cost.
Q Why is chunk size important?
A If the chunk is too large, retrieval may lack precision, potentially introducing a large amount of irrelevant content, wasting tokens, and interfering with the LLM. If the chunk is too small, there may be insufficient context, and the retrieved fragments may not fully answer the question. For general Q&A scenarios, a chunk size of 256–512 tokens is recommended, with 10%–20% overlap to avoid semantic discontinuity.
Q Can RAG completely eliminate hallucinations?
A It cannot completely eliminate them, but it can significantly reduce them. RAG provides contextual constraints, so LLMs are more likely to base their answers on the given information. However, LLMs may still: ignore the context, over-inference, or make things up when the context itself is incomplete. You can further reduce this by: explicitly instructing the model to “answer based solely on the context,” setting
temperature=0, incorporating reranking, and requiring the answer to cite sources.📖 Summary
- LLMs suffer from knowledge cutoff and hallucination issues, and cannot directly answer questions related to private data.
- RAG uses a "retrieval → enhancement → generation" process to enable LLMs to answer questions based on external documents, significantly reducing hallucinations.
- Embedding converts text into vectors; text vectors with similar meanings are also similar
- Cosine similarity measures the consistency of vector directions and is a core metric in semantic search.
- Chunking strategies affect retrieval quality, and recursive character chunking is the most commonly used balancing approach
- Vector databases (Chroma, FAISS, Pinecone) provide efficient vector storage and retrieval
- Complete RAG workflow: Document chunking → Embedding → Storage → Querying → Retrieval → Prompt concatenation → LLM generation
📝 Exercises
Basics (⭐)
Use the OpenAI Embeddings API to generate vectors for the following five text segments, and calculate the cosine similarity between each pair:
PYTHON
texts = [
"Cats are popular pets known for their independence.",
"Dogs are loyal companions that love to play fetch.",
"The stock market rose 3% today on strong earnings.",
"Felines enjoy climbing and sleeping in sunny spots.",
"Interest rates were raised by the central bank.",
]
Output the results as a 5×5 similarity matrix to see which texts are most similar.
Advanced (⭐⭐)
Implement a generic Top-K retrieval function that supports the following features:
PYTHON
def vector_search(
query: str,
corpus: list[str],
k: int = 5,
embedding_model: str = "text-embedding-3-small",
) -> list[dict]:
"""
Returns top-k most similar documents with scores.
Each result is a dict: {"text": ..., "score": ..., "rank": ...}
"""
# Your implementation here
pass
Requirements: Use a test document consisting of at least 10 paragraphs to verify the validity of the search results.
Challenge (⭐⭐⭐)
Build a mini RAG system containing at least 5 documents and fully implement the following workflow:
- Document Loading and Recursive Chunking (chunk_size=300, overlap=60)
- Generate and store embeddings
- User query → Retrieve Top-3 → Concatenate and enhance the prompt → LLM response
- Compare the differences in answers to the same question with and without RAG
- Test a question that does not exist in the knowledge base to verify whether RAG correctly refuses to answer it
Extension Challenge: Add a reranking step—first perform a Top-10 retrieval, then have the LLM score and rank the 10 results, and feed the Top-3 into the final generation.
