Today's Journey

The Limits of Parametric Memory

What are the fundamental limitations of storing knowledge in model parameters?

Problems with purely parametric models:

• Knowledge cutoff: No information after training date
• Hallucinations: Models confidently generate false information
• No source attribution: Can't cite where information comes from
• Expensive updates: Retraining for new information costs millions
• Privacy concerns: Sensitive data baked into parameters
• Domain specificity: Limited knowledge of specialized domains
• Outdated facts: World changes but model weights don't

Solution: Combine parametric knowledge with non-parametric retrieval!

Retrieval Augmented Generation: Definition

Core Idea: Instead of relying only on learned parameters, the model can "look things up"!

Reference: Lewis et al. (2020) - "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"

RAG Architecture

Worked Example: "What causes the Northern Lights?"

RAG: Step-by-Step Walkthrough

Query: "What is the capital of Kazakhstan?"

1# Step 1: Embed the query
2query = "What is the capital of Kazakhstan?"
3query_embedding = embedding_model.encode(query)
4# Result: numpy array of shape (384,)
5
6# Step 2: Search vector database
7results = vector_db.search(query_embedding, top_k=3)
8# Returns: [
9#   {"text": "Astana is the capital of Kazakhstan...", "score": 0.94},
10#   {"text": "Kazakhstan's capital moved from Almaty...", "score": 0.89},
11#   {"text": "The city was renamed Nur-Sultan in 2019...", "score": 0.85}
12# ]
13
14# Step 3: Build augmented prompt
15context = "\n".join([r["text"] for r in results])
16prompt = f"""Answer based on the context below.
17Context: {context}
18Question: {query}
19Answer:"""
20

RAG: Step-by-Step Walkthrough

21# Step 4: Generate with LLM
22response = llm.generate(prompt)
23# "Astana (previously known as Nur-Sultan) is the capital of Kazakhstan."

RAG Implementation Example

21)
22vectordb = Chroma.from_documents(
23    documents=chunks,
24    embedding=embeddings,
25    persist_directory="./chroma_db"
26)
27
28# 3. Set up retriever
29retriever = vectordb.as_retriever(
30    search_type="similarity",
31    search_kwargs={"k": 3}  # Retrieve top 3 chunks
32)

RAG Implementation (cont.)

1# 4. Create LLM
2llm = HuggingFacePipeline.from_model_id(
3    model_id="meta-llama/Llama-2-7b-chat-hf",
4    task="text-generation",
5    model_kwargs={"temperature": 0.7, "max_length": 512}
6)
7
8# 5. Create RAG chain
9qa_chain = RetrievalQA.from_chain_type(
10    llm=llm,
11    retriever=retriever,
12    return_source_documents=True,
13    chain_type="stuff"  # How to combine documents
14)
15
16# 6. Query the system
17query = "What is retrieval augmented generation?"
18result = qa_chain({"query": query})
19
20print("Answer:", result['result'])

RAG Implementation (cont.)

21print("\nSources:")
22for doc in result['source_documents']:
23    print(f"- {doc.metadata['source']}: {doc.page_content[:100]}...")

Tutorial: HuggingFace Advanced RAG - https://huggingface.co/learn/cookbook/advanced_rag

Embedding Models for Retrieval

Choosing the Right Embedding Model:

Code Example: Dense vs Hybrid Retrieval

1# Dense retrieval (semantic similarity)
2from sentence_transformers import SentenceTransformer
3model = SentenceTransformer('all-MiniLM-L6-v2')
4query_vec = model.encode("What causes headaches?")
5# Finds: "Migraines are often triggered by..." (semantically similar)
6
7# Sparse retrieval (keyword matching with BM25)
8from rank_bm25 import BM25Okapi
9bm25 = BM25Okapi(tokenized_corpus)
10scores = bm25.get_scores(query.split())
11# Finds: "Headaches can be caused by..." (exact keyword match)
12
13# Hybrid: Combine both for best results!
14final_score = 0.7 * dense_score + 0.3 * sparse_score

Prompt Engineering for RAG

21
22question = "When was the Eiffel Tower built?"
23
24response = llm.generate(RAG_PROMPT.format(context=context, question=question))
25# "The Eiffel Tower was completed in 1889 for the World's Fair [1]."

Production Challenges

Evaluation Metrics for RAG

How to measure RAG quality:

Retrieval Quality:

• Recall@k: Are relevant docs in top-k?
• MRR (Mean Reciprocal Rank): Where is first relevant doc?
• NDCG (Normalized Discounted Cumulative Gain): Ranked quality

Generation Quality:

• Factual accuracy: Are answers correct?
• Faithfulness: Does answer match retrieved docs?
• Relevance: Does answer address the question?
• Citation quality: Are sources correctly attributed?

End-to-End:

• Answer correctness: Human evaluation or LLM-as-judge
• Groundedness: Verifiable in retrieved docs
• User satisfaction: Thumbs up/down, A/B testing

Common RAG Failure Modes

1. Retrieval Failures
   • Wrong documents retrieved

• Relevant docs not in knowledge base
• Poor query formulation

2. Context Problems
   • Too much irrelevant context

• Context too long for LLM
• Important info not in retrieved chunks

3. Generation Issues
   • Ignores retrieved context

• Hallucinates despite good context
• Incorrect citations
• Overly dependent on parametric knowledge

4. System Issues
   • High latency

• Embedding drift
• Stale index

Mitigation: Monitoring, A/B testing, human-in-the-loop validation

Multimodal RAG

Beyond text: Retrieving images, tables, code, etc.

• Vision + Text

• Use CLIP embeddings for images

• Retrieve relevant diagrams, charts

• Generate answers referencing visual content

  \item Code Retrieval

  • Embed code snippets

• Retrieve relevant functions/examples

• Code completion and debugging

  \item Structured Data

  • Tables, databases

• Knowledge graphs

• SQL generation from natural language

  \item Audio/Video

  • Transcribe and embed

• Retrieve relevant segments

• Timestamp-aware responses

Future RAG systems will seamlessly integrate multiple modalities!

Graph-Based RAG

Combining knowledge graphs with RAG:

Traditional RAG: Flat document chunks

Graph RAG: Documents + relationships

Advantages:

• Capture entity relationships
• Multi-hop reasoning (A → B → C)
• Better for complex queries
• Explicit knowledge structure

Implementation:

1. Extract entities and relations from documents
2. Build knowledge graph
3. Traverse graph during retrieval
4. Combine graph paths with document chunks

Tools: Neo4j, Amazon Neptune, LangChain Graph QA

Future of RAG

Emerging trends and research directions:

1. Agentic RAG
   • LLM decides retrieval strategy

• Multi-step reasoning with retrieval
• Tool use (web search, APIs, databases)

2. Long-context RAG
   • Models with 1M+ token windows

• Entire books as context
• Retrieval still useful for efficiency

3. Personalized RAG
   • User-specific knowledge bases

• Privacy-preserving retrieval
• Federated learning

4. Real-time RAG
   • Live web scraping

• Streaming document updates
• Event-driven retrieval

Key Takeaways

1. RAG solves fundamental LLM limitations
   • Knowledge cutoff, hallucination, no citations
2. Core pipeline: Retrieve → Augment → Generate
   • Vector search for relevant documents

• Incorporate into prompt

3. Many variants exist
   • Naive RAG → Self-RAG → Corrective RAG → Agentic RAG
4. Key components matter
   • Chunking strategy, embedding model, vector DB
5. Production requires careful engineering
   • Latency, cost, quality evaluation
6. RAG + Fine-tuning is powerful combo
   • Fine-tune for style, RAG for knowledge

Readings

Required:

1. Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
   [arXiv]
2. Asai et al. (2023): Self-RAG: Learning to Retrieve, Generate, and Critique
   [arXiv]

Recommended:

• Yan et al. (2024): Corrective RAG [arXiv]
• Gao et al. (2022): Precise Zero-Shot Dense Retrieval (HyDE) [arXiv]
• HuggingFace RAG Tutorial [Tutorial]
• LangChain RAG Docs [Docs]

Questions?

Next: Mixture of Experts!