Learn RAG From Scratch: Python AI Tutorial by a LangChain Engineer

Introduction to RAG with LangChain

Retrieval Augmented Generation (RAG) is a cornerstone of modern AI, blending large language models (LLMs) with private data. In this tutorial, Lance Martin, a software engineer at LangChain, breaks down RAG from scratch using Python. Why RAG? Most data is private, unlike the public datasets LLMs are trained on. With context windows expanding from 4,000 tokens (a dozen pages) to over a million (thousands of pages), RAG bridges this gap, making LLMs smarter with your data.

Watch the full course on YouTube!

What is RAG?

RAG combines retrieval and generation in three steps:

1. Indexing: Process external data into a searchable format (e.g., vector stores).
2. Retrieval: Fetch relevant documents based on a query.
3. Generation: Use an LLM to craft answers grounded in retrieved data.

It’s powerful because it unites LLM capabilities with private data, like corporate docs or personal files, not natively in their training sets.

“RAG makes LLMs the center of a new operating system, feeding them external data,” says Lance.

RAG Basics: Indexing, Retrieval, Generation

Indexing

Indexing converts documents into numerical vectors for easy retrieval. Lance explains:

• Sparse Vectors: Statistical methods (e.g., word frequency) from Google.
• Embeddings: Machine-learned, fixed-length vectors capturing semantic meaning (e.g., OpenAI embeddings, 1536 dimensions).

Documents are split (due to limited context windows, like 512-8,000 tokens), embedded, and stored in a vector store like Chroma.

Retrieval

Retrieval uses similarity search (e.g., k-nearest neighbors, KNN) in a high-dimensional space:

• Embed the query.
• Find nearby document vectors (e.g., set k=1 for one result).

In code: retriever.get_relevant_documents("What is task decomposition?") fetches relevant splits.

Generation

Generation stuffs retrieved documents into an LLM’s context window with a prompt (e.g., “Answer based on this context: {context}”). Lance uses LangChain’s LCEL to chain a prompt, LLM (like GPT-3.5), and parser:

chain = prompt | llm | StrOutputParser()

Invoke it with chain.invoke({"context": docs, "question": query}).

Advanced RAG Techniques

Query Translation

Optimize queries for better retrieval:

• Multi-Query: Rewrite a query into multiple perspectives (e.g., 5 rephrased questions) and union results.
• RAG Fusion: Like multi-query, but ranks results with reciprocal rank fusion.
• Decomposition: Break queries into sub-questions (e.g., “What are agent components?” → sub-tasks solved sequentially).
• Step-Back Prompting: Ask a broader question (e.g., “Agent memory?” → “What’s memory in AI?”) for context.
• HyDE: Generate a hypothetical document to align queries with document space.

Routing

Send queries to the right source (e.g., vector store, SQL DB):

• Logical Routing: LLM decides (e.g., Python vs. JS docs) using structured outputs.
• Semantic Routing: Embed queries and prompts, pick the closest match.

Query Construction

Convert natural language to structured queries (e.g., metadata filters like “videos after 2024”) using function calling.

Active RAG (Flow Engineering)

Use LangGraph for adaptive flows:

• Retrieve → Grade relevance → Web search if irrelevant → Generate → Check hallucinations → Regenerate if needed.

Example: CoHERE’s Command R (35B parameters) routes and grades fast, enhancing reliability.

Is RAG Dead? Long Context LLMs vs. RAG

With million-token context windows (e.g., Claude 3), can we skip RAG? Lance’s multi-needle tests with GPT-4 (128k tokens) reveal:

• More Needles, Worse Recall: 60% success with 10 facts vs. 100% with 1.
• Reasoning Hurts: Asking for first letters drops accuracy.
• Recency Bias: Early context facts are forgotten.

Context stuffing costs more ($1/100k tokens), lacks auditability, and raises security issues. RAG evolves to document-centric approaches:

• Multi-Representation Indexing: Summarize docs for retrieval, pass full docs to LLMs.
• Raptor: Cluster and summarize docs hierarchically for cross-document reasoning.

“RAG isn’t dead—it’s changing. Think flow engineering, not just chunking,” Lance argues.

Conclusion

This LangChain tutorial equips you with RAG fundamentals and advanced techniques. From indexing to adaptive flows, Lance shows Python code (shared in notebooks) to build robust AI systems. Long-context LLMs won’t kill RAG—they’ll refine it. Experiment with LangGraph, Command R, and these methods—your private data deserves it!