LangChain RAG: How to Build a RAG Pipeline (with Code)

LangChain RAG means building a retrieval-augmented generation pipeline using LangChain, an open-source framework that gives you ready-made building blocks document loaders, text splitters, embeddings, vector stores, retrievers, and chains so you wire RAG together instead of coding every step from scratch. Think of LangChain as the set of Lego bricks; RAG is the thing you build with them.

If you already understand the idea of RAG and now want to build it with LangChain specifically, this page walks through each component, shows a minimal working build, and covers the failure modes the docs don't warn you about.

For the conceptual foundation, see the guide to large language models that power the generation step.

RAG vs LangChain: Are They the Same Thing?

No. This is the confusion the search results are full of, so let's clear it up in one paragraph.

RAG is a technique: retrieve relevant data from an external source, then generate a grounded answer using that data. LangChain is a framework: a Python (and JavaScript) library that gives you pre-built components to wire RAG and other LLM applications together faster. You can build RAG without LangChain using raw API calls. You can use LangChain for things other than RAG. They're not alternatives; LangChain is a tool you build RAG with.

Most people searching "RAG vs LangChain" are really asking: "do I need LangChain to build RAG?" The answer is no, but it saves significant boilerplate. Let's look at the components it provides.

The Core LangChain Components for RAG

LangChain RAG is built from seven components that snap together. Here's what each one does and why it exists.

Document Loaders

Document loaders pull in your source material: PDFs (PyPDFLoader), web pages (WebBaseLoader), Notion pages, CSV files, or databases. Each loader returns a list of Document objects chunks of text plus metadata (source file, page number, etc.). LangChain ships loaders for most common formats, so you don't write parsing code from scratch.

Text Splitters (Chunking)

LLMs have limited context windows, and retrieving an entire 200-page PDF is wasteful. Text splitters break documents into smaller chunks. The most common is RecursiveCharacterTextSplitter, which splits on paragraph, then sentence, then word boundaries, with configurable chunk_size (typically 500-1000 characters) and chunk_overlap (typically 50-100 characters) so sentences aren't cut in half.

This is where most LangChain RAG quietly goes wrong. Chunks too big bury the answer in noise. Chunks too small lose context. The right size depends on your content, and it's worth experimenting.

Embeddings

An embedding model turns each chunk into a vector a list of numbers that captures what the text means. Similar meanings produce similar vectors, which is what makes semantic search possible. LangChain wraps embedding models from multiple providers:

• OpenAI: text-embedding-3-small or text-embedding-3-large — high quality, requires an API key and costs money.
• Hugging Face (open-source): sentence-transformers/all-MiniLM-L6-v2 and similar models — free to run locally, good quality, no API cost.

The choice depends on your budget and whether you need to keep data private (local models keep everything on your machine).

Build an AI-First Career, Master the Complete Skillset

Choose from our industry-leading programs designed for career success

NSDC Certified

Modern Software and AI Engineering Program

Master full-stack development with AI integration

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

Modern Data Science and ML with specialisation in AI

Advanced data science techniques with AI specialization

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

Advanced AIML with Specialisation in Agentic AI

Deep dive into AIML with focus on Agentic systems

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

DevOps, Cloud & AI Platform Engineering

Build and manage AI-powered cloud infrastructure

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

NSDC Certified

AI Engineering Advanced Certification by IIT-Roorkee

Premier AI engineering certification from IIT-Roorkee

3 MonthsDuration

AI-LedCurriculum

Career SupportSupport

Go to Program

NSDC Certified

AI Forward Deployed Engineer Program

Full-stack engineering, production AI and client-facing consulting

12 MonthsDuration

AI-LedCurriculum

Career SupportSupport

+1000 more

Go to Program

Vector Stores

A vector store stores the embedded vectors and finds the most similar ones to a query in milliseconds. LangChain integrates with multiple options:

• FAISS (Meta AI): open-source, runs in memory, great for local development and prototyping.
• Chroma: open-source, lightweight, easy to set up.
• Pinecone: hosted, managed, production-ready, costs money.

The key method is .as_retriever(), which converts a vector store into a retriever object that the rest of the chain can query.
For a deeper look at how embeddings and vector databases work together, that guide covers the retrieval layer in more technical detail.

Retrievers

A retriever is the object that fetches the top-k most relevant chunks for a given query. It embeds the query using the same model, searches the vector store, and returns the closest matches. Two common search modes:

• Similarity search: returns the k most similar chunks by cosine similarity. Simple, fast, usually good enough.
• MMR (Maximum Marginal Relevance): returns chunks that are both relevant and diverse, reducing redundancy. Useful when your knowledge base has overlapping content.

Prompt Templates

A prompt template defines the instruction the LLM sees. For RAG, the template typically includes the retrieved context and the user's question, with an instruction like "answer only using the context above." LangChain's ChatPromptTemplate makes it easy to inject retrieved chunks into the prompt dynamically.

Chains and LCEL

Chains are how the pieces connect. LangChain Expression Language (LCEL) uses a pipe (|) syntax to chain components together: the retriever fetches chunks, the prompt template injects them, and the LLM generates the answer. The modern way to build RAG in LangChain is create_retrieval_chain, which wires the retriever, prompt, and LLM into a single callable object.

┌──────────────┐    ┌──────────────┐    ┌──────────────┐  
│  Document    │───▶│  Text        │───▶│  Embeddings  │  
│  Loaders     │    │  Splitters   │    │              │  
└──────────────┘    └──────────────┘    └──────────────┘  
                                              │  
                                              ▼  
┌──────────────┐    ┌──────────────┐    ┌──────────────┐  
│  Chain /     │◀───│  Prompt      │◀───│  Vector      │  
│  LCEL        │    │  Template    │    │  Store \+     │  
│              │    │              │    │  Retriever   │  
└──────────────┘    └──────────────┘    └──────────────┘  
       │  
       ▼  
┌──────────────┐  
│  LLM         │  
│  (Generated  │  
│   Answer)    │  
└──────────────┘

**LangChain RAG components: document loader, text splitter, embeddings, vector store, retriever, prompt template and chain**  

Build a RAG Pipeline with LangChain (Step by Step)

Here's a minimal, working LangChain RAG pipeline. It loads a document, splits it, embeds it, stores the vectors, retrieves relevant chunks, and generates a grounded answer. The whole thing fits on one screen.

▶ Watch it built live: Complete RAG Tutorial | Indexes, Embeddings, Vectors & Retrieval Step by Step

What You Need Before You Start

• Python 3.9+
• An LLM API key (OpenAI, Anthropic, etc.) or a local model via Ollama
• An embedding model (OpenAI or open-source via Hugging Face)
• A vector store (FAISS for local development)

Install the packages:
pip install langchain langchain-community langchain-openai faiss-cpu

Note: LangChain's API moves fast. This code was written against LangChain v0.2+. Check the official LangChain RAG tutorial for the current signatures.

The Six Steps, in Code

# Step 1: Load documents

# Step 2: Split into chunks

# Step 3: Embed and store in a vector database

# Step 4: Create a retriever

Context: {context}

Question: {input}
""")

# Step 6: Ask a question

That's a working LangChain RAG pipeline. Six steps, one screen, grounded answers from your own documents.
For hands-on projects that build on this foundation, see GenAI projects to build to get you hired.

Turning It into a Conversational RAG Chatbot

The basic pipeline answers one question at a time. A conversational RAG chatbot remembers previous messages, so follow-up questions like "what about part-time employees?" work naturally.

LangChain handles this with a history-aware retriever: it rewrites the follow-up question using the chat history before searching the vector store. The key addition:

from langchain.chains import create_history_aware_retriever
from langchain_core.prompts import MessagesPlaceholder

# Rewrite the question using chat history
contextualize_prompt = ChatPromptTemplate.from_messages([
("system", "Given the chat history, reformulate the user's question to be standalone."),
MessagesPlaceholder("chat_history"),
("human", "{input}"),
])

history_aware_retriever = create_history_aware_retriever(
llm, retriever, contextualize_prompt
)

Then use history_aware_retriever instead of the plain retriever in the chain. The rest stays the same. The chat history is passed as a list of messages with each invocation.

Advanced LangChain RAG Patterns (When the Basics Aren't Enough)

The basic retrieve-then-generate chain works for most use cases. When it doesn't, LangChain supports several advanced patterns.

Agentic RAG

Instead of a fixed retrieve-then-generate chain, an agent decides whether to retrieve, what to retrieve, and when to stop. It can loop: retrieve, check the answer, retrieve again from a different source if needed. It can also call tools (a calculator, a code interpreter, an API) alongside retrieval.

LangChain implements this with create_react_agent or LangGraph, where the retriever becomes one tool among several. The agent decides the best action at each step, rather than following a fixed pipeline.

For a deeper look at how agentic AI systems are structured, that guide covers the architecture behind agent-driven RAG.

Graph RAG

Standard RAG retrieves over flat text chunks. Graph RAG retrieves over a knowledge graph, where entities and relationships are connected. This helps with multi-hop questions: "Which employees report to the manager who approved the remote work policy?" requires following relationships, not just finding a similar chunk.

LangChain integrates with graph databases like Neo4j, and the pattern involves extracting entities and relationships from documents, storing them as a graph, and querying the graph alongside (or instead of) vector search.

Hiring Partners:

GoogleAmazonMicrosoftFlipkartAdobe1200+ more

Multimodal RAG

Standard RAG handles text. Multimodal RAG handles images, tables, charts, and audio. The embedding model processes multiple modalities, and a single vector store holds embeddings from all of them. A text query can retrieve an image, and an image query can retrieve relevant text.
LangChain supports this through multimodal embedding models and document loaders that extract images and tables from PDFs and other documents.

Re-ranking

Vector search returns a rough top-k. A re-ranker scores those candidates more precisely and reorders them so the single best chunk sits at the top of the prompt. It costs a little latency but measurably improves answers.

LangChain integrates with re-ranking models (Cohere Rerank, cross-encoders from Hugging Face) through the ContextualCompressionRetriever. This is the step beginners usually skip, and it often improves answer quality more than switching models.

Why Your LangChain RAG Isn't Working (and How to Fix It)

This is the section most tutorials skip. Here are the five most common failure modes and the fix for each.

Bad Chunking

• The symptom: the model gives vague, incomplete, or irrelevant answers even though the right information is in your documents.
• The cause: chunks are too big (the answer is buried in noise) or too small (the answer is split across chunks and neither half is useful on its own).
• The fix: experiment with chunk_size and chunk_overlap. Start with 500 characters and 50 overlap. If answers lose context, increase the size. If answers are noisy, decrease it. A chunk should be roughly one self-contained idea.

Weak Retrieval

• The symptom: the model never sees the right chunk, so it guesses.
• The cause: the embedding model doesn't fit your domain, the query needs rewriting, or k (the number of chunks retrieved) is too low.
• The fix: try a different embedding model (domain-specific if available), increase k from 4 to 8, and check whether the right chunk is actually in the vector store. If the chunk isn't there, the problem is upstream (loading or splitting), not retrieval.

No Re-ranking

• The symptom: the right chunk is retrieved but buried in fourth or fifth place, where the model ignores it.
• The cause: vector similarity search returns results by cosine similarity, which isn't always the best ranking for relevance.
• The fix: add a re-ranking step. LangChain's ContextualCompressionRetriever with a Cohere or cross-encoder re-ranker pushes the best evidence to the top of the prompt. This one change often improves answer quality more than switching LLMs.

Hallucination When Nothing Relevant Is Found

• The symptom: the model confidently answers a question even though the retrieved chunks don't contain the answer.
• The cause: the LLM is filling in from its training data when the context is insufficient.
• The fix: explicit instruction in the prompt: "If the context doesn't contain the answer, say 'I don't know' or 'not found in the documents.'" Also set a similarity threshold below which you return nothing, rather than passing irrelevant chunks to the model.

Version and API Drift

• The symptom: code that worked last month throws import errors or unexpected arguments.
• The cause: LangChain's API changes frequently. Classes get renamed, modules get reorganised, and methods get deprecated.
• The fix: pin your LangChain version in requirements.txt and check the official LangChain RAG tutorial when something breaks. The docs are updated more reliably than third-party tutorials. This is the honest reality of building with a fast-moving framework.

LangChain RAG vs Building RAG from Scratch

Should you use LangChain, or build RAG with raw API calls?

The honest verdict: most people should start with LangChain. It gets you a working pipeline in minutes and teaches you the components. When you need more control or hit API drift, you can always replace individual LangChain components with raw code. The framework is a starting point, not a cage.

For a broader view of how RAG differs from agentic AI, that guide covers where RAG fits in the larger AI landscape.