Prerequisites
- Python 3.10+
- Install:
pip install -U langchain langchain-openai langchain-community chromadb - An OpenAI (or other provider) API key
Step 1: Load & split your documents
from langchain_community.document_loaders import DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
docs = DirectoryLoader("./knowledge_base", glob="**/*.md").load()
chunks = RecursiveCharacterTextSplitter(
chunk_size=800, chunk_overlap=100
).split_documents(docs)Step 2: Embed & store in a vector database
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
vectorstore = Chroma.from_documents(chunks, OpenAIEmbeddings())
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})Step 3: 2-Step RAG (retrieve, then generate)
from langchain.chat_models import init_chat_model
model = init_chat_model("openai:gpt-4o-mini")
def answer(question: str) -> str:
docs = retriever.invoke(question)
context = "\n\n".join(d.page_content for d in docs)
prompt = f"Context:\n{context}\n\nQuestion: {question}"
return model.invoke(prompt).content
answer("How do I reset my API key?")Step 4: Agentic RAG (model decides when to retrieve)
from langchain.agents import create_agent
from langchain.tools import tool
@tool
def search_docs(query: str) -> str:
"""Search the internal knowledge base for relevant passages."""
results = retriever.invoke(query)
return "\n\n".join(d.page_content for d in results)
agent = create_agent(
model="openai:gpt-4o-mini",
tools=[search_docs],
system_prompt="Use search_docs before answering questions about our product.",
)Which pattern should you use?
- Use 2-Step RAG for predictable, single-lookup Q&A such as FAQs or documentation bots.
- Use Agentic RAG when the model needs to decide whether, and how many times, to search before answering.
Retrieval quality is usually the real bottleneck, not the model. Test different chunk sizes and different values of k before assuming the LLM is the problem.
