Agentic RAG: The Complete Pipeline Architecture

Introduction

Naive RAG — chunk documents, embed them, store in a vector DB, retrieve top-K, stuff into the LLM prompt — works well enough for demos but breaks down in production. Recall degrades when:

• Queries are ambiguous or require multi-hop reasoning across documents
• Domain vocabulary is not well represented in the embedding model
• The most relevant document chunk was split across a chunk boundary
• The user query implies a time constraint or entity that requires structured filtering

Agentic RAG treats retrieval not as a single lookup but as a dynamic, multi-step reasoning loop where the model can issue multiple retrieval queries, synthesize intermediate results, and decide when it has enough context to answer.

---

Full Pipeline Architecture

User Query
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│                     QUERY UNDERSTANDING                      │
│  • Intent classification (factual / analytical / creative)  │
│  • Entity extraction (dates, names, products, versions)     │
│  • Query decomposition for multi-hop questions              │
│  • HyDE: Generate hypothetical answer for embedding         │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│                      HYBRID RETRIEVAL                        │
│                                                             │
│  Dense Retrieval (Semantic)    Sparse Retrieval (Lexical)   │
│  ┌──────────────────────┐      ┌──────────────────────┐     │
│  │ Query → text-emb-3   │      │ BM25 / BM42 index    │     │
│  │ → HNSW vector search │      │ TF-IDF keyword match │     │
│  │ → cosine similarity  │      │ exact term matching  │     │
│  └──────────────────────┘      └──────────────────────┘     │
│            │                             │                   │
│            └──────────┬──────────────────┘                   │
│                       ▼                                     │
│              Reciprocal Rank Fusion (RRF)                   │
│              score = Σ 1/(k + rank_i)                       │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│                    CROSS-ENCODER RERANKING                   │
│  • Input: (query, candidate_chunk) pairs                    │
│  • Model: bge-reranker-v2-m3, Cohere Rerank, ColBERT        │
│  • Full attention over query+document (no compression loss) │
│  • Discard bottom K, keep top-N for LLM context            │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│                    CONTEXT CONSTRUCTION                      │
│  • Fetch parent chunks (small-to-big retrieval)            │
│  • Deduplicate overlapping chunks                           │
│  • Inject structured metadata (source, date, section)      │
│  • Respect max context window budget                        │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│                    AGENTIC LOOP (optional)                   │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ LLM evaluates: "Is context sufficient to answer?"  │   │
│  │   YES → Generate final answer                      │   │
│  │   NO  → Generate follow-up retrieval query         │   │
│  │          → Re-run pipeline with refined query       │   │
│  │          → Loop up to max_iterations (default: 3)   │   │
│  └─────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
┌─────────────────────────────────────────────────────────────┐
│                       GENERATION + GROUNDING                 │
│  • LLM generates answer with citations ([1], [2], ...)      │
│  • Hallucination guard: verify claims against source chunks │
│  • Structured output: {"answer": ..., "sources": [...]}     │
└─────────────────────────────────────────────────────────────┘

---

Step-by-Step Implementation

flowchart TD
    A["1. Document Ingestion & Chunking Strategy"]
    B["2. Hybrid Search Setup (Qdrant + BM42)"]
    C["3. Cross-Encoder Reranking"]
    D["4. Agentic Loop with Sufficiency Check"]
    A --> B
    B --> C
    C --> D

Step 1: Document Ingestion & Chunking Strategy

The most impactful decision in any RAG pipeline is chunk strategy. Naive fixed-size chunking loses semantic coherence.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

# Option A: Recursive character split (fast, good baseline)
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,        # tokens, not chars
    chunk_overlap=64,      # 12.5% overlap for boundary continuity
    separators=["\n\n", "\n", ". ", " ", ""],  # hierarchy-aware
)

# Option B: Semantic chunking (split on meaning breaks, not size)
semantic_splitter = SemanticChunker(
    embeddings=OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,  # split when embedding distance > 95th percentile
)

# Small-to-big: store small child chunks for retrieval, large parent for context
def create_parent_child_chunks(documents):
    parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=128)
    child_splitter  = RecursiveCharacterTextSplitter(chunk_size=256,  chunk_overlap=32)
    parent_chunks = parent_splitter.split_documents(documents)
    child_chunks  = []
    for i, parent in enumerate(parent_chunks):
        children = child_splitter.split_documents([parent])
        for child in children:
            child.metadata["parent_id"] = i   # link child → parent
        child_chunks.extend(children)
    return parent_chunks, child_chunks

Step 2: Hybrid Search Setup (Qdrant + BM42)

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, SparseVectorParams

client = QdrantClient(url="http://localhost:6333")

# Collection with BOTH dense and sparse vectors
client.create_collection(
    collection_name="knowledge_base",
    vectors_config={
        "dense": VectorParams(size=1536, distance=Distance.COSINE),
    },
    sparse_vectors_config={
        "sparse": SparseVectorParams(),  # BM42 sparse vectors
    }
)

def hybrid_search(query: str, top_k: int = 20):
    # Dense embedding
    dense_vec = embed_model.embed_query(query)   # [1536 floats]
    
    # Sparse (BM42) — keyword weighted token vectors
    sparse_vec = sparse_model.encode(query)       # SparseVector(indices, values)
    
    results = client.query_points(
        collection_name="knowledge_base",
        prefetch=[
            Prefetch(query=dense_vec, using="dense", limit=top_k),
            Prefetch(query=sparse_vec, using="sparse", limit=top_k),
        ],
        query=FusionQuery(fusion=Fusion.RRF),  # Reciprocal Rank Fusion
        limit=top_k,
    )
    return results

Step 3: Cross-Encoder Reranking

from FlagEmbedding import FlagReranker

reranker = FlagReranker("BAAI/bge-reranker-v2-m3", use_fp16=True)

def rerank_results(query: str, candidates: list[str], top_n: int = 5) -> list[str]:
    pairs = [[query, doc] for doc in candidates]
    scores = reranker.compute_score(pairs, normalize=True)   # [0.0, 1.0]
    ranked = sorted(zip(scores, candidates), reverse=True)
    return [doc for _, doc in ranked[:top_n]]

Step 4: Agentic Loop with Sufficiency Check

from openai import OpenAI

client_llm = OpenAI()

SUFFICIENCY_PROMPT = """You are a retrieval judge. Given the user's question and the retrieved context, decide:
- SUFFICIENT: The context contains all information needed to answer the question accurately.
- INSUFFICIENT: Critical information is missing. Provide a refined search query to retrieve the missing information.

Respond as JSON: {"decision": "SUFFICIENT"|"INSUFFICIENT", "refined_query": "..." (if INSUFFICIENT)}"""

def agentic_rag(user_query: str, max_iterations: int = 3) -> dict:
    query = user_query
    all_context = []
    
    for i in range(max_iterations):
        # Retrieve
        candidates = hybrid_search(query, top_k=20)
        top_chunks  = rerank_results(query, candidates, top_n=5)
        all_context.extend(top_chunks)
        
        # Judge sufficiency
        judgment = client_llm.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": SUFFICIENCY_PROMPT},
                {"role": "user", "content": f"Question: {user_query}\n\nContext:\n" + "\n---\n".join(all_context)}
            ],
            response_format={"type": "json_object"}
        )
        result = json.loads(judgment.choices[0].message.content)
        
        if result["decision"] == "SUFFICIENT":
            break
        
        # Refine query for next iteration
        query = result.get("refined_query", user_query)
    
    # Final generation with citations
    response = client_llm.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer based solely on the provided context. Cite sources as [1], [2], etc."},
            {"role": "user", "content": f"Question: {user_query}\n\nContext:\n" + "\n---\n".join(
                [f"[{i+1}] {chunk}" for i, chunk in enumerate(all_context)]
            )}
        ]
    )
    return {"answer": response.choices[0].message.content, "sources": all_context}

---

Key Takeaways

• Hybrid Search (Dense + Sparse) + RRF is the single highest-impact upgrade over naive dense-only retrieval — typically improves recall@10 by 15–25%.
• Cross-encoder rerankers recover precision lost by approximate vector search — the trade-off is ~100ms added latency per rerank.
• Small-to-big retrieval: retrieve small chunks (high precision), return their large parent chunks (full context) to the LLM.
• Agentic loops are necessary for multi-hop questions that require chaining evidence across multiple documents — limit iterations to 3 to control latency.
• HyDE (Hypothetical Document Embeddings): generating a hypothetical answer and embedding it instead of the raw query dramatically improves recall for short or ambiguous queries.