How Retrieval-Augmented Generation Works

The Retrieval-Augmented Generation Paradigm

Retrieval-Augmented Generation (RAG) resolves two primary limitations of Large Language Models: knowledge cut-off dates and hallucinations. Instead of relying solely on the static facts encoded in the model’s weights during pre-training, a RAG system retrieves relevant documents from an external storage index and embeds them directly inside the prompt context before inference.

graph TD
    UserQuery[User Query] --> Embeddings[Query Embeddings Generator]
    Embeddings --> VectorSearch[Vector Index Semantic Search]
    VectorSearch --> ContextInjection[Context Packing & Prompt Construction]
    ContextInjection --> LLMGen[LLM Token Generation]
    LLMGen --> Response[Final Answer Output]

---

1. The Ingestion Pipeline

To make external documents searchable, they must be processed into queryable formats:

1. Document Parsing: Extracting raw text from unstructured formats (PDFs, Markdown files, HTML sheets, or Word documents).
2. Text Chunking: Splitting raw text into smaller, overlapping segments (e.g. 512-token chunks with 10% overlap). Chunking ensures the semantic density remains high and prevents context window overflow.
3. Embedding Vector Generation: Passing each text chunk through a dense encoder model (e.g. text-embedding-3-small) to compile a high-dimensional vector representation (typically 1536 float values).
4. Vector Store Indexing: Storing these coordinate float vectors alongside their raw text metadata inside a dedicated index structure (such as HNSW index trees in Qdrant or Pinecone).

---

2. The Retrieval Loop

When a user submits a search request, the live retrieval operation executes:

1. Query Encoding: The incoming user question is passed through the same embedding model used during ingestion to generate a query vector $q$.
2. Cosine Similarity Search: The vector database calculates the similarity distance between $q$ and target index vectors $d_i$ using dot products or cosine distance formulas: $\text{Similarity}(q, d_i) = \frac{q \cdot d_i}{\|q\| \|d_i\|}$
3. Top-K Retrieval: The database returns the $K$ chunks (typically 3 to 5) with the highest similarity scores.

---

3. Context Injection and LLM Prompt Generation

The retrieved document chunks are formatted as a background context panel and concatenated directly to the original user query:

[System Instructions]
You are a factual systems engineer. Answer the user query using only the provided context.

[Retrieved Context Documents]
Document 1: EFS is a managed NFS file system supporting concurrent mounts...
Document 2: EBS volumes are blocked storage bound to a single availability zone...

[User Question]
Which storage system supports shared multi-host mounts?

The compiled prompt is then dispatched to the LLM. The model processes the injected documents to generate a highly accurate, grounded answer with zero hallucinations.