How the right chunks reach the LLM — and how to inject them properly
Where We Are
So far in Phase 5:
Module 5.1 → Understood what RAG is and why it exists
Module 5.2 → Learned how to split documents into chunks
Module 5.3 → THIS MODULE
How to retrieve the right chunks
How to inject them into the LLM prompt
You have chunks stored in a vector database. A user asks a question. Now what?
This module covers exactly what happens between the question and the answer.
The Retrieval Step — Full Picture
Retrieval is not just "search vector DB and take results." There are decisions to make at every step.
User Question
↓
Clean and prepare the question
↓
Embed the question
↓
Search vector DB (with filters if needed)
↓
Get top K results
↓
Check quality — are scores good enough?
↓
(Optional) Rerank results
↓
Take final top N chunks
↓
Context Injection → build prompt
↓
Send to LLM
↓
Answer
Let's go through each step.
Step 1 — Clean and Prepare the Question
The user's raw question is not always ideal for search.
User types: "umm so like what r the side effects??"
Problems:
→ "umm", "so", "like" = noise words
→ "r" instead of "are" = informal spelling
→ "??" = extra punctuation
→ "the side effects" — side effects of WHAT?
For simple cases — you can search with the raw question. It works surprisingly well because embedding models handle informal language.
But for production apps — you can add a preprocessing step:
async function prepareQuery(rawQuestion, openai) { // rawQuestion = exactly what user typed // example: "umm what r the side effects of aspirin??"
// Option 1 — Simple cleaning (fast, free) const cleaned = rawQuestion .replace(/[^\w\s?.,!]/g, " ") // remove weird characters — keep letters, numbers, basic punctuation .replace(/\s+/g, " ") // replace multiple spaces with single space .trim(); // remove leading/trailing whitespace // example result: "umm what r the side effects of aspirin"
return cleaned;
// Option 2 — LLM rewriting (slower, costs tokens, better quality) // Useful when users ask vague or poorly worded questions const response = await openai.chat.completions.create({ model: "gpt-4o", temperature: 0, // temperature 0 = consistent, deterministic output messages: [ { role: "system", content: `You rewrite user questions to be clear and specific. Fix spelling, remove filler words, make the question precise. Return only the rewritten question — nothing else.` }, { role: "user", content: rawQuestion // example: "umm what r the side effects of aspirin??" } ] });
return response.choices[0].message.content; // example result: "What are the side effects of aspirin?" // much cleaner — will embed better → better search results }
Step 2 — Embed the Question
You already know this from Phase 3 and 4. Same process:
async function embedQuery(question, openai) { // question = cleaned user question string // example: "What are the side effects of aspirin?"
const response = await openai.embeddings.create({ model: "text-embedding-3-small", // MUST be same model used when embedding chunks during indexing // mixing models = garbage results
input: question, });
return response.data[0].embedding; // returns array of 1536 numbers // example: [0.19, -0.82, 0.38, 0.71, ...] × 1536 }
Step 3 — Search Vector Database
async function retrieveChunks(queryVector, collection, options = {}) {
const { topK = 10, // get top 10 candidates first // we'll filter down after checking scores
scoreThreshold = 0.5, // only keep results with similarity above 0.5 // below 0.5 = probably not relevant
filter = null, // optional metadata filter // example: { category: "medication", source: "handbook.pdf" } } = options; // destructure options with defaults
// Query Chroma const results = await collection.query({ queryEmbeddings: [queryVector], // our question as a vector
nResults: topK, // get top 10 first — we'll trim down after
where: filter || undefined, // apply metadata filter if provided // undefined means no filter
include: ["documents", "metadatas", "distances"], });
// results.ids[0] = array of matched IDs // results.documents[0] = array of matched text chunks // results.metadatas[0] = array of metadata objects // results.distances[0] = array of distance scores (lower = more similar)
// Convert results to cleaner format const chunks = results.ids[0].map((id, index) => ({ id: id, // example: "med_001_chunk_3"
text: results.documents[0][index], // the actual text of this chunk // example: "Aspirin can cause stomach bleeding..."
metadata: results.metadatas[0][index], // example: { source: "handbook.pdf", page: 23 }
similarity: 1 - results.distances[0][index], // convert distance to similarity score // distance 0.06 → similarity 0.94 }));
// Filter by score threshold const relevant = chunks.filter(chunk => chunk.similarity >= scoreThreshold); // keep only chunks with similarity above our threshold // example: threshold 0.5 → drops anything below 50% similar
console.log(`Retrieved: ${results.ids[0].length} total`); console.log(`After threshold filter: ${relevant.length} chunks\n`);
return relevant; // returns array of relevant chunk objects // each has: id, text, metadata, similarity }
Step 4 — Check Quality
Before injecting chunks into the LLM — check if you actually found anything useful.
This is important. If no relevant chunks were found — don't pretend otherwise.
function checkRetrievalQuality(chunks, question) { // chunks = array of retrieved chunks with similarity scores // question = original user question
if (chunks.length === 0) { // nothing found at all return { hasResults: false, quality: "none", message: "No relevant information found in the knowledge base." }; }
const topScore = chunks[0].similarity; // highest similarity score // example: 0.87
if (topScore < 0.5) { // best result is still below 50% similar // probably not relevant return { hasResults: false, quality: "poor", message: "Found some results but they don't seem relevant to the question." }; }
if (topScore < 0.7) { // results exist but not very confident return { hasResults: true, quality: "medium", message: "Found some potentially relevant information." }; }
// topScore >= 0.7 — good results return { hasResults: true, quality: "good", message: "Found relevant information." }; }
Step 5 — Context Injection
This is where most developers make mistakes.
Context injection = taking your retrieved chunks and building them into the LLM prompt correctly.
There are three ways to do this — from bad to best:
Way 1 — Bad: Just dump chunks in
// ❌ BAD — don't do this const prompt = ` ${chunks.map(c => c.text).join(" ")}
Question: ${question} Answer:`;
Problems:
→ No clear separation between chunks
→ LLM doesn't know where one chunk ends and another begins
→ No source information
→ No instruction on how to use the context
→ LLM might ignore context and use its own knowledge
Way 2 — OK: Basic structure
// ⚠️ OK — works but not great const context = chunks.map(c => c.text).join("\n\n");
const prompt = `Context: ${context}
Question: ${question} Answer:`;
Better — but still missing source info and clear instructions.
Way 3 — Best: Proper structure with sources and instructions
function buildPrompt(question, chunks) { // question = user's question string // chunks = array of retrieved chunk objects
// Build context section with numbered sources const contextSection = chunks .map((chunk, index) => { // format each chunk as a numbered source return `[Source ${index + 1}] File: ${chunk.metadata.source} Page: ${chunk.metadata.page} Relevance Score: ${(chunk.similarity * 100).toFixed(0)}%
${chunk.text}`; // example output: // [Source 1] // File: medical_handbook.pdf // Page: 23 // Relevance Score: 94% // // Aspirin can cause stomach bleeding with long term use... }) .join("\n\n─────────────────────\n\n"); // separator between sources — makes it clear where each one ends
// System prompt — tells LLM how to behave const systemPrompt = `You are a helpful assistant that answers questions based strictly on the provided context.
Rules you must follow: 1. Answer ONLY using information from the context below 2. If the answer is not in the context — say "I don't have that information in my knowledge base" 3. Always mention which source your answer comes from (Source 1, Source 2, etc.) 4. Never make up information or use knowledge outside the context 5. If context is partially relevant — use what's relevant and ignore the rest 6. Keep your answer clear and concise`;
// User prompt — context + question const userPrompt = `Here is the relevant information from the knowledge base:
───────────────────── ${contextSection} ─────────────────────
Based only on the above information, please answer this question: ${question}`;
return { systemPrompt, userPrompt }; // returns both prompts separately // systemPrompt → goes in "system" role // userPrompt → goes in "user" role }
Step 6 — Send to LLM and Get Answer
async function generateAnswer(systemPrompt, userPrompt, openai) {
const response = await openai.chat.completions.create({ model: "gpt-4o", // use gpt-4o for best quality answers
temperature: 0.1, // very low temperature = factual, consistent answers // we want accuracy not creativity for RAG
max_tokens: 1000, // limit response length // 1000 tokens ≈ 750 words — enough for most answers
messages: [ { role: "system", content: systemPrompt, // behavior instructions for the LLM }, { role: "user", content: userPrompt, // context + question } ], });
return response.choices[0].message.content; // the LLM's answer as a string // example: "According to Source 1 (medical_handbook.pdf, page 23), // aspirin can cause stomach bleeding with long-term use..." }
The Complete Retrieval Pipeline — All Steps Together
async function ragQuery(userQuestion, collection, openai) { // userQuestion = what the user typed // collection = Chroma collection with stored chunks // openai = OpenAI client
console.log(`\n${"=".repeat(60)}`); console.log(`Question: "${userQuestion}"`); console.log("=".repeat(60));
// ── STEP 1: Clean question ──────────────────────────────── const cleanedQuestion = userQuestion .replace(/\s+/g, " ") .trim(); console.log(`\n📝 Cleaned question: "${cleanedQuestion}"`);
// ── STEP 2: Embed question ──────────────────────────────── console.log("\n🔢 Embedding question..."); const queryVector = await embedQuery(cleanedQuestion, openai); // queryVector = [0.19, -0.82, ...] (1536 numbers) console.log(" Done — question converted to vector");
// ── STEP 3: Retrieve chunks ─────────────────────────────── console.log("\n🔍 Searching knowledge base..."); const chunks = await retrieveChunks(queryVector, collection, { topK: 5, // get top 5 results scoreThreshold: 0.4, // keep results above 40% similarity });
// ── STEP 4: Check quality ───────────────────────────────── const quality = checkRetrievalQuality(chunks, cleanedQuestion);
if (!quality.hasResults) { // no good results found — don't hallucinate console.log(`\n⚠️ ${quality.message}`); return { answer: quality.message, chunks: [], quality: quality.quality }; }
// Show what was retrieved console.log(`\n📚 Retrieved ${chunks.length} relevant chunks:`); chunks.forEach((chunk, i) => { console.log(`\n Source ${i + 1}: ${chunk.metadata.source} (Page ${chunk.metadata.page})`); console.log(` Similarity: ${(chunk.similarity * 100).toFixed(1)}%`); console.log(` Preview: ${chunk.text.substring(0, 80)}...`); });
// ── STEP 5: Build prompt ────────────────────────────────── console.log("\n📋 Building prompt with context..."); const { systemPrompt, userPrompt } = buildPrompt(cleanedQuestion, chunks);
const totalTokensEstimate = Math.round( (systemPrompt.length + userPrompt.length) / 4 ); // rough token estimate: 1 token ≈ 4 characters console.log(` Estimated prompt size: ~${totalTokensEstimate} tokens`);
// ── STEP 6: Generate answer ─────────────────────────────── console.log("\n🤖 Generating answer with GPT-4o..."); const answer = await generateAnswer(systemPrompt, userPrompt, openai);
console.log("\n💬 ANSWER:"); console.log("─".repeat(60)); console.log(answer); console.log("─".repeat(60));
return { answer: answer, // the final answer string
chunks: chunks, // the chunks used to generate the answer
quality: quality.quality // "good" or "medium" }; }
Seeing It All Together — Real Example Output
When you run this pipeline with the question "What are the side effects of aspirin?":
============================================================
Question: "What are the side effects of aspirin?"
============================================================
📝 Cleaned question: "What are the side effects of aspirin?"
🔢 Embedding question...
Done — question converted to vector
🔍 Searching knowledge base...
Retrieved: 5 total
After threshold filter: 3 chunks
📚 Retrieved 3 relevant chunks:
Source 1: medical_handbook.pdf (Page 23)
Similarity: 94.2%
Preview: Common side effects of aspirin include nausea, stomach pain...
Source 2: drug_guide.pdf (Page 7)
Similarity: 87.6%
Preview: Aspirin has been associated with gastrointestinal bleeding...
Source 3: faq.pdf (Page 2)
Similarity: 71.3%
Preview: Aspirin is not recommended for children under 16 due to...
📋 Building prompt with context...
Estimated prompt size: ~820 tokens
🤖 Generating answer with GPT-4o...
💬 ANSWER:
────────────────────────────────────────────────────────────
According to Source 1 (medical_handbook.pdf, page 23),
common side effects of aspirin include nausea, stomach pain,
and heartburn.
Source 2 (drug_guide.pdf, page 7) also notes that aspirin
has been associated with gastrointestinal bleeding, particularly
with long-term use.
Additionally, as mentioned in Source 3 (faq.pdf, page 2),
aspirin is not recommended for children under 16 due to the
risk of Reye's syndrome.
────────────────────────────────────────────────────────────
Every claim is sourced. No hallucination. The LLM only said what was in the retrieved chunks.
Why Context Injection Order Matters
Remember from Module 1.3 — the "Lost in the Middle" problem?
LLM pays most attention to:
→ Beginning of context ← HIGH attention
→ Middle of context ← LOW attention ⚠️
→ End of context ← HIGH attention
So always put your BEST chunks first and last:
function orderChunksForInjection(chunks) { // chunks = already sorted by similarity (highest first)
if (chunks.length <= 2) return chunks; // if 2 or fewer — order doesn't matter much
// Best chunk → first position (high attention) // Second best → last position (high attention) // Rest → middle (lower attention — but still there)
const best = chunks[0]; // highest similarity chunk const secondBest = chunks[1]; const rest = chunks.slice(2); // remaining chunks
return [...rest, secondBest, best]; // order: medium chunks in middle, best chunk at end // LLM pays most attention to end → best chunk gets most focus }
This small optimization can meaningfully improve answer quality for long context windows.
Context Window Budget Planning
Before building the prompt — plan your token budget:
function planContextBudget(chunks, maxContextTokens = 8000) { // maxContextTokens = how many tokens to use for context // leave room for: system prompt (~300) + question (~100) + answer (~500) // so for 128k context window: 8000 tokens for context is safe
let usedTokens = 0; const selectedChunks = [];
for (const chunk of chunks) { const chunkTokens = Math.ceil(chunk.text.length / 4); // estimate tokens: 1 token ≈ 4 characters
if (usedTokens + chunkTokens > maxContextTokens) { console.log(`Stopping at ${selectedChunks.length} chunks — budget reached`); break; // stop adding chunks when budget is reached }
selectedChunks.push(chunk); usedTokens += chunkTokens; }
console.log(`Using ${selectedChunks.length} chunks (~${usedTokens} tokens)`); return selectedChunks; }
3-Line Summary
- Retrieval is not just "search and take results" — you clean the question, embed it, search with filters, check quality scores, and filter out low-relevance chunks before anything reaches the LLM.
- Context injection structure matters enormously — number your sources, include file names and page numbers, give the LLM explicit instructions to only use the provided context and cite sources — this prevents hallucination and makes answers verifiable.
- The "Lost in the Middle" problem means LLMs pay less attention to context in the middle — put your highest-similarity chunks at the beginning and end of the context section for best results.
Module 5.3 — Complete ✅
Coming up — Module 5.4 — Hallucination & Grounding
We go deep on the single biggest problem in production AI — hallucination. What exactly causes it, how RAG reduces it, what grounding means, and the specific techniques you use in code to prevent your AI application from confidently making things up.