The search mechanics that power every RAG system — in depth
What This Module Covers
Last module gave you the big picture of how vector databases work.
This module goes one level deeper — into the actual search mechanics you will control as a developer.
When you query a vector database in your RAG application — you make decisions about:
How many results to return → Top-K
How to filter by metadata → Metadata Filtering
How to balance speed vs quality → Search Parameters
Getting these right is the difference between a RAG system that works and one that works well.
Part 1 — Top-K Search
What it Means
Top-K simply means:
Return the K most similar vectors to my query.
K = 1 → return only the single best match
K = 5 → return top 5 most similar results
K = 10 → return top 10 most similar results
You've already seen this. But let's go deeper into what it actually returns and how you use it.
What a Top-K Result Looks Like
When you query a vector database with K=5, you get back something like this:
// Query result — top 5 most similar chunks [ { id: "handbook_chunk_47", score: 0.9421, // cosine similarity score — how close this is to your query // 0.94 = very similar
text: "Common side effects of aspirin include nausea, stomach pain, and heartburn.In rare cases...", // the original text chunk that was stored
metadata: { source: "medical_handbook.pdf", page: 23, category: "side_effects", lastUpdated: "2024-01" } // all the extra info stored alongside the vector }, { id: "research_paper_chunk_12", score: 0.8876, text: "Aspirin has been associated with gastrointestinal bleeding in approximately 2% of long- term users...", metadata: { source: "research_paper_2023.pdf", page: 7, category: "clinical_studies", lastUpdated: "2023-11" } }, { id: "drug_guide_chunk_89", score: 0.8234, text: "Patients taking aspirin regularly should be aware of potential bleeding risks...", metadata: { source: "drug_guide.pdf", page: 45 } }, { id: "faq_chunk_3", score: 0.7123, text: "Is aspirin safe for everyone? Aspirin is not recommended for children under 16...", metadata: { source: "faq.pdf", page: 2 } }, { id: "history_chunk_5", score: 0.6234, text: "The history of aspirin dates back to ancient willow bark used by Egyptians...", metadata: { source: "aspirin_history.pdf", page: 1 } } ]
Notice the scores drop from 0.94 down to 0.62. The first three are clearly relevant. The last two are less relevant but still returned because K=5.
Choosing the Right K — Practical Guide
This is a real decision you make in every RAG system:
K too small (e.g. K=1 or K=2):
Risk: Miss important context
Example: User asks a complex question that needs
information from multiple document sections
→ K=1 only returns one chunk
→ LLM doesn't have enough context
→ Answer is incomplete
K too large (e.g. K=20 or K=50):
Risk 1: Context window fills up
→ 20 chunks × 500 words each = 10,000 words
→ Might exceed context limit
→ Definitely expensive
Risk 2: Noisy context confuses the LLM
→ Irrelevant chunks get included
→ LLM gets confused by contradictory information
→ Answer quality drops
Risk 3: Higher cost
→ More tokens injected = more tokens billed
The sweet spot for most RAG applications:
K = 3 to 5 → good starting point for most use cases
K = 5 to 10 → when questions need broader context
K = 1 to 3 → when you want precise, focused answers
In production you'll tune this based on testing. Start with K=5 and adjust.
Score Thresholds — Filtering by Quality
Top-K always returns exactly K results — even if some are not very relevant.
To avoid injecting low-quality chunks into your LLM prompt — you can add a score threshold:
const results = await vectorDB.query({ vector: questionEmbedding, topK: 10, // get up to 10 results
scoreThreshold: 0.70, // but only return results with similarity score above 0.70 // results below 0.70 are dropped even if K hasn't been reached });
// Example outcome: // Requested K=10 but only 4 chunks scored above 0.70 // → returns 4 results (not 10) // Better to have 4 relevant chunks than 10 with 6 bad ones
Typical thresholds:
> 0.90 → nearly identical meaning (very strict)
> 0.75 → very relevant (strict — good for focused tasks)
> 0.65 → reasonably relevant (balanced — good default)
> 0.50 → somewhat related (loose — more results, less precise)
< 0.50 → probably not relevant (usually not worth including)
Part 2 — Metadata Filtering
Why Metadata Filtering Exists
Pure vector search finds semantically similar content. But sometimes similarity alone is not enough.
Real scenarios where you need more:
Scenario 1 — Multi-tenant app
You have 100 companies using your RAG product.
Each company's documents must stay separate.
User from Company A must NEVER see Company B's data.
Pure vector search:
→ Might return similar chunks from ANY company ❌
Vector search + metadata filter:
→ WHERE companyId = "company_A" ✓
Scenario 2 — Time-sensitive information
User asks about "current tax rates"
Your database has tax documents from 2019, 2021, 2023.
Pure vector search:
→ Returns most semantically similar — might return 2019 doc ❌
Vector search + metadata filter:
→ WHERE year = 2023 ✓ (only search recent documents)
Scenario 3 — Category-specific search
Medical app — user is a doctor asking about pediatric dosage.
Pure vector search:
→ Returns all aspirin-related content including adult dosage ❌
Vector search + metadata filter:
→ WHERE patientType = "pediatric" ✓
How Metadata Filtering Works
The filter runs at the same time as the vector search — not after it.
WITHOUT filter:
Search ALL vectors → find top K similar → return results
WITH filter:
Search ONLY vectors matching the filter → find top K similar → return results
The filter narrows down the search space FIRST.
Then similarity search runs within that space.
This is important to understand because it affects your results:
Example:
Total vectors: 100,000
After filter (category = "medical"): 8,000 vectors
Top K from those 8,000: 5 results
You searched 8,000 not 100,000.
Results are all medical AND most similar to your question.
Metadata Filtering Syntax
Different vector databases have slightly different syntax — but the concepts are the same.
Here are the common filter operations:
Exact match:
filter: { category: "medical" } // only return vectors where category equals "medical"
Multiple conditions (AND):
filter: { category: "medical", language: "english" } // category = "medical" AND language = "english"
Comparison operators:
filter: { year: { $gte: 2022 }, // year greater than or equal to 2022
pageCount: { $lte: 100 } // pageCount less than or equal to 100 }
OR conditions:
filter: { $or: [ { category: "medical" }, { category: "pharmaceutical" } ] } // category = "medical" OR category = "pharmaceutical"
IN operator:
filter: { source: { $in: ["handbook.pdf", "guidelines.pdf", "faq.pdf"] } } // source is one of these three files
NOT operator:
filter: { category: { $ne: "archived" } } // category is NOT "archived"
Designing Good Metadata — Practical Tips
The metadata you store when indexing determines what filters you can apply when searching.
Think ahead — what filters will your users need?
Bad metadata design:
// Storing too little { source: "document1.pdf" } // Can only filter by filename — not very useful
Good metadata design:
// Storing everything useful { source: "medical_handbook_2023.pdf", // which file this came from
page: 47, // which page in the file
chunkIndex: 3, // which chunk on that page (for ordering)
category: "side_effects", // what type of content
documentType: "handbook", // what type of document
specialty: "cardiology", // medical specialty
lastUpdated: "2024-01-15", // when this document was last updated
authorVerified: true, // quality flag
language: "english", // language of the content
companyId: "company_abc", // for multi-tenant apps — critical for data isolation }
Good metadata makes your RAG system dramatically more useful.
Part 3 — Search Quality Parameters
Beyond K and filters — vector databases expose parameters that let you control the tradeoff between search speed and accuracy.
The Speed vs Accuracy Tradeoff
Remember HNSW from last module — it skips most vectors to search fast.
But how many vectors does it check? You control this.
Parameter: ef (in HNSW — also called efSearch)
→ Higher ef = check more candidates = more accurate = slower
→ Lower ef = check fewer candidates = faster = slightly less accurate
ef = 10 → very fast, ~90% accuracy
ef = 50 → fast, ~97% accuracy ← good default
ef = 200 → slower, ~99% accuracy
ef = 500 → slow, ~99.9% accuracy
For most RAG applications — the default ef setting is fine. You'd only tune this if:
- You need maximum accuracy (medical, legal) → increase ef
- You have extremely high traffic needs → decrease ef
Reranking — A Second Pass for Better Results
This is an advanced technique but important to know about early.
The problem with ANN search alone:
ANN search returns top 5 by vector similarity.
But vector similarity is not perfect.
Sometimes a chunk that's "semantically similar"
is not actually the best answer to the specific question.
The solution — Reranking:
Step 1 — Initial ANN search
→ Get top 20 results by vector similarity (fast)
→ Cast a wide net
Step 2 — Reranker model
→ Take those 20 candidates
→ Run a more expensive but more accurate model
→ Score each one specifically for this question
→ Re-order them
Step 3 — Take top 5 from reranked results
→ Now you have the best 5 from the best 20
→ Much higher quality than top 5 from ANN alone
Without reranking:
ANN returns [A, B, C, D, E] by vector similarity
→ C and D might not actually answer the question well
With reranking:
ANN returns top 20: [A, B, C, D, E, F, G, H...]
Reranker reorders: [A, F, B, K, E] ← F and K moved up
→ Better answers, even if they weren't top 5 by similarity
We'll use reranking in Phase 5 when building the full RAG pipeline.
Part 4 — Putting It All Together
Here is a complete, realistic vector database query for a RAG system:
async function searchDocuments(userQuestion, userId, category) {
// Step 1 — Embed the user's question const questionEmbedding = await getEmbedding(userQuestion); // questionEmbedding = [0.23, -0.87, 0.41, ...] (1536 numbers)
// Step 2 — Query the vector database const searchResults = await vectorDB.query({
vector: questionEmbedding, // the question as a vector
topK: 10, // get up to 10 candidates
scoreThreshold: 0.65, // only results with similarity above 0.65
filter: { userId: userId, // CRITICAL — only search this user's documents // prevents data leakage between users
category: category, // only search within the relevant category
isArchived: { $ne: true }, // exclude archived documents },
includeMetadata: true, // return metadata alongside results
includeValues: false, // don't return the actual vectors (saves bandwidth) });
// searchResults is now an array of up to 10 objects // each has: id, score, text, metadata
// Step 3 — Filter out low quality results const relevantChunks = searchResults .filter(result => result.score > 0.65) // double-check threshold (some DBs handle this internally)
.slice(0, 5); // take only top 5 from the results
// relevantChunks is now array of 5 most relevant chunks // example: // [ // { score: 0.94, text: "Aspirin side effects include...", ... }, // { score: 0.88, text: "Common aspirin reactions are...", ... }, // ... // ]
return relevantChunks; }
Using the Results in Your LLM Prompt
Once you have the chunks — here's how they flow into the LLM:
async function answerQuestion(userQuestion, userId, category) {
// Get relevant chunks from vector DB const chunks = await searchDocuments(userQuestion, userId, category);
if (chunks.length === 0) { return "I couldn't find relevant information to answer your question."; // Handle the case where nothing relevant was found }
// Build context string from chunks const context = chunks .map((chunk, index) => `[Source ${index + 1}: ${chunk.metadata.source}, Page ${chunk.metadata.page}] ${chunk.text}` ) .join("\n\n");
// context now looks like: // "[Source 1: medical_handbook.pdf, Page 23] // Common side effects of aspirin include nausea... // // [Source 2: research_paper.pdf, Page 7] // Aspirin has been associated with gastrointestinal bleeding..."
// Build the LLM prompt const prompt = `Answer the user's question using ONLY the context provided below. If the answer is not in the context, say "I don't have that information." Always mention which source your answer comes from.
CONTEXT: ${context}
QUESTION: ${userQuestion}
ANSWER:`;
// Send to LLM const response = await openai.chat.completions.create({ model: "gpt-4o", messages: [{ role: "user", content: prompt }], temperature: 0.1, // low temperature = more factual, less creative // good for RAG where accuracy matters });
return response.choices[0].message.content; }
This is the complete RAG search-to-answer flow. Everything we've learned in Phase 3 and Phase 4 comes together here.
The Complete Mental Model — Search to Answer
User question
↓
Embed question → vector [1536 numbers]
↓
Query Vector DB:
→ topK: 5-10
→ scoreThreshold: 0.65+
→ filter: {userId, category, ...}
↓
ANN search (HNSW) runs in ~2ms
↓
Get back: text chunks + scores + metadata
↓
(Optional) Rerank for better ordering
↓
Take top 3-5 chunks
↓
Build prompt: system + context + question
↓
Send to LLM (temperature: 0.1)
↓
LLM generates answer grounded in real context
↓
User gets accurate, sourced answer
3-Line Summary
- Top-K controls how many similar chunks you retrieve — K=3 to 5 is the sweet spot for most RAG systems — too small misses context, too large adds noise and cost — score thresholds ensure only genuinely relevant chunks are returned.
- Metadata filtering lets you combine semantic similarity with structured conditions — like userId for data isolation, date for freshness, or category for relevance — this runs before the vector search to narrow the search space first.
- For production quality, reranking adds a second pass — get top 20 by ANN search, then use a more precise model to reorder them, then take the top 5 — this gives much better results than ANN similarity alone.
Module 4.3 — Complete ✅
Coming up — Module 4.4 — Hands-on: Working With Chroma (Local Vector Database)
Theory is done. Now we write real code. We'll set up Chroma locally, store real embeddings, run real similarity searches, and use metadata filters — all in JavaScript. By the end you'll have a working local vector database with documents stored and searchable.