Honest Write-Up: Why “Just Retrieve More Chunks” Doesn’t Scale in RAG
RAG top-k retrieval precision is the real cost hiding behind the “just retrieve more chunks” advice — bump k too high and precision quietly collapses even as recall keeps climbing.
This is the first in an occasional “Honest Write-Up” series on NexByteLab: when an experiment fails, fails to scale, or doesn’t behave the way the tutorials and vendor docs say it will, that result goes into the post right alongside the wins. We’ve quietly been doing this already — the INT8 quantization piece found real accuracy gains but no energy savings, and the model collapse simulation showed exactly how badly recursive synthetic training degrades a model. This post makes that stance explicit and adds a new case: retrieval-augmented generation’s most common piece of “just do this” advice.
Concept
Almost every RAG tutorial repeats some version of the same advice: if your answers are missing information, retrieve more chunks. Bump top-k from 5 to 10, or 10 to 20, “to be safe.” It’s treated as a free dial — more retrieved context can only help, or at worst do nothing.
The falsifiable hypothesis for this experiment: precision@k degrades as k increases, and it degrades sharply once k exceeds the number of documents that are actually relevant to the query — meaning “retrieve more” quietly stops being free well before most people notice, because recall keeps climbing (looks like progress) while precision is already collapsing (looks fine until you check).
This follows directly from our earlier chunk-size RAG retrieval experiment, which tested how document splitting affects retrieval; this one holds chunking fixed and instead tests the other lever everyone reaches for — how many chunks to pull back per query.
Experiment
A synthetic but controlled setup, so the ground truth is known exactly (no LLM judging its own retrieval): 8 topics (database indexing, Kubernetes networking, gradient descent, supply chain, plant biology, credit risk, compiler design, climate modeling), 25 documents per topic (200 documents total), each document built from its topic’s vocabulary plus realistic noise — 40% of each document’s words are pulled from one or two other topics, so documents aren’t artificially clean. 40 queries (5 per topic), built the same way from topic vocabulary.
Retrieval is TF-IDF + cosine similarity — the same reproducible, dependency-light method used in the chunk-size post — run at k ∈ {1, 3, 5, 10, 20, 40, 80}. For each k, we measure precision@k (what fraction of the retrieved chunks are actually on-topic) and recall@k (what fraction of the topic’s 25 relevant documents got retrieved).
vectorizer = TfidfVectorizer()
doc_matrix = vectorizer.fit_transform(corpus) # 200 synthetic docs, 8 topics
query_matrix = vectorizer.transform(queries) # 40 queries
sims = cosine_similarity(query_matrix, doc_matrix)
for k in [1, 3, 5, 10, 20, 40, 80]:
for qi, true_topic in enumerate(query_topics):
top_k = np.argsort(-sims[qi])[:k]
relevant = (doc_topics[top_k] == true_topic).sum()
precision = relevant / k
recall = relevant / DOCS_PER_TOPIC # 25Results
Real output, one run, seeded for reproducibility (200 documents, 40 queries, 8 topics, 25 relevant documents per topic):
| k | precision@k | recall@k |
|---|---|---|
| 1 | 1.000 | 0.040 |
| 3 | 1.000 | 0.120 |
| 5 | 1.000 | 0.200 |
| 10 | 0.997 | 0.399 |
| 20 | 0.976 | 0.781 |
| 40 | 0.614 | 0.982 |
| 80 | 0.312 | 0.998 |
This is the part we’re not going to round off: TF-IDF retrieval was genuinely excellent through k=20, holding 97.6% precision — better than we expected given how much cross-topic noise was baked into every document. The advertised behavior mostly held. But between k=20 and k=40, precision fell off a cliff: 0.976 → 0.614, and by k=80 it’s 0.312 — meaning more than two out of three retrieved chunks are off-topic. Recall, meanwhile, keeps climbing the whole time and is essentially saturated (0.982, 0.998) by the point precision has already broken. Anyone watching only recall, or only “did the right document show up somewhere in the results,” would see nothing but good news the entire time.
Explanation
The mechanism here isn’t exotic — it’s arithmetic. Once k exceeds the number of documents that are actually relevant to a query (25, in this corpus), the retriever is mathematically forced to fill the rest of the slots with something, and “something” means the next-closest off-topic matches. The cliff sits exactly at k=20–40, straddling that 25-document relevant set, which is the experiment behaving exactly as the math predicts rather than as a fluke of this particular corpus.
This matches what’s being written about production RAG systems right now, not just this toy corpus. theneuralbase’s breakdown of top-k retrieval makes the same point directly: “the top-k cutoff is arbitrary: k=5 may include 3 wrong chunks; k=10 may include 7,” because vector and lexical similarity score query-document closeness, not answer quality — a chunk can rank high on overlap while containing no actual answer. And the effect doesn’t stop at the retriever: the well-known “Lost in the Middle” study (Liu et al.) found that LLMs handling long, multi-document contexts already struggle to use information buried in the middle of what they’re given — so the irrelevant chunks a high-k retrieval step dumps into the prompt aren’t neutral padding, they’re actively competing for the model’s attention with the answer that’s actually in there.
The practical takeaway isn’t “use a small k.” It’s that k should be set relative to how many documents are actually likely to be relevant to a given query — a fixed k=50 “to be safe” is exactly the setting most likely to bury a correct answer in noise, not protect against missing one. The standard fix in production systems is a second-stage reranker: retrieve broadly (say k=20–50) to protect recall, then rerank down to the 3–5 chunks that are actually worth putting in front of the model — which is the same two-stage shape theneuralbase’s piece recommends.
Resources
- Why the top-k retrieved chunks are not always the most relevant — the precision-dilution mechanism, and the retrieve-then-rerank fix
- Lost in the Middle: How Language Models Use Long Contexts (Liu et al.) — why irrelevant retrieved context actively hurts, not just wastes tokens
- RAG Metrics: Measure & Improve Your Pipeline — precision@k, recall@k, and F1@k defined with production examples
- scikit-learn TfidfVectorizer docs — the retrieval method used in this experiment
Related Reading
- How Much Does Chunk Size Really Affect RAG Retrieval? — same TF-IDF methodology, the other major lever in RAG retrieval tuning
- Building Your First RAG Agent — where top-k is usually set once and forgotten in a real pipeline
- INT8 Quantization Keeps Accuracy Intact, But Doesn’t Automatically Save Energy — another Honest Write-Up: the claim that held, and the one that didn’t
- Honest Write-Up: The Law of Large Numbers Doesn’t Always Save You — same series, a much older theorem that also breaks under the wrong assumptions
