How Much Does Chunk Size Really Affect RAG Retrieval? A Reproducible Experiment

This chunk size RAG retrieval experiment exists because nobody actually shows you the numbers.

Every RAG tutorial says the same thing: “pick a good chunk size.” Almost none of them show you what happens when you don’t. This post runs a small, fully reproducible experiment to find out — and then checks the result against what larger published benchmarks have found.

Concept

A retrieval-augmented generation (RAG) pipeline can only answer a question as well as the chunk it retrieves lets it. Split a document into chunks that are too small, and a sentence’s meaning gets severed from the context it depends on. Split it into chunks that are too large, and a single chunk mixes several unrelated ideas together, diluting the vector that’s supposed to represent it.

The hypothesis worth testing is simple: retrieval quality should be low at very small chunk sizes, rise as chunks approach one coherent “thought,” and plateau (or eventually fall again) once chunks get large enough to blur multiple ideas together. Rather than take that on faith, this experiment builds the smallest possible harness to check it.

Experiment

The setup is deliberately minimal so it runs anywhere in a few seconds with no API keys and no GPU:

• A toy knowledge base of 6 short technical documents (~115–130 words each), each on an unrelated topic — database indexing, TCP congestion control, vitamin D metabolism, git rebase vs. merge, JPEG compression, and Raft consensus.
• Each document has one question whose answer lives in a single identifiable sentence.
• Each document is split into chunks at a range of chunk sizes (10–100 words) and overlap ratios (0%, 15%, 30%), using a simple sliding-window word splitter.
• Chunks are indexed with TF-IDF (via scikit-learn) instead of a neural embedding model. This is a deliberate simplification — TF-IDF has no external dependencies, downloads nothing, and runs offline, which keeps the experiment reproducible for anyone. The mechanism being tested — does fragmenting text hurt a retriever’s ability to match a query to the right passage — is representative of dense embedding retrievers too, even though the absolute numbers would differ.
• For each question, we check whether the chunk containing the answer sentence appears in the top 3 chunks ranked by cosine similarity.

The core of the harness:

def chunk_text(text, chunk_size_words, overlap_ratio=0.15):
    words = text.split()
    step = max(1, int(chunk_size_words * (1 - overlap_ratio)))
    chunks = []
    for start in range(0, len(words), step):
        chunk = words[start : start + chunk_size_words]
        if not chunk:
            break
        chunks.append(" ".join(chunk))
        if start + chunk_size_words >= len(words):
            break
    return chunks

def run_trial(chunk_size_words, overlap_ratio, top_k=3):
    # chunk every document, remember which chunks contain
    # the ground-truth answer sentence, then TF-IDF + cosine
    # similarity each question against every chunk and check
    # whether an answer-bearing chunk lands in the top_k
    ...
    vectorizer = TfidfVectorizer(stop_words="english")
    chunk_matrix = vectorizer.fit_transform(all_chunks)
    query_matrix = vectorizer.transform(queries)
    sims = cosine_similarity(query_matrix, chunk_matrix)

The full script (corpus, questions, and sweep loop included) is under 120 lines and reproduces the table below exactly — the corpus is fixed and a random seed is set, so there’s no run-to-run variance to worry about.

Results

Hit-rate (fraction of the 6 questions whose answer-bearing chunk was retrieved in the top 3) at 15% overlap, across chunk sizes:

Chunk size (words)	Avg. actual chunk length	Hit rate
10	9.8	33%
15	14.5	67%
20	18.7	83%
25	24.3	100%
30	27.9	67%
40	37.4	83%
60	54.8	100%
100	65.4	100%

The overall shape matches the hypothesis: retrieval is unreliable when chunks are small enough to cut a sentence in half (10–20 words), then stabilizes at 100% once chunks are large enough to contain a full sentence with a little surrounding context (25+ words). Note the dip back to 67% at 30 words — with only 6 questions, each one is worth ~17 percentage points, so a single unlucky tie-break in the cosine ranking swings the score visibly. That’s a useful reminder on its own: small-sample retrieval evals are noisy, and a single-point “hit rate went down” result needs a bigger eval set before you trust it.

Explanation

Two things are worth separating here: what this toy experiment shows, and what it doesn’t.

What it shows directly: fragmentation is the mechanism. When a chunk boundary falls in the middle of the sentence that contains the answer, TF-IDF similarity to the query drops enough that the chunk falls out of the top 3. Once chunks are comfortably larger than one sentence, that failure mode disappears — for this corpus.

What it doesn’t show: the well-known result from production-scale RAG benchmarks that chunks which are too large also hurt retrieval. That effect needs a corpus large and varied enough that an oversized chunk starts blending multiple unrelated ideas — six 120-word documents are too short for any tested chunk size to reach that regime. Published benchmarks that use hundreds of thousands of tokens across many documents do see this: a 2026 benchmark across 50 academic papers (905,000+ tokens) found recursive character splitting at ~512 tokens with 50–100 tokens of overlap to be the strongest general-purpose baseline (69% accuracy), with the useful nuance that factoid questions do fine at 256–512 tokens while multi-hop, analytical questions benefit from 512–1,024 token chunks, and dense financial documents did best around 1,024 tokens. The same piece flags an important trap: semantic chunking scored highest on raw retrieval recall (91.9%) but worst on end-to-end answer accuracy (54%), because its fragments averaged just 43 tokens — too small to give the generator enough context even when retrieval technically “worked.” That’s the large-scale mirror of the small-corpus effect measured above, just showing up at the generation step instead of the retrieval step.

Weaviate’s guidance lands on the same starting point — 512 tokens with 50–100 token overlap (10–20%) as a baseline — and frames chunk-size tuning as an empirical loop rather than a one-time decision: start from that baseline, measure hit rate/precision/recall on your own queries, and adjust for your document type rather than trusting a single “best” number from any blog post, including this one.

The practical takeaway for your own pipeline: don’t guess a chunk size and move on. Build a harness this small — even TF-IDF and a dozen question/answer pairs from your own documents — and you’ll find your corpus’s fragmentation point directly, instead of importing a number tuned on someone else’s data.

Resources

• RAG Chunking Strategies: The 2026 Benchmark Guide — PremAI
• Chunking Strategies to Improve LLM RAG Pipeline Performance — Weaviate
• Chunking Methods on RAG: Effectiveness vs. Computational Cost (arXiv)
• scikit-learn: TfidfVectorizer docs
• Related on NexByteLab: Building Your First RAG Agent

---

Related Reading

• How Much Real Data Stops Model Collapse? A Minimal Recursive-Training Simulation — a recursive-training simulation using the same small, repeated-trial methodology
• Honest Write-Up: Why “Just Retrieve More Chunks” Doesn’t Scale in RAG — the other major retrieval lever: what happens to precision as top-k grows