This chunk size RAG retrieval experiment measures exactly where the precision/context trade-off appears, instead of relying on intuition about chunking.

Chunk size is one of the first settings you encounter when building a Retrieval-Augmented Generation system.

And the advice sounds deceptively simple:

Use smaller chunks for precise retrieval.

Or:

Use larger chunks so the model gets enough context.

Both sound reasonable.

Neither tells us where the trade-off actually appears.

So instead of choosing a chunk size by intuition, I wanted to measure it.

The question was simple:

How much does chunk size affect retrieval quality when everything else is held constant?

To isolate that variable, I deliberately used a simple TF-IDF retrieval pipeline rather than a dense embedding model.

That choice is important.

This experiment is not intended to establish the universally best chunk size for RAG.

It asks a narrower and more reproducible question:

If the retrieval method, documents and queries remain fixed, what changes when we alter only chunk size?

1. Why Chunking Exists

A RAG system usually cannot treat every source document as one enormous retrieval unit.

Documents are therefore divided into smaller pieces:

document

↓

chunks

↓

retrieval index

↓

query

↓

relevant chunks

↓

LLM context

Chunking creates a trade-off.

If chunks are too small, they may lose useful context.

If chunks are too large, they may contain substantial irrelevant text.

The retriever then has to rank these chunks against the user’s query.

So chunk size affects what the retrieval system is actually allowed to retrieve.

2. The Intuition

Suppose a document contains this paragraph:

The Apollo 11 mission launched on 16 July 1969 from Kennedy Space Center. Neil Armstrong, Buzz Aldrin and Michael Collins formed the crew. Armstrong and Aldrin landed on the Moon while Collins remained in lunar orbit.

Now imagine the query:

Who remained in lunar orbit during Apollo 11?

A sufficiently large chunk may contain:

the mission,
all three astronauts,
the landing,
and the answer.

That is useful context.

But imagine that same passage sits inside a 2,000-token chunk containing several unrelated sections.

The answer is still there.

But the useful signal occupies a smaller proportion of the retrieved text.

At the opposite extreme, tiny chunks may produce:

“while Collins remained”

without enough surrounding context to identify what the sentence refers to.

So there is a natural tension:

smaller chunks → more focused retrieval

but potentially:

less context

while:

larger chunks → more context

but potentially:

more irrelevant material.

3. The Research Question

The experiment asks:

How does retrieval performance change as chunk size increases?

The hypothesis is not:

“Smaller chunks are always better.”

Instead:

There should be a trade-off: very small chunks may fragment useful context, while very large chunks may dilute query-relevant terms with unrelated text.

The location of that trade-off depends on the retrieval system and dataset.

4. Why Use TF-IDF?

Modern production RAG systems often use dense embedding models.

So why use TF-IDF?

Because the purpose of this experiment is isolation.

TF-IDF is:

deterministic,
inexpensive,
transparent,
reproducible,
easy to inspect.

It removes several additional variables:

embedding model choice,
embedding dimensionality,
API changes,
model version,
remote inference,
proprietary ranking behaviour.

That gives us a cleaner experiment around chunk size itself.

But it creates an important limitation:

Results from TF-IDF retrieval should not automatically be assumed to hold for dense embedding retrieval.

Dense retrievers represent semantic similarity differently from lexical TF-IDF retrieval.

The experiment therefore establishes a chunk-size effect for this retrieval setup, not a universal RAG law.

5. The Experimental Setup

Keep the following fixed:

source documents,
query set,
relevance labels,
retrieval method,
vectorisation settings,
ranking procedure,
evaluation metric.

Change only:

chunk size

For example:

64

128

256

512

1024

tokens or approximately equivalent text units, depending on the chunking implementation.

For every chunk size:

split the same documents,
build the retrieval index,
run the same queries,
retrieve the top-k chunks,
calculate retrieval metrics.

That makes chunk size the primary manipulated variable.

6. Chunking Changes the Retrieval Units

This is easy to overlook.

When we change chunk size, we do not merely change a numerical hyperparameter.

We change the objects being ranked.

With small chunks:

one document → many retrieval units

With large chunks:

one document → fewer retrieval units

This affects:

how much information each candidate contains,
how many candidates exist,
term frequency,
ranking competition,
and the granularity of relevance.

So chunk size is tightly coupled to the structure of the retrieval problem itself.

7. Build the TF-IDF Index

A simple implementation can use scikit-learn.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

vectorizer = TfidfVectorizer(
    lowercase=True,
    stop_words="english"
)

chunk_matrix = vectorizer.fit_transform(
    chunks
)

query_vector = vectorizer.transform(

[query]

) scores = cosine_similarity( query_vector, chunk_matrix ).flatten()

Then rank chunks by similarity:

ranked_indices = scores.argsort()[::-1]

This gives us a deliberately simple retriever.

No LLM is involved in retrieval.

No embedding API is involved.

That is intentional.

8. What Should We Measure?

A RAG retrieval experiment should not be evaluated by whether a few examples “look good.”

We need explicit metrics.

Recall@k

Of the relevant information we wanted to retrieve, how often did it appear within the top k retrieved chunks?

Conceptually:

Recall@k = relevant items retrieved in top k / relevant items available

Precision@k

Of the k chunks retrieved:

How many were actually relevant?

Precision@k = relevant chunks in top k / k

Hit Rate@k

For each query:

Did at least one relevant chunk appear in the top k?

This is useful when retrieving one supporting passage is sufficient.

Mean Reciprocal Rank

If the first relevant result appears at rank r:

RR = 1/r

Average that across queries to obtain:

MRR.

Together, these metrics give us a better picture than a single retrieval score.

9. Why Recall Alone Can Mislead

Suppose we retrieve more and more text.

Eventually, the answer is likely to appear somewhere.

Recall rises.

That sounds good.

But if we retrieve:

1 useful chunk,
plus 9 irrelevant chunks,

we have increased the amount of noise passed downstream.

So retrieval quality is not simply:

“Did we find the answer?”

It is also:

“How much irrelevant material did we retrieve along with it?”

That matters because the retrieved context will eventually be given to a language model.

10. Small Chunks: Precision vs Fragmentation

Smaller chunks have an obvious advantage.

They can isolate relevant sentences or paragraphs.

A query about one fact may retrieve a focused passage instead of an entire section.

This can improve retrieval precision.

But small chunks introduce another problem:

fragmentation.

A useful statement may depend on the text immediately before or after it.

For example:

Chunk A:

“The experiment was repeated across five independent seeds.”

Chunk B:

“The resulting mean accuracy was 91.4%.”

If these become separate retrieval units, retrieving Chunk B alone loses information about how the result was produced.

The smaller the chunk, the greater the risk of separating facts from the context needed to interpret them.

11. Large Chunks: Context vs Dilution

Large chunks preserve more context.

That can be useful when an answer depends on several related sentences.

But they also contain more unrelated material.

For lexical retrieval such as TF-IDF, this can dilute the relative importance of the terms matching the query.

A large chunk may contain the correct answer while also containing hundreds of unrelated words.

So larger chunks can trade retrieval focus for contextual completeness.

Again, neither direction is universally better.

12. What the Experiment Found

Across the tested chunk sizes, retrieval behaviour changed meaningfully.

Very small chunks could produce focused matches but sometimes fragmented useful context.

As chunks became larger, each retrieved unit contained more surrounding information.

Beyond a certain point, however, the additional text could reduce retrieval precision because relevant information was packaged together with increasingly unrelated content.

The resulting pattern was not:

“The smallest chunk always wins.”

Nor was it:

“The largest chunk always wins.”

Instead, the experiment showed a trade-off between retrieval granularity and contextual completeness.

That is the more useful result.

13. About the 512-Token Result

Chunk sizes around 512 tokens are frequently used in RAG examples and practical systems.

They can also perform well in particular experiments.

But that does not make:

512 tokens

a universal optimum.

The best chunk size depends on:

document structure,
query type,
retriever,
embedding model,
overlap,
top-k,
downstream LLM,
evaluation metric.

So if an external benchmark or this experiment finds good performance around 512 tokens, the correct interpretation is:

512 was effective under those experimental conditions.

Not:

“512 is the correct chunk size for RAG.”

There is no reason to expect one number to be optimal for every corpus.

14. TF-IDF Fragmentation Is Not Dense-Retrieval Fragmentation

This distinction is particularly important.

TF-IDF relies heavily on lexical overlap.

Suppose the query says:

“What causes myocardial infarction?”

and the document says:

“A heart attack commonly occurs when blood flow to part of the heart is blocked.”

A lexical system may struggle because:

myocardial infarction

and:

heart attack

are different strings.

A dense embedding retriever may recognise the semantic relationship.

Therefore, chunk-size effects observed with TF-IDF may change when dense embeddings are used.

The same general issues—fragmentation, context and retrieval granularity—still exist.

But their magnitude and optimum may differ.

That is why the experiment should be read as:

evidence that chunk size materially affects retrieval

rather than:

a universal measurement of the best chunk size for all RAG systems.

15. Overlap Changes the Experiment Again

Chunk size is not the only chunking parameter.

Consider:

chunk size = 256

with:

overlap = 0

versus:

chunk size = 256

with:

overlap = 64

These are not equivalent retrieval systems.

Overlap can preserve information that crosses chunk boundaries.

For example:

Chunk 1: tokens 1–256

Chunk 2: tokens 193–448

The shared region reduces the probability that a useful idea is split exactly at a boundary.

But overlap also:

increases the number of chunks,
increases index size,
creates redundancy,
can cause nearly identical chunks to occupy multiple top-k positions.

So chunk size should eventually be studied together with overlap.

16. Top-k Interacts With Chunk Size

Suppose:

k = 5

With 128-token chunks, the retriever may return roughly:

640 tokens

before overlap and formatting.

With 512-token chunks:

2,560 tokens

With 1,024-token chunks:

5,120 tokens

So holding k constant does not hold retrieved context volume constant.

This matters.

A fair comparison can therefore be designed in at least two ways.

Fixed number of chunks

Keep:

top-k = constant

This measures the behaviour of the retrieval units themselves.

Fixed context budget

Allow each chunk size to retrieve approximately the same total number of tokens.

This may be more representative of a production RAG system with a limited context budget.

Both experiments are useful.

They answer different questions.

17. Retrieval Quality Is Not Answer Quality

This experiment evaluates retrieval.

It does not directly evaluate the final answer produced by an LLM.

That distinction matters.

A chunking configuration may achieve excellent retrieval metrics but still produce worse answers because:

relevant evidence is fragmented,
too much duplicate context is returned,
the model cannot combine the passages effectively,
irrelevant context distracts generation.

Conversely, slightly lower retrieval precision may still produce excellent answers if the correct evidence is consistently present.

So a complete RAG evaluation should eventually measure:

retrieval quality

and:

answer quality.

18. What This Experiment Actually Shows

The experiment supports several useful conclusions.

1. Chunk size matters

Changing only chunk size can materially change retrieval behaviour.

2. Smaller is not automatically better

Very small chunks can fragment useful context.

3. Larger is not automatically better

Large chunks can package relevant evidence with substantial irrelevant material.

4. There is a trade-off

Retrieval granularity and contextual completeness pull in different directions.

5. The optimum is system-dependent

The best value depends on the corpus, queries, retriever and evaluation objective.

6. TF-IDF results should remain scoped to TF-IDF

The experiment demonstrates the effect under lexical retrieval.

Dense retrieval needs to be tested separately.

19. What This Experiment Does NOT Show

The experiment does not prove:

“512 tokens is the best RAG chunk size.”

It does not prove:

“Small chunks are better than large chunks.”

It does not prove:

“TF-IDF behaves exactly like embedding retrieval.”

It does not prove:

“Better retrieval metrics automatically produce better LLM answers.”

And it does not establish:

a universal chunking rule for every RAG application.

The result is deliberately narrower:

Chunk size materially affects retrieval, and its effect should be measured rather than selected purely by convention.

20. A Better Follow-Up Experiment

The obvious next experiment is:

TF-IDF vs Dense Retrieval

Keep:

the same documents,
same queries,
same relevance labels,
same chunk sizes,
same evaluation metrics.

Change only:

retrieval method

Compare:

TF-IDF

against:

Dense embeddings

Then ask:

Does the chunk-size curve change when retrieval becomes semantic rather than primarily lexical?

That would directly test how far the findings in this experiment generalise.

21. Then Add Chunk Overlap

After that, introduce a second variable:

overlap

For example:

Chunk size	Overlap
128	0
128	32
256	0
256	64
512	0
512	128
1024	0
1024	256

Now measure whether overlap reduces fragmentation enough to justify the additional redundancy.

This becomes a much more realistic RAG optimisation study.

22. Measure Retrieval Under a Fixed Token Budget

A particularly useful production experiment would hold the total retrieved context approximately constant.

For example:

context budget = 2,048 tokens

Then:

128-token chunks → retrieve up to ~16 chunks
256-token chunks → retrieve up to ~8 chunks
512-token chunks → retrieve up to ~4 chunks
1,024-token chunks → retrieve up to ~2 chunks

Now the downstream LLM receives approximately the same amount of text in every condition.

That removes one important confounding variable.

The question becomes:

Given the same context budget, which chunk size gives us the best evidence?

That is probably more useful than asking for a universally optimal chunk size.

23. Test Different Query Types

Chunk size may also interact with the kind of question being asked.

Fact lookup

“When was the company founded?”

Small chunks may work well.

Explanatory question

“Why did the project fail?”

The answer may span several paragraphs.

Comparative question

“How does Product A differ from Product B?”

Evidence may appear in multiple sections.

Multi-hop question

“Which policy applies to employees hired after the merger?”

Answering may require combining several pieces of information.

There may therefore be no single optimal chunk size even within one corpus.

Different query types may prefer different retrieval granularity.

24. The Bigger Lesson

Chunk size looks like a small implementation detail.

It is not.

Chunking determines the unit of retrieval.

And the unit of retrieval determines:

what can be ranked,
what can be retrieved,
what context accompanies the answer,
how much noise reaches the LLM.

That means chunking is part of the retrieval model itself.

The common question:

“What chunk size should I use?”

probably has the wrong shape.

A better question is:

“What chunking strategy performs best for my documents, queries, retriever and context budget?”

That is an experimental question.

25. The Honest Interpretation

The tempting conclusion from a chunk-size experiment is:

“We tested the options and found the best number.”

But that number is rarely portable.

Change the:

documents,
tokenizer,
embedding model,
retrieval algorithm,
overlap,
top-k,
context window,
question distribution,

and the result may change.

So the useful outcome is not a magic chunk size.

It is a method:

Treat chunking as a measurable retrieval hyperparameter rather than a number copied from a tutorial.

That lesson survives even when the exact optimum does not.

Conclusion

Chunk size changes retrieval.

But there is no evidence here for a universal rule such as:

“Always use 512 tokens.”

Small chunks can improve focus while fragmenting context.

Large chunks preserve context while introducing irrelevant material.

The balance depends on the retrieval system.

In this experiment, TF-IDF gives us a transparent way to isolate that trade-off.

The next step is not to assume that dense embeddings will behave identically.

It is to run the comparison.

So if you are building a RAG system, don’t begin with:

“What chunk size does everyone use?”

Begin with:

“What chunk size performs best on the queries my system actually needs to answer?”

Measure that.

Then choose.

References and Further Reading

[1] Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems.
The foundational RAG paper combining retrieval with generation.
https://arxiv.org/abs/2005.11401

[2] Robertson, S. (2004). Understanding Inverse Document Frequency: On Theoretical Arguments for IDF. Journal of Documentation, 60(5), 503–520.
Background on inverse document frequency and lexical information retrieval.
https://doi.org/10.1108/00220410410560582

[3] scikit-learn Documentation. TfidfVectorizer.
Documentation for the TF-IDF implementation used in experiments of this type.
https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html

[4] Karpukhin, V., Oguz, B., Min, S., et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.
An important reference for dense semantic retrieval and a useful contrast with lexical TF-IDF retrieval.
https://arxiv.org/abs/2004.04906

[5] Gao, Y., Xiong, Y., Gao, X., et al. (2023). Retrieval-Augmented Generation for Large Language Models: A Survey.
A broad survey of RAG architectures, retrieval and augmentation strategies.
https://arxiv.org/abs/2312.10997

Reproducibility and Interpretation Note

This experiment uses TF-IDF lexical retrieval to isolate the effect of changing chunk size while keeping the retrieval method fixed.

The results should therefore not be interpreted as establishing the optimal chunk size for dense embedding retrieval or RAG systems generally.

Chunk size interacts with the corpus, query distribution, overlap, retrieval algorithm, top-k setting and downstream context budget.

Any chunk size that performs best in this experiment should therefore be reported as the best-performing value under the tested configuration, not as a universal RAG recommendation.

1. Why Chunking Exists

2. The Intuition

3. The Research Question

4. Why Use TF-IDF?

5. The Experimental Setup

6. Chunking Changes the Retrieval Units

7. Build the TF-IDF Index

8. What Should We Measure?

Recall@k

Precision@k

Hit Rate@k

Mean Reciprocal Rank

9. Why Recall Alone Can Mislead

10. Small Chunks: Precision vs Fragmentation

11. Large Chunks: Context vs Dilution

12. What the Experiment Found

13. About the 512-Token Result

14. TF-IDF Fragmentation Is Not Dense-Retrieval Fragmentation

15. Overlap Changes the Experiment Again

16. Top-k Interacts With Chunk Size

Fixed number of chunks

Fixed context budget

17. Retrieval Quality Is Not Answer Quality

18. What This Experiment Actually Shows

1. Chunk size matters

2. Smaller is not automatically better

3. Larger is not automatically better

4. There is a trade-off

5. The optimum is system-dependent

6. TF-IDF results should remain scoped to TF-IDF

19. What This Experiment Does NOT Show

20. A Better Follow-Up Experiment

TF-IDF vs Dense Retrieval

TF-IDF

Dense embeddings

21. Then Add Chunk Overlap

22. Measure Retrieval Under a Fixed Token Budget

23. Test Different Query Types

Fact lookup

Explanatory question

Comparative question

Multi-hop question

24. The Bigger Lesson

25. The Honest Interpretation

Conclusion

References and Further Reading

Reproducibility and Interpretation Note

Related Reading

Similar Posts

Leave a Reply Cancel reply