Honest Write-Up: The Law of Large Numbers Doesn’t Always Save You
This law of large numbers failure is easy to miss because it only shows up once your data has fat tails — the sample mean never settles down no matter how many samples you add.
Third in the Honest Write-Up series: when a result fails, fails to scale, or doesn’t hold up the way it’s taught, it goes in the post next to the wins. Previous entries: retrieving more chunks doesn’t scale in RAG, and INT8 quantization’s accuracy claim held but its energy claim didn’t. This one goes further back than any ML technique — it’s the 300-year-old theorem most of statistics is built on.
Concept
The Law of Large Numbers (LLN) is arguably the single most-cited justification in applied statistics: “average enough samples and you’ll converge on the truth.” The Central Limit Theorem (CLT) is its close partner: “average enough samples and that average becomes normally distributed, with a spread that shrinks like 1/√n.” Together they’re the reason A/B tests, polling, and “just collect more data” all feel safe by default.
Both theorems quietly assume the underlying distribution has a finite mean (LLN) and finite variance (CLT). Most datasets people reach for in a stats class satisfy this without anyone thinking about it. Falsifiable hypothesis for this experiment: for a distribution that violates that assumption — the Cauchy distribution, which has infinite variance and no defined mean — the running sample mean will not converge as n grows, and the spread of the sample mean will not shrink with more data, no matter how large the sample gets.
Experiment
Deterministic, seeded, pure NumPy, runs in under a second. Two distributions: standard Normal (finite mean and variance — LLN/CLT should hold) and standard Cauchy (undefined mean, infinite variance — they shouldn’t). Two tests: track the running/cumulative mean as a single sample grows from n=10 to n=200,000; and, across 2,000 independent trials per n, measure how much the sample mean’s spread (IQR) shrinks as n grows from 10 to 100,000.
normal_path = rng.normal(0, 1, N_MAX)
cauchy_path = rng.standard_cauchy(N_MAX)
for n in [10, 100, 1_000, 10_000, 100_000, 200_000]:
print(normal_path[:n].mean(), cauchy_path[:n].mean())
# and, across 2000 independent trials per n:
normal_means = rng.normal(0, 1, size=(2000, n)).mean(axis=1)
cauchy_means = rng.standard_cauchy(size=(2000, n)).mean(axis=1)Results
Real output from one run. Running mean as n grows (single long sample):
| n | normal mean | cauchy mean |
|---|---|---|
| 10 | −0.2023 | 2.6901 |
| 100 | −0.1729 | 0.1639 |
| 1,000 | −0.0723 | −0.8080 |
| 10,000 | −0.0123 | 9.5339 |
| 100,000 | −0.0013 | −0.7217 |
| 200,000 | 0.0010 | 0.1516 |
The normal mean walks calmly toward 0, exactly as expected. The Cauchy mean does not trend anywhere — at n=10,000 it’s out at 9.53, then at n=100,000 (ten times more data) it’s back at −0.72. More data did not make it more accurate.
Spread of the sample mean across 2,000 trials, per n:
| n | normal IQR | expected ~1/√n | cauchy IQR |
|---|---|---|---|
| 10 | 0.397 | 0.430 | 1.993 |
| 100 | 0.135 | 0.140 | 2.067 |
| 1,000 | 0.043 | 0.041 | 2.135 |
| 10,000 | 0.013 | 0.013 | 1.971 |
| 100,000 | 0.004 | 0.005 | 2.015 |
The normal IQR tracks the CLT’s 1/√n prediction almost exactly. The Cauchy IQR sits at ~2.0 at n=10 — and at ~2.0 at n=100,000. Ten thousand times more data, zero reduction in spread. As a sanity check on how real this is: the fraction of trial means landing outside [−10, 10] stayed at 5.6–7.5% whether n was 10 or 100,000 — collecting more data didn’t shrink the tail risk of a wildly wrong estimate at all.
Explanation
This isn’t noise or a small-sample artifact — it’s the Cauchy distribution’s defining property. It’s stable: the average of n independent Cauchy draws is itself Cauchy-distributed, with exactly the same spread as a single draw. Averaging performs no averaging. The mechanism is that the LLN requires E|X| to be finite, and the CLT requires finite variance; Cauchy’s tails are heavy enough that both integrals diverge, so neither theorem’s precondition is met — the theorems aren’t “weakened,” they simply don’t apply.
This isn’t just a stats-class curiosity. The QuantEcon course materials on heavy tails run the identical Cauchy simulation and reach the identical conclusion — “the sample mean shows no sign of converging” — and tie it directly to asset returns, which are well documented to be heavier-tailed than a normal distribution assumes. Nassim Taleb’s technical note on the Law of Large Numbers under fat tails makes the broader practical point: once you’re in fat-tailed territory, the sample mean converges so slowly (if the mean is even defined) that “collect more data” stops being reliable advice, and a handful of extreme observations can dominate the entire estimate no matter how large n gets. The practical warning for anyone doing applied ML or analytics: things like network latency, ad revenue per user, viral content reach, and extreme financial returns are routinely heavier-tailed than a quick histogram suggests — and “the average will stabilize with more samples” is exactly the assumption that quietly breaks there.
Resources
- Heavy-Tailed Distributions — QuantEcon — runs the same Cauchy convergence simulation, with a proof sketch of why LLN fails
- The Law of Large Numbers Under Fat Tails — Nassim Taleb — the practical consequences for anyone relying on sample averages
- When Heavy Tails Disrupt Statistical Inference — The American Statistician (2024) — broader survey of where this bites standard inference
- NumPy random.Generator.standard_cauchy docs — the distribution used in this experiment
Related Reading
- Honest Write-Up: Why “Just Retrieve More Chunks” Doesn’t Scale in RAG — the previous entry in this series: more isn’t automatically better, part one
- INT8 Quantization Keeps Accuracy Intact, But Doesn’t Automatically Save Energy — same series, a claim that only half-held once measured
- How Much Real Data Stops Model Collapse? — another seed-averaged simulation where the honest result mattered more than the headline
- Reproducing Double Descent: Why 300 Features Beat 39 on the Same 40 Data Points — another case where a textbook guarantee breaks down right where the parameter/data ratio gets interesting
