INT8 quantization experiment comparing accuracy, speed and energy savings
|

INT8 Quantization Keeps Accuracy Intact, But Doesn’t Automatically Save Energy

BySrinivas Saripalli

This INT8 quantization energy savings check starts from a claim repeated everywhere in efficiency discussions: that shrinking a model’s numerical precision automatically cuts energy use, not just memory.

Quantization gets pitched constantly as a free lunch: shrink a model’s numerical precision, and you get a smaller, cheaper, greener model with “negligible” accuracy loss. The accuracy part of that claim is well studied. The energy part is where the story gets murkier — and it matters, because data-centre electricity demand grew 17% in 2025 alone, with AI-specific facilities growing several times faster than the grid overall, according to the IEA’s 2025 data-centre analysis. If quantization is one of the cheapest levers available for reducing that footprint, it’s worth checking exactly what it buys you, at what precision, and under what conditions.

Concept

Post-training quantization takes a trained model’s weights — normally stored as 32-bit or 64-bit floats — and represents them with fewer bits (commonly INT8, increasingly INT4). The immediate, guaranteed win is memory: an INT8 tensor takes a quarter of the space of its FP32 equivalent, an INT4 tensor an eighth. The less guaranteed win is compute/energy, which depends on whether the hardware actually executes the low-precision operations natively or just stores the weights compactly and upconverts them back to float before doing math.

This gives us two separable, falsifiable questions rather than one vague claim:

  • H1 (accuracy): INT8 post-training weight quantization of a small classifier will degrade test accuracy by less than 1 percentage point relative to the full-precision baseline; INT4 will show a larger, measurable drop.
  • H2 (speed/energy proxy): Fake-quantizing weights (quantize, then dequantize back to float before running inference) will not produce a meaningful wall-clock speedup, because the arithmetic underneath is still floating-point — the compute savings only materialize with genuine low-precision execution kernels, not from smaller storage alone.

H2 is the part that’s easy to gloss over in “quantize your model to save energy” advice. It’s also directly testable on a laptop CPU without any specialized hardware, which is exactly what the experiment below does.

Experiment

The setup is intentionally small and fully reproducible: scikit-learn‘s built-in load_digits dataset (1,797 handwritten-digit images, 10 classes, 64 features — no download required) and a single-hidden-layer MLPClassifier (64 hidden units). For each of 10 random seeds, the model is trained fresh, then its weight matrices (coefs_) and biases (intercepts_) are quantized with symmetric per-tensor fake quantization at INT8 and INT4, dequantized back to float, and re-evaluated on the same held-out test split.

The core quantize/dequantize step:

def quantize_dequantize(arr, bits):
    qmax = 2 ** (bits - 1) - 1          # 127 for int8, 7 for int4
    scale = np.max(np.abs(arr)) / qmax if np.max(np.abs(arr)) > 0 else 1.0
    q = np.clip(np.round(arr / scale), -qmax - 1, qmax)
    deq = (q * scale).astype(np.float32)
    mse = float(np.mean((arr - deq) ** 2))
    return deq, mse

Each seed’s quantized model is a deep copy of the trained classifier with its coefs_/intercepts_ attributes swapped for the dequantized versions, then run through predict() directly — sklearn’s forward pass reads those attributes at inference time, so no custom forward-pass code is needed. Inference is timed by averaging 20 repeated predict() calls per condition. The whole script (training + quantizing + evaluating 10 seeds x 3 precisions) runs in about 12 seconds on a single CPU core.

One useful surprise surfaced immediately: sklearn stores MLPClassifier weights as float64 by default, not float32. That’s a good reminder that “the baseline” isn’t always what a library’s name or your assumptions suggest — it’s worth checking dtypes explicitly rather than trusting the FP32 label most quantization writeups reach for by habit.

Results

Mean test accuracy across 10 seeds (train/test = 70/30, stratified):

Precision Mean accuracy Std dev Δ vs FP64 Weight MSE vs FP64 Weight memory
FLOAT64 (baseline) 0.9778 0.0048 — — 38,480 bytes
INT8 (fake-quantized) 0.9778 0.0048 +0.00 pp 4.77 × 10-6 4,810 bytes (8.0× smaller)
INT4 (fake-quantized) 0.9722 0.0057 −0.56 pp 1.55 × 10-3 2,405 bytes (16.0× smaller)

INT8 accuracy was identical to the float64 baseline to four decimal places across all 10 seeds — not approximately equal, exactly equal on this dataset/model combination, which makes sense given the weight range for this small MLP maps cleanly onto 8-bit resolution. INT4 lost 0.56 percentage points on average, with per-seed drops ranging from 0 to about 1.7 points — small, but real and consistent with the ~325x larger quantization error (MSE) at 4 bits.

The wall-clock numbers are the more interesting (negative) result. Mean inference time per predict() call over the 540-sample test set:

Precision Mean time / call
FLOAT64 0.501 ms
INT8 (dequantized) 0.474 ms
INT4 (dequantized) 0.448 ms

These differences (5–11%) are within normal run-to-run noise for a sub-millisecond CPU operation timed with perf_counter — not a systematic speedup. That’s expected: the “quantized” weights here are dequantized back to float32 before the matmul runs, so the CPU is doing the exact same floating-point arithmetic regardless of precision label. The only thing that changed was storage size, which a wall-clock inference timer can’t see at all.

Explanation

The accuracy half of these results lines up with the broad consensus: INT8 post-training quantization is close to a free lunch for accuracy on small-to-medium models, and INT4 starts to cost you something measurable, which is exactly the tradeoff curve described across the current LLM quantization literature comparing INT4/INT8/FP8/AWQ/GPTQ schemes.

The speed/energy half is the part worth sitting with. A 2026 study on LLM inference energy (“Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use”) found that quantization’s energy benefit depends heavily on which phase of inference you’re in: during the compute-bound prefill phase on larger models (LLaMA 8B, Qwen 14B), dropping to lower precision cut GPU energy by up to 4x. But during the memory-bound decode phase — which dominates real chatbot-style usage — INT8 and INT4 actually increased energy use by 2–3x relative to float32, because on-the-fly dequantization added fragmented memory operations that outweighed any bandwidth savings, and the GPU sat idle between kernel launches drawing power regardless of numeric precision. Smaller models (0.5B–1.5B parameters) saw little benefit or even slight regressions from quantization overhead, since there wasn’t enough compute work to amortize it.

A related study on sustainable LLM inference for edge devices reaches a compatible conclusion from a different angle: energy efficiency, output accuracy, and latency don’t move together predictably across quantization schemes — the “right” precision is workload- and hardware-dependent, not a universal setting to flip.

Our own toy experiment is a small, clean illustration of the mechanism behind that nuance: quantization only pays off computationally when the hardware runs genuinely low-precision kernels end-to-end. Simulating quantization by shrinking-then-restoring precision in software — which is effectively what happens if you quantize weights for storage but the runtime dequantizes to float before computing, or if you’re on hardware/software that doesn’t have an optimized low-precision matmul path — gets you the memory win with none of the compute win. That’s a smaller-scale echo of exactly what the decode-phase finding above describes at LLM scale: memory savings are real and easy; compute/energy savings require the full hardware-software stack to cooperate, and self-reported “efficiency” claims that only cite decreased model size are answering a different question than the one that determines electricity use. It’s the same caution we applied when testing how much real data actually stops model collapse rather than trusting the plausible-sounding rule of thumb directly.

None of this means quantization is a bad sustainability lever — the IEA’s own analysis notes that per-task AI energy efficiency is improving “at a rate unprecedented in energy history,” and quantization is part of that broader efficiency push. It means the specific claim “we quantized it, so it uses less energy” needs the qualifier “on hardware with native low-precision execution, in the phase of inference that’s compute-bound” attached, or it’s a memory claim wearing an energy claim’s clothes.

Resources

Related Reading

Leave a Reply

Your email address will not be published. Required fields are marked *