This INT8 quantization energy savings check starts from a common claim: that shrinking a model’s numerical precision automatically cuts energy use too, not just memory.

Quantization is often presented as one of the easiest ways to make machine-learning models smaller and more efficient.

The basic argument sounds straightforward:

FP32 uses 32 bits per weight. INT8 uses 8. Therefore INT8 should use roughly one-quarter of the weight storage and consume less energy.

The first part is straightforward.

The second is not.

In this experiment, I wanted to separate three claims that are often bundled together:

Does reducing numerical precision preserve model accuracy?
How much smaller could the model’s weights be if stored in lower precision?
Does quantization automatically reduce runtime energy consumption?

The results show why those should be treated as three different questions.

The experiment preserved accuracy surprisingly well after INT8 — and even INT4 — quantization.

But the execution path used here dequantizes the weights back to floating point before inference.

So this experiment does not benchmark a native INT8 or INT4 inference kernel.

That distinction turns out to be central to interpreting the results.

The Question

The experiment asks:

How much accuracy do we lose when model weights are quantized from FP32 to INT8 or INT4, and does that numerical compression automatically translate into lower runtime energy consumption?

The expectation might be:

lower precision → smaller model → less computation → less energy

But each arrow in that chain requires assumptions.

Quantization can reduce how many bits are required to represent a weight.

Whether those lower-precision values remain compressed during computation depends on the execution stack.

And whether that produces an energy saving depends on hardware, kernels, memory movement and workload.

So we should measure these separately.

1. Experimental Setup

The experiment uses a deliberately small classification problem.

The goal is not to establish universal quantization performance.

It is to create a controlled environment where we can compare:

an FP32 baseline,
an INT8-quantized representation,
an INT4-quantized representation.

The same trained model is used as the starting point for all three conditions.

That is important.

We are testing the effect of changing numerical precision rather than retraining three unrelated models.

2. What Quantization Does

A neural-network weight stored as FP32 uses 32 bits.

For example:

0.173829

Quantization maps floating-point values onto a smaller discrete numerical representation.

With 8-bit signed integers, values can be represented using integer levels such as:

-128 ... 127

A simplified symmetric quantization process can be written as:

q = round(w / s)

where:

w is the original floating-point weight,
s is the scale factor,
q is the quantized integer.

To approximately reconstruct the weight:

ŵ = q × s

The reconstructed value is usually not exactly equal to the original value.

That difference is the quantization error.

The central question is whether that error is large enough to affect the model’s predictions.

3. Why INT8 Should Reduce Weight Storage

Suppose a model contains:

1,000,000 parameters

Ignoring metadata and other overhead:

FP32 representation requires approximately:

1,000,000 × 32 bits = 32,000,000 bits

INT8 requires:

1,000,000 × 8 bits = 8,000,000 bits

So the theoretical packed INT8 weight representation is approximately:

25% of the FP32 weight storage.

INT4 could theoretically reduce packed weight storage further:

1,000,000 × 4 bits = 4,000,000 bits

or approximately:

12.5% of FP32 weight storage.

But this is a statement about representation.

It does not automatically tell us how much memory the complete application consumes during inference.

And it tells us even less about energy.

4. The Important Distinction: Quantized Storage vs Quantized Execution

This experiment uses simulated—or fake—post-training quantization.

Weights are:

converted to a lower-precision quantized representation,
reconstructed/dequantized,
used in floating-point computation during inference.

Conceptually:

FP32 weights

↓

quantize

↓

INT8 or INT4 representation

↓

dequantize

↓

floating-point weights used for inference

This allows us to test an important question:

How much numerical error can the model tolerate?

But it does not reproduce the execution path of a system using native INT8 matrix-multiplication kernels throughout inference.

Therefore:

The INT8 and INT4 storage figures in this experiment represent theoretical packed weight storage, not the actual runtime memory footprint of the dequantized inference model.

That distinction is essential.

5. Measuring Accuracy

The first measurement is straightforward.

Evaluate the original FP32 model.

Then quantize its weights to INT8, dequantize them, and evaluate again.

Repeat for INT4.

The comparison becomes:

Precision	Test Accuracy
FP32	Measured baseline
INT8	Measured result
INT4	Measured result

In the experiment, the important observation was that reducing numerical precision did not automatically destroy predictive performance.

INT8 retained accuracy very well.

Even INT4 remained surprisingly competitive on this particular task.

That is interesting because neural networks can often tolerate a meaningful amount of numerical perturbation without changing their predictions.

But this result must remain scoped correctly:

The experiment shows robustness to weight quantization for this model and dataset. It does not establish that INT8 or INT4 will preserve accuracy for every neural network.

Different architectures, datasets and quantization methods can behave differently.

6. Measuring Weight Storage Correctly

This is where terminology matters.

It would be misleading to label the INT8 and INT4 values simply as:

“Weight memory during inference.”

That is not what this experiment measures.

Because the quantized weights are dequantized back into floating-point values before inference, the smaller values describe what the weights could occupy if stored in packed low-precision form.

The correct comparison is therefore:

Precision	Bits per weight	Theoretical packed weight storage relative to FP32
FP32	32	100%
INT8	8	~25%
INT4	4	~12.5%

These are idealised weight-storage ratios.

Real model files can contain additional information such as:

scale factors,
zero points,
metadata,
biases,
unquantized layers,
alignment/padding,
framework-specific overhead.

Actual storage therefore may not match these theoretical ratios exactly.

And runtime memory can differ substantially from packed model storage.

7. What About Energy?

This was the more interesting part of the experiment.

A common intuition is:

Smaller numbers require less computation, therefore quantization should automatically reduce energy use.

But that conclusion only follows if the hardware and software actually execute the model using lower-precision operations.

In this experiment, they do not.

The quantized values are reconstructed into floating-point values before inference.

So the processor still performs floating-point computation.

The experiment therefore did not show a clear energy-saving benefit merely from applying fake quantization.

That is not a failure of quantization.

It is a reminder that:

Numerical compression and computational acceleration are not the same thing.

8. Why Native INT8 Can Be Different

Modern processors and accelerators can contain hardware specifically designed for low-precision arithmetic.

When a framework uses an appropriate native INT8 execution path, several things may change:

less data may need to move through memory,
more values may fit in caches,
specialised vector instructions may process more values per operation,
arithmetic throughput may increase,
memory bandwidth requirements may fall.

Under those conditions, quantization can improve inference efficiency.

But those benefits depend on the complete stack:

model

↓

quantization method

↓

framework

↓

kernel implementation

↓

hardware

A model merely containing quantized values is not enough.

9. Why Energy Measurement Is Hard

Energy is also more difficult to measure than accuracy.

Accuracy can be calculated directly from predictions and labels.

Energy consumption depends on many environmental factors:

CPU or GPU utilisation,
clock frequency,
memory activity,
cooling,
background processes,
batch size,
inference duration,
operating-system scheduling,
measurement interval,
power-management behaviour,
hardware-specific acceleration.

For short experiments, measurement noise can be surprisingly large.

This means a small apparent difference in power consumption should not automatically be interpreted as a real energy improvement.

Repeated measurements and appropriate hardware-level instrumentation are preferable.

10. Power Is Not Energy

There is another distinction worth making.

Power is the rate at which energy is being consumed.

It is commonly measured in watts.

Energy is accumulated consumption over time.

Conceptually:

Energy = Power × Time

A model could draw slightly more instantaneous power but finish much faster and therefore consume less total energy.

Or it could draw slightly less power but run considerably longer and consume more energy overall.

So when comparing efficiency, runtime and power need to be considered together.

11. What the Experiment Actually Shows

The safest interpretation of the experiment is:

Finding 1 — Accuracy can survive aggressive quantization

For this particular model and dataset, INT8 quantization introduced little or no meaningful accuracy degradation.

Even INT4 remained surprisingly robust.

That suggests the trained network did not require full FP32 precision for every weight to maintain its predictions.

Finding 2 — Packed low-precision weights can be much smaller

At the representation level:

INT8 ≈ 25% of FP32 weight storage

and:

INT4 ≈ 12.5% of FP32 weight storage

before accounting for quantization metadata and implementation overhead.

Finding 3 — Fake quantization does not automatically produce runtime energy savings

Because this experiment dequantizes the weights before inference, computation still takes place through a floating-point execution path.

The experiment therefore provides no basis for claiming that native INT8 arithmetic itself failed to save energy.

That was not what was tested.

Finding 4 — Efficiency depends on implementation

To establish the real energy benefit of INT8, the next experiment needs to use a framework and device capable of native low-precision execution.

12. What This Experiment Does NOT Prove

The experiment does not prove:

INT8 never saves energy.

It does not prove:

INT4 is always as accurate as FP32.

It does not prove:

Quantization reduces complete application memory by exactly 75% or 87.5%.

And it does not prove:

Quantization is ineffective.

What it demonstrates is narrower:

Reducing numerical precision can preserve predictive accuracy, but the theoretical storage advantage of low-precision weights does not automatically become a runtime energy advantage when inference still executes in floating point.

That is a much more useful conclusion.

13. Why This Matters in Practice

Machine-learning optimisation often gets reduced to simple rules:

quantization = smaller

pruning = faster

smaller model = lower energy

lower precision = cheaper inference

Each can be true.

None is automatically true in every execution environment.

For production systems, optimisation has to be evaluated end to end.

If the goal is deployment on an edge device, we should measure:

actual model-file size,
peak runtime memory,
inference latency,
throughput,
energy per inference,
hardware compatibility,
accuracy degradation.

A theoretical reduction in bits per parameter is useful.

But it is only one part of the deployment story.

14. A Better Follow-Up Experiment

The natural next step is to compare:

FP32 inference

against:

Native INT8 inference

on hardware that genuinely supports INT8 operations.

The experiment should measure:

Metric	FP32	Native INT8
Accuracy	Measured	Measured
Model storage	Measured	Measured
Peak runtime memory	Measured	Measured
Mean inference latency	Measured	Measured
Throughput	Measured	Measured
Average power	Measured	Measured
Energy per inference	Measured	Measured

Run each condition repeatedly.

Then report variability rather than relying on a single measurement.

That would let us answer the stronger question:

When low-precision computation is actually executed natively, how much efficiency do we gain per unit of accuracy lost?

That is a much more meaningful deployment benchmark.

15. The Bigger Lesson

The most interesting result from this experiment is not that quantization “worked” or “failed.”

It is that several ideas that sound equivalent are actually different:

Numerical precision

is not the same as

model storage

which is not the same as

runtime memory

which is not the same as

execution precision

which is not the same as

latency

which is not the same as

energy consumption.

Quantization can affect all of them.

But measuring one does not automatically tell us what happened to the others.

That distinction matters whenever machine-learning efficiency claims are evaluated.

So the conclusion from this experiment is deliberately modest:

INT8 quantization can preserve accuracy and substantially reduce the theoretical packed representation of model weights. But energy savings are an implementation-level result, not an automatic consequence of storing weights with fewer bits.

Sometimes the most useful experiment is the one that tells us the shortcut in our mental model was wrong.

References and Further Reading

[1] Jacob, B., Kligys, S., Chen, B., et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. Proceedings of CVPR 2018.
This paper describes integer-only inference techniques and the practical relationship between quantized representations and efficient execution.
https://arxiv.org/abs/1712.05877

[2] PyTorch Documentation. Quantization.
PyTorch documentation covering quantization approaches and low-precision model execution.
https://docs.pytorch.org/docs/stable/quantization.html

[3] Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021). A Survey of Quantization Methods for Efficient Neural Network Inference.
A broad review of neural-network quantization methods, trade-offs and deployment considerations.
https://arxiv.org/abs/2103.13630

[4] Horowitz, M. (2014). 1.1 Computing’s Energy Problem (and What We Can Do About It). IEEE International Solid-State Circuits Conference.
A widely cited discussion of the energy costs of computation and data movement in modern hardware.
https://doi.org/10.1109/ISSCC.2014.6757323

Reproducibility and Interpretation Note

This article evaluates simulated/fake quantization, in which weights are quantized and subsequently dequantized for floating-point inference.

The INT8 and INT4 storage reductions discussed above therefore represent theoretical packed weight representations, not measured runtime-memory reductions during the dequantized inference process.

Likewise, the absence of a clear energy saving in this experiment should not be interpreted as evidence that native INT8 execution cannot reduce energy consumption.

Demonstrating that requires a separate benchmark using hardware and kernels that execute the relevant operations directly at low precision.

INT8 Quantization Keeps Accuracy Intact, But Doesn’t Automatically Save Energy

The Question

1. Experimental Setup

2. What Quantization Does

3. Why INT8 Should Reduce Weight Storage

4. The Important Distinction: Quantized Storage vs Quantized Execution

5. Measuring Accuracy

6. Measuring Weight Storage Correctly

7. What About Energy?

8. Why Native INT8 Can Be Different

9. Why Energy Measurement Is Hard

10. Power Is Not Energy

11. What the Experiment Actually Shows

Finding 1 — Accuracy can survive aggressive quantization

Finding 2 — Packed low-precision weights can be much smaller

Finding 3 — Fake quantization does not automatically produce runtime energy savings

Finding 4 — Efficiency depends on implementation

12. What This Experiment Does NOT Prove

13. Why This Matters in Practice

14. A Better Follow-Up Experiment

FP32 inference

Native INT8 inference

15. The Bigger Lesson

References and Further Reading

Reproducibility and Interpretation Note

Related Reading

Leave a Reply Cancel reply