LLM Quantization: Separate Weight Storage, Runtime Memory, and Answer Quality

Reviewed October 6, 2026. Arithmetic examples use Python 3.12.14. The seven-billion-parameter model is a hypothetical size calculation, not a downloaded model. No GPU, latency, or language-model quality benchmark was performed.

A model is described as four-bit, so someone estimates its memory requirement as exactly one quarter of a sixteen-bit version and assumes it must run four times faster. The nominal storage arithmetic is useful, but both conclusions skip important parts of the system.

Quantization represents selected values with a lower-precision scheme. A deployment also has scales, metadata, unquantized components, temporary working memory, cached attention state, and implementation-specific storage. Runtime performance depends on kernels, hardware, data movement, batch size, and the workload.

Practical rule: calculate nominal weight payload as a baseline, then measure actual loaded and peak runtime memory under a stated generation workload. Evaluate output quality separately on representative tasks. A bit-width label alone does not establish speed, total memory, or suitability.

What the bit-width arithmetic actually says

For N parameters represented uniformly with b bits each, the nominal payload is N times b divided by eight bytes. This simplified calculation ignores packing overhead, scales, zero points, architecture-specific treatment, and every non-weight allocation.

Decimal gigabytes divide by one billion bytes. Binary gibibytes divide by 2 to the power of thirty. A number can look different across tools because those units are different, even before considering actual runtime overhead.

parameters = 7_000_000_000
rows = []
for bits in [16, 8, 4]:
    payload_bytes = parameters * bits // 8
    rows.append({"bits": bits, "bytes": payload_bytes,
                 "GB": payload_bytes / 1_000_000_000,
                 "GiB": payload_bytes / (2 ** 30)})
assert [row["GB"] for row in rows] == [14.0, 7.0, 3.5]
assert rows[0]["bytes"] == 14_000_000_000
print(rows)

The four-bit payload is one quarter of this uniform sixteen-bit payload. It is not a promise that a real model fits in exactly 3.5 GB of total memory. Training, inference, and checkpoint storage are also different workloads; do not reuse one figure for all three.

A scale connects integer codes to approximate values

A simple symmetric uniform quantizer maps real values to bounded integer codes and uses a scale to reconstruct approximations. More sophisticated formats may use different codebooks, group-specific scales, outlier handling, or mixed precision. A toy uniform quantizer explains the boundary without pretending to implement those production schemes.

def quantize(values, bits):
    if type(bits) is not int or not 2 <= bits <= 8:
        raise ValueError("bits must be an integer from 2 to 8")
    if not values:
        raise ValueError("nonempty values required")
    maximum = max(abs(value) for value in values)
    limit = 2 ** (bits - 1) - 1
    scale = maximum / limit if maximum else 1.0
    codes = [max(-limit, min(limit, round(value / scale))) for value in values]
    reconstructed = [code * scale for code in codes]
    mse = sum((original - restored) ** 2 for original, restored
              in zip(values, reconstructed)) / len(values)
    return codes, reconstructed, scale, mse

values = [-1.0, -0.3, 0.2, 0.9]
codes, restored, scale, mse = quantize(values, 4)
assert codes == [-7, -2, 1, 6]
assert abs(scale - 1 / 7) < 1e-12
assert mse > 0
assert all(-7 <= code <= 7 for code in codes)
zero_codes, zero_values, _, zero_error = quantize([0.0, 0.0], 4)
assert zero_codes == [0, 0] and zero_error == 0
print({"codes": codes, "restored": restored, "scale": scale, "MSE": mse})

The input domain is a small sequence of finite real values. The helper validates its bit range and nonempty input, not arbitrary external documents or nonfinite values. A deployment implementation needs a fully defined type and resource contract.

This scheme uses codes from negative seven through seven for four bits, leaving one of the sixteen possible bit patterns unused. That is a deliberate symmetric toy choice. Do not infer that every real four-bit format uses this codebook or stores Python integers as packed four-bit values.

The example measures reconstruction error for four numbers, not downstream language-model quality. A small weight error can interact with the rest of a network in ways this table does not establish. The correct next step is task-level evaluation, not claiming accuracy from the quantizer's mean squared error alone.

Storage precision and compute precision can differ

A model can store some weights in a quantized representation while performing parts of computation in another dtype. Kernels may dequantize groups as needed or use specialized arithmetic. Activations, normalization components, and other tensors need their own treatment.

The Transformers bitsandbytes guide describes particular quantized layers and configuration options. Those are implementation capabilities, not a universal specification for every artifact labeled four-bit. Verify the chosen backend, model architecture, supported hardware, and exact package versions.

Dequantizing cannot generally recover information discarded by quantization. Restoring a higher-precision container does not recreate the original unrounded weights. A conversion back to a larger dtype should therefore not be described as reversing every quality effect.

Compatibility and performance can change with backend releases. This article does not prescribe a current hardware purchase or install recipe. It explains which evidence a concrete deployment must collect before drawing capacity or quality conclusions.

Runtime memory includes more than weights

Autoregressive generation can retain key and value tensors from attention computation to avoid recomputing preceding context. This KV cache can become a substantial allocation for longer contexts, larger batches, or architecture-specific cache behavior.

Weight quantization does not automatically quantize the KV cache. Cache strategies such as offloading or quantization are separate choices with their own memory, latency, and quality implications. The Transformers cache guide discusses those tradeoffs, including cases where a memory-saving cache strategy can worsen latency.

The model's parameter count alone does not define all these allocations. Record the actual context length, prompt length, generated length, batch size, cache policy, and dtype configuration when measuring memory. A short single-user example cannot establish the peak requirement for a long-context multi-user workload.

Benchmark the same workload before comparing speed

A lower nominal bit width can reduce data transfer while adding conversion or kernel overhead. Hardware support can make one format efficient and another disappointing. Compare configurations under the same inputs, output-length policy, batch size, and warm-up method.

Measure time to first token and sustained token generation separately if they matter to the application. Also record peak memory, throughput, and failures such as out-of-memory behavior. No single latency number captures every interactive or batched use case.

A reproducible comparison identifies the exact original model, quantized artifact or configuration, backend versions, hardware, and generation settings. Without those details, a broad claim such as “four-bit is faster” cannot be responsibly applied to another deployment.

Evaluate quality at the task boundary

Use a fixed held-out set covering the application's actual requirements: factual answers, unsupported-question handling, code correctness, structured output, or other relevant tasks. Compare against the baseline with the same prompting and scoring method. Inspect failed cases, not only an aggregate score.

Include the tasks that are difficult for your application. A short generic demonstration can miss errors in numeric reasoning, long context, rare terminology, or exact syntax. If model output is stochastic, document the sampling policy and how variation was handled.

Quantization is a tradeoff to evaluate under a concrete workload. The payload table and toy reconstruction establish useful concepts; they do not claim that a particular model retains its quality, fits your hardware, or achieves a measured speed gain.

Sources and reproducibility

Primary references: Transformers bitsandbytes quantization and KV cache strategies. Both arithmetic examples run locally with the Python standard library. The table uses decimal GB and binary GiB explicitly; no model weights were downloaded and no hardware benchmark was run.