AI Talk - GGUF and Quants for Non-AI Engineers

Navigate the files tab of almost any model on Hugging Face and you find a list like this:

model-Q8_0.gguf
model-Q6_K.gguf
model-Q5_K_M.gguf
model-Q4_K_M.gguf
model-Q4_0.gguf
model-IQ3_XXS.gguf
model-IQ2_XS.gguf

As someone who just got into running local AI recently, I was admittedly...confused. I've been reading up and figured I'd put it all together here if anyone cares to read it. This post is my attempt to put it in one place.

Size in the hypothetical model list above goes from "fitting in laptop RAM" to "get a bigger system". By the end you'll know what GGUF is, what each part of a quant name means, how to calculate whether a model will fit your hardware, and how much quality you lose along the way.

A model is mostly a giant array of floats

Through the marketing jargon, a trained model is a set of named tensors: multi-dimensional arrays of numbers, called weights. The "8B" in a model name indicates about 8 billion of them. Inference is largely a series of matrix multiplications against those arrays, repeated for every token generated.

Models are generally trained and released in 16-bit floating point (FP16 or BF16), which is 2 bytes per weight. Do the math:

8,000,000,000 weights × 2 bytes = 16 GB

That's 16 GB to simply load the 8B model into memory, before you've produced any tokens. A 70B model is 140 GB. None of us probably have that much RAM at our disposal, and even if we did, there's a second problem. Generating a token requires processing most of the weights of the model through the processor. For consumer hardware, memory bandwidth is the bottleneck, not compute. Fewer bytes per weight means more tokens per second.

And that is why there are two reasons to reduce the size of the weights: to make the model fit at all, and to optimize its performance if it does fit. Reducing the size is quantization.

GGUF is the format

GGUF is just a file format.

It comes from the llama.cpp project. It is what llama.cpp, Ollama, LM Studio, and other inference programs read. A GGUF file is a unified description of the model that contains:

• Metadata: architecture, number of layers, context length, the tokenizer's vocabulary and the chat template
• Tensors: all weight arrays

Think of a .NET assembly and its manifest. Everything required to load and run the model in one file, and the file itself tells you what's in there.

What's important is that every tensor comes with its own type. A tensor may be stored at 8 bits, and its neighbor may be stored at 4. GGUF doesn't care. The per-tensor flexibility is what allows the mixed quants the rest of this rambling is about.

A GGUF file may contain full FP16 weights. The Q4_K_M in the filename represents how the weights in that GGUF are quantized. The .gguf file itself just describes the packaging.

Quantization

If you're here, I assume you've used System.Decimal. Under the hood it's a big integer with a scale factor. Quantization works similarly, with two differences. The big integer is small (4 bits, let's say), and there's one scale factor applied to the whole block of weights.

Here's the simplest quantization method, close to what llama.cpp calls Q4_0:

1. Pick 32 consecutive weights.
2. Compute the maximal absolute value of the elements in the block and determine a scale that brings that element to the boundary of a 4-bit range (-8 to 7).
3. Divide each element by the scale and round the result to the nearest integer.
4. Store 32 small integers and 1 scale factor.

To restore a weight, multiply it: weight ≈ q × scale. In C#, something like this:

static (Half Scale, sbyte[] Q) QuantizeBlock(ReadOnlySpan<float> w)
{
    float maxAbs = 0;
    foreach (var x in w) maxAbs = MathF.Max(maxAbs, MathF.Abs(x));

    float scale = maxAbs / 7f;
    var q = new sbyte[w.Length];
    for (int i = 0; i < w.Length; i++)
        q[i] = (sbyte)Math.Clamp(MathF.Round(scale == 0 ? 0 : w[i] / scale), -8, 7);

    return ((Half)scale, q);
}

The storage formula: 32 weights × 4 bits = 16 bytes, plus 2-byte FP16 scale = 18 bytes per block. That's 4.5 bits per weight, compared to 16. An 8B model goes from 16 GB to about 4.5 GB.

Why that doesn't break the model

You're losing a lot of precision here, but why does the model work? Two reasons.

• Weights within a block usually have similar magnitude. A per-block scale works well with the local range, so most of the rounding error is small.
• Neural networks are robust to perturbations. Every output is a weighted sum of thousands of products, so small and unbiased errors cancel each other.

The weakness is outliers. One big weight in a block will require a big scale, leaving the remaining 31 weights with very few levels to use. Most of the improvements in the next section come from handling outliers and the tensors most susceptible to them.

Decoding the names

A quant name consists of three parts: the bit width, the quantization scheme family, and sometimes mix size.

The number is the nominal bits per weight. There are some scales and overhead, which is why Q4 ends up at 4.5-4.9 bits.

_0 and _1 are original, simple schemes. Q4_0 is the algorithm described above. Q4_1 adds an additional per-block offset (a minimum) to the scale and can represent any range, not necessarily centered on 0. They work, but have been superseded by newer ones.

K means k-quants, the second generation. Blocks are aggregated into super-blocks of 256 weights. Each 32-weight sub-block gets its own scale, but these scales are quantized themselves with 1 FP16 scale per super-block. You get per-block scales for about the same storage. A K quant is generally better choice than an equivalent legacy one at any size.

S, M, L (small, medium, large) describe the mix. All tensors are not equally affected by rounding. Q4_K_M stores most tensors at 4 bits, but some of them, such as attention and feed-forward parts, are stored at 6 bits. _S contains fewer higher-precision storage, _L contains more. That's where the per-tensor types in GGUF shine.

IQ means i-quants, the most recent family. They use more sophisticated encoding schemes, matching the block of weights to the codebook of likely patterns, and not rounding individual elements. Also they require an importance matrix (imatrix). It's constructed by running calibration text through the full-precision model and recording the weights influencing its output. This way quantization allocates precision where it counts. That's why IQ2 and IQ3 quants are usable at all. An imatrix can also be used to improve K quants, and the model pages usually mention when it's done.

Approximate bits-per-weight figures. Mixed quants may vary a little depending on the model architecture.

Does it fit? The memory calculation

The weights are easy. Multiply the number of parameters by bits per weight and divide by 8:

weights (GB) ≈ parameters (billions) × bits per weight ÷ 8

That will give you these approximate sizes of GGUF files:

These numbers are approximate and slightly vary depending on the architecture.

The KV cache is another thing

The file size is not your memory requirement. While performing inference, the model keeps attention keys and values for every token in context. This is the KV cache, and it grows linearly with the context size:

KV bytes = 2 × number of layers × number of KV heads × head dimension × bytes per value × number of tokens

For a Llama 3.1 8B-class model at FP16 that is 128 KB per token. 32K context consumes 4 GB on top of weights, and 128K — 16 GB, which is 3x more than Q4 weights. 70B-class model takes about 320 KB per token, or 10 GB at 32K.

Practical implications:

• Account for the cache, not just for the file. 4.9 GB 8B-class model "fitting" in 8 GB VRAM will not fit once you add a long context.
• The cache can be quantized. llama.cpp can store it at 8 bits, halving its size with negligible quality loss.

Remember that you should leave some space for the operating system and the runtime too. Filling all memory will either make the program page swap, or fail to load it.

How much quality do you lose

There are two figures you'll see in the quantization comparisons.

• Perplexity measures how surprised the model is by a reference text. Lower is better. If a quant raises perplexity only slightly over FP16, it behaves like the original.
• KL divergence calculates the difference between the quantized model and the original directly, token by token. It's a more honest figure of "how different is this from the original", and is calculated by llama.cpp.

Neither of these measures how the model performs on your task. They are useful for comparing quants of the same model, but useless for comparing different models.

The rules of thumb that stand the test of time:

• Q8_0 is effectively lossless. Use it as the reference point, not the default.
• Q6_K and Q5_K_M are difficult to distinguish from Q8 in general usage.
• Q4_K_M is the sweet spot for most people. Quality loss is measurable, but hardly noticeable.
• Below about 4 bits, the losses increase significantly, and become more pronounced for small models.
• Larger models cope better with aggressive quantizations. A 70B model at Q3 loses less of its full-precision quality than an 8B model at Q3 does, because there is more redundancy to absorb the errors.
• For the same memory requirements, a larger model at Q4 will usually perform better than a smaller model at Q8. When you need to choose between them, start with a larger model.
• Coding, math, and long-context work are most sensitive to quality loss. These tasks require exact tokens, so if you're using a model for coding, go one quant higher.

This last point is the most important one if, like me, you want a local model for coding.

The cheat sheet

When you're looking at that list of files:

1. Calculate the memory requirements first. Weights ≈ parameters × bits ÷ 8 + KV cache at the context length you need + overhead.
2. Start with Q4_K_M. Go to Q5_K_M or Q6 if you have the memory, especially for coding.
3. Prefer a larger model with lower quant over a smaller model with higher quant.
4. Choose K or IQ quantization schemes over legacy _0/_1 ones, and prefer imatrix builds if available.
5. Think of sub-4-bit quantizations as a way to fit a model that wouldn't fit otherwise, not as default.

That should cover the naming. Now you have to learn how they translate to actual performance and quality on real hardware. In my next post I'll benchmark local models at different quantization levels on my personal machine...for science!