GGUF 量化后缀解密：Q8_0、Q6_K、Q4_K_M 到底怎么选

你在 Hugging Face 上找个模型，一排文件砸下来：Q8_0、Q6_K、Q5_K_M、Q4_K_M、Q4_K_S、Q3_K_M、IQ2_XXS，后缀还有带 .imx 的。

同一个 7B 模型，最小的文件比最大的小一半还多。选大的？显存不够。选小的？怕变傻子。

这篇文章把后缀拆开讲清楚。看完你就能闭着眼睛选。

先花 30 秒搞懂量化是什么

大模型的权重默认是 fp16，也就是每个数字用 16 位（2 字节）存。7B 模型 = 700 亿个数字 ≈ 14GB。

量化就是：用更少的位存这些数字。4 位、5 位、6 位、8 位……位越少文件越小，但精度损失越大。

关键技巧是分块量化：不是每个权重单独找最优值，而是把权重切成一小块一小块，每块共享一个"缩放系数"（scale），把误差摊薄到整块上。量化方法之间的区别，主要就是"块多大、每块存几个辅助参数、怎么解码"。

衡量体积的单位叫 bpw（bits per weight）——每个权重平均占多少位。8.5 bpw 就是每个权重 8.5 位，2.06 bpw 就是 2.06 位。所有后缀的本质，都是告诉你这个数字。

后缀解码：先拆 Q4_K_M

Q4_K_M 这个名字其实是四段：

• Q = Quantized（量化）
• 4 = 每权重 4 位（基础精度）
• K = 方法族。K = K-quants；没有 K 的是老方法（如 Q4_0）；I 开头的是 I-quants（如 IQ2_XXS）
• _M = 档位，S/M/L（Small/Medium/Large）。M 是中间档，兼顾质量和体积

搞清楚这个命名规则，剩下的全是细节。

第一代：传统量化（Q4_0、Q4_1、Q8_0）

这是最老、最简单的一批，2023 年中期之前的主流。

结构：32 个权重一组（block），每组共享常数。

• Q4_0：32 个权重 × 4 位 + 1 个 fp16 scale = 4.5 bpw
• Q4_1：在 Q4_0 基础上多存一个 min（最小值/偏移）常数 = 5.0 bpw。多存这个数，是为了处理分布不居中的权重
• Q8_0：32 × 8 位 + 1 个 scale = 8.5 bpw。接近无损

特点：实现极其简单。 解压只需位移、AND、乘法这些基础指令，不需要查表，不需要复杂计算。所以在某些老显卡、老 CPU 上，传统量化反而可能更快。

缺点：简单也意味着笨。 所有权重一视同仁，没有重点保护。Q4_0 的误差在同体积下明显大于后来的方法。今天它基本只出现在 Q8_0（因为 Q8_0 质量太好了，接近无损）和兼容性场景里。

第二代：K-quants（Q6_K、Q5_K_M、Q4_K_M）——现在的主流

2023 年 8 月，llama.cpp 的 PR #1684 引入了 K-quants，核心思路是"精度要花在刀刃上"。它做了两件事：

第一件：块内混合精度。 block 从 32 个权重扩大到 256 个，内部再分成 16 个子块（每块 16 个权重），子块用低精度量化，同时整个 block 配一个高精度的"超级缩放系数"（super-scale）来修正。效果是：离群的大权重不会把整块的误差带崩，重要权重的精度被保住了。

第二件：不同张量，不同待遇。 这就是 _S/_M/_L 的含义：

• Q4_K_S：所有张量统一用 Q4_K 块，4.5 bpw
• Q4_K_M：大部分层用 Q4_K，但 attention 的 q/k/v 和 feed-forward 的输出层用更高精度的 Q6_K，4.85 bpw
• Q5_K_M：同理，5.5 bpw 左右
• Q6_K：所有张量用 6 位块，6.56 bpw。注意：Q6_K 只有一个变体，没有 _S/_M/_L——网上写 Q6_K_M 的是没搞懂

效果：同样体积下，质量明显优于传统量化。 llama.cpp 社区在 Llama 2 7B 上的实测（perplexity 越低越好，fp16 基准 5.9565）：

结论：K-quants 是今天 90% 场景的答案。 解压方式和传统量化一样快，误差却低一截，没有理由不用。

第三代：I-quants（IQ2_XXS、IQ3_S）——为极限压缩而生

2024 年 2 月的 PR #4773 带来了 I-quants，灵感来自 QuIP# 论文。它解决的是 K-quants 到不了的地方：2-4 bpw 的超低比特率。

两个新招：

第一招：量化前先"转一转"。 对权重做一次 Hadamard 变换（随机旋转），让权重的分布变得更均匀，量化误差因此大幅下降。这是从 QuIP# 偷师的核心。

第二招：查找表。 解码时用码本（lookup table）查特殊值——IQ2 系列用 E8 格码本，IQ3/IQ4 用 k-means 聚类码本。非均匀编码让小值省位、大值保留精度。

代价：解码变贵。 查表 + 复杂计算，让 I-quants 从"内存带宽受限"变成"CPU 算力受限"。在内存带宽不足但 CPU 很强（或者纯 CPU 推理）的机器上，IQ 系列可能比同体积的 K-quants 更慢。

效果：IQ2_XXS 只有 2.06 bpw，质量却吊打老的 Q2_K。 如果你的内存/显存连 Q4_K_M 都放不下，IQ 系列是唯一还能保留对话质量的选择。注意 IQ4_NL 是例外——NL = No Lookup table，4.5 bpw 但没有查表开销，速度接近 K-quants。

隐藏外挂：imatrix（后缀 .imx）

这个和量化方法无关，是独立的一层增强。

问题：所有量化方法默认把每个权重当平等的。但权重不平等——有些权重对模型输出的影响大得多（比如 attention 相关的层）。误差砸在重要权重上，模型就变傻；砸在不重要的权重上，几乎无感。

imatrix 的答案：量化前先量一量。 拿一批校准文本（几十到一百 MB 的代表性语料）跑一遍，统计每个权重的重要性，生成一个"重要性矩阵"。量化时优先保证重要权重的精度，把误差集中在无关紧要的权重上。

用法： 先 llama-imatrix 生成矩阵文件，量化时 llama-quantize --imatrix 指定。带 imatrix 的 GGUF 文件通常在文件名里标 .imx。

效果： llama.cpp 社区实测，Q4 级别能获得可观的 perplexity 改善，Q3 及以下几乎"用了才是人能用的水平"。免费的午餐，能选 .imx 就别选不带 .imx 的。

一张表：到底选哪个

照着你的内存/显存选：

三个补充原则：

1. 老硬件优先传统量化？ 不绝对。只有在老显卡（Pascal 及以前）或极端兼容场景，Q8_0/Q4_0 才可能更快。新 CPU 和近五年显卡上，K-quants 全面占优。
2. 纯 CPU 推理 + 内存紧张：IQ 系列值得试——内存带宽是瓶颈时，IQ 用算力换体积，可能反而更快。
3. 能带 .imx 就带 .imx：同样后缀，imatrix 版本通常只大几个 MB，质量高一档。

最后一句

别在下载页纠结二十分钟。默认 Q4_K_M，跑起来；显存有富余升 Q6_K，想要保险升 Q5_K_M；放不下再降 Q3_K_M 或 IQ 系列。 模型参数本身的质量，比后缀差的那点精度重要得多——一个 7B 的 Q8_0，不会比一个 32B 的 Q3_K_M 聪明。

后缀是选衣服，模型才是人。

版权没有，随意转载。过意不去就加一句「原文发表于 cn-res.vip」，不加也无所谓。

GGUF Quant Suffixes, Decoded: Q8_0, Q6_K, Q4_K_M and How to Choose

You're on Hugging Face, you pick a model, and a wall of files hits you: Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q4_K_S, Q3_K_M, IQ2_XXS — and some with a .imx tag.

Same 7B model, the smallest file is less than half the largest. Pick big? Won't fit in VRAM. Pick small? Afraid it'll turn into a lobotomy patient.

This article decodes the suffixes. After reading it, you'll pick without hesitation.

Quantization in 30 Seconds

Model weights default to fp16 — each number takes 16 bits (2 bytes). A 7B model = 7 billion numbers ≈ 14 GB.

Quantization means: storing those numbers in fewer bits. 4 bits, 5, 6, 8... fewer bits = smaller file = more precision loss.

The key trick is block quantization: instead of optimizing each weight individually, group weights into blocks and let each block share a scale factor, spreading the error across the block. The difference between methods comes down to: block size, extra parameters per block, and how decoding works.

The unit to remember is bpw (bits per weight). 8.5 bpw = 8.5 bits per weight; 2.06 bpw = 2.06. Every suffix is ultimately telling you this number.

Reading the Label: Q4_K_M

Q4_K_M breaks into four parts:

• Q = Quantized
• 4 = 4 bits per weight (base precision)
• K = method family. K = K-quants; no K = legacy (Q4_0); I prefix = I-quants (IQ2_XXS)
• _M = tier, S/M/L (Small/Medium/Large). M is the middle, balanced choice

Understand that naming scheme and the rest is just details.

Generation One: Legacy Quants (Q4_0, Q4_1, Q8_0)

The oldest, simplest batch — mainstream before mid-2023.

Structure: 32 weights per block, sharing constants.

• Q4_0: 32 × 4 bits + one fp16 scale = 4.5 bpw
• Q4_1: Q4_0 plus a min (offset) constant = 5.0 bpw. The extra constant handles weights that aren't centered around zero
• Q8_0: 32 × 8 bits + scale = 8.5 bpw. Nearly lossless

Strength: brutally simple to implement. Decompression uses shifts, ANDs, and multiplies — no table lookups, no heavy math. On some older GPUs and CPUs, legacy quants can actually be faster.

Weakness: simple means dumb. All weights treated equally, nothing protected. Q4_0's error at the same size is noticeably worse than later methods. Today it survives mainly as Q8_0 (because 8-bit is so close to lossless it doesn't matter) and in compatibility scenarios.

Generation Two: K-quants (Q6_K, Q5_K_M, Q4_K_M) — the Current Mainstream

August 2023, llama.cpp PR #1684 introduced K-quants. The core idea: spend precision where it matters. Two things:

First: mixed precision inside the block. Block size grows from 32 to 256 weights, split into 16 sub-blocks (16 weights each). Sub-blocks quantize at low precision, while the block carries a high-precision super-scale to correct them. Outlier weights no longer drag the whole block down; important weights keep their fidelity.

Second: different tensors, different treatment. That's what _S/_M/_L mean:

• Q4_K_S: uniform Q4_K blocks everywhere, 4.5 bpw
• Q4_K_M: most layers at Q4_K, but attention q/k/v and feed-forward output layers get Q6_K, 4.85 bpw
• Q5_K_M: same scheme, ~5.5 bpw
• Q6_K: 6-bit blocks everywhere, 6.56 bpw. Note: Q6_K has no _S/_M/_L variants — anyone writing "Q6_K_M" doesn't know the format

Result: at the same size, quality is clearly better than legacy. llama.cpp community measurements on Llama 2 7B (perplexity, lower is better; fp16 baseline 5.9565):

Bottom line: K-quants answer 90% of today's use cases. Same decompression speed as legacy, lower error. No reason not to use them.

Generation Three: I-quants (IQ2_XXS, IQ3_S) — Built for Extreme Compression

February 2024, PR #4773. Inspired by the QuIP# paper, I-quants target what K-quants can't reach: ultra-low 2–4 bpw.

Two new tricks:

First: rotate before you quantize. A Hadamard transform (random rotation) on the weights makes their distribution more uniform, dramatically reducing quantization error. That's the QuIP# trick.

Second: lookup tables. Decoding consults codebooks — E8 lattice for IQ2, k-means codebooks for IQ3/IQ4. Non-uniform encoding lets small values save bits while big values keep precision.

The cost: decoding gets expensive. Table lookups and heavier math flip I-quants from memory-bandwidth-bound to compute-bound. On machines with weak compute but tight memory bandwidth, IQ variants can be slower than K-quants of the same size.

The payoff: IQ2_XXS at 2.06 bpw still produces usable conversations, where the old Q2_K was a disaster. If your RAM/VRAM can't fit Q4_K_M, IQ is the only family that keeps the model human. IQ4_NL is the exception — NL = No Lookup table, 4.5 bpw with speed close to K-quants.

The Hidden Extra: imatrix (the .imx suffix)

Orthogonal to the quant method — a separate enhancement layer.

The problem: every quant method treats all weights as equal. They aren't. Some weights matter far more to model output (attention-related layers especially). Error on an important weight degrades the model; error on a minor weight is noise.

The imatrix answer: measure before you quantize. Run a calibration corpus (a few tens to a hundred MB of representative text) through the model, estimate each weight's importance, and produce an importance matrix. Quantization then protects important weights and pushes error onto the rest.

Usage: generate the matrix with llama-imatrix, then quantize with llama-quantize --imatrix. GGUF files quantized this way are usually labeled .imx.

Effect: community measurements show meaningful perplexity improvements at Q4 and below; below Q3 it's practically mandatory. It's a free lunch — pick the .imx file when one exists.

The One-Table Decision Guide

Pick by your memory budget:

Three rules of thumb:

1. "Old hardware prefers legacy"? Not a blanket rule. Only on old GPUs (Pascal and earlier) or extreme compatibility cases might Q8_0/Q4_0 be faster. On modern CPUs and GPUs from the last five years, K-quants win outright.
2. Pure CPU + tight RAM: try IQ — when memory bandwidth is the bottleneck, IQ trades compute for size and can end up faster.
3. Take the .imx version whenever it exists: same suffix, a few MB bigger, one tier better quality.

The Last Word

Don't burn twenty minutes on the download page. Default to Q4_K_M and start; bump to Q6_K if VRAM is spare, Q5_K_M if you want insurance; drop to Q3_K_M or IQ only when you must. The model's base quality matters more than the suffix's last bit of precision — a 7B at Q8_0 won't outsmart a 32B at Q3_K_M.

The suffix is the outfit. The model is the person.

No copyright reserved — share it freely. If you feel like it, a credit line to cn-res.vip is appreciated. Or don't. Either way.