GPTQ¶

GPTQ is a Hessian-based post-training quantization method that finds optimal quantized weights by minimizing the layer-wise output error.

Reference

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, "GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers," ICLR 2023.

Algorithm¶

GPTQ formulates quantization as a per-layer optimization problem:

\[ \min_{\hat{W}} \| W X - \hat{W} X \|_F^2 \]

It solves this column-by-column using the inverse Hessian \(H^{-1} = (2 X X^T)^{-1}\). For each column \(i\):

Quantize column \(i\) of \(W\) to the nearest quantization level
Compensate the remaining unquantized columns using the Hessian information

This produces significantly better results than simple round-to-nearest (RTN) quantization, especially at lower bit-widths.

Parameters¶

Parameter	Type	Description	Default
`wbits`	`int`	Quantization bit-width	`4`
`groupsize`	`int`	Group size for group-wise quantization (-1 = none)	`-1`
`sym`	`bool`	Symmetric quantization	`True`
`actorder`	`bool`	Reorder columns by activation magnitude	`False`
`percdamp`	`float`	Hessian damping percentage	`0.01`
`bitpack_on_quantize`	`bool`	Pack `qweight` and `qzeros` after each layer; supported resolved `wbits` are `{2, 3, 4, 8}` only	`True`

Usage¶

Basic 4-bit quantization¶

from onecomp import GPTQ

gptq = GPTQ(wbits=4, groupsize=128)

3-bit with activation ordering¶

gptq = GPTQ(wbits=3, groupsize=128, actorder=True)

Asymmetric quantization¶

gptq = GPTQ(wbits=4, sym=False)

With QEP for improved quality¶

from onecomp import Runner, ModelConfig

model_config = ModelConfig(model_id="meta-llama/Llama-2-7b-hf", device="cuda:0")
gptq = GPTQ(wbits=3, groupsize=128)

runner = Runner(
    model_config=model_config,
    quantizer=gptq,
    qep=True,
)
runner.run()

Quantize-time Bitpacking¶

When bitpack_on_quantize=True (the default), GPTQ stores qweight and qzeros in bitpacked form immediately after each layer is quantized. GPTQ supports this mode only when every resolved bit-width is one of {2, 3, 4, 8}. Here, resolved means the width selected after applying mlp_wbits or module_wbits; validation fails if any resolved width is not packable.

With bitpack_on_quantize=False, the GPTQ quantization algorithm accepts any integer wbits from 1 through 15 and keeps the quantization results unpacked.

Packing when creating or saving a model

bitpack_on_quantize controls only the representation held in the quantization results. Runner.create_quantized_model() and Runner.save_quantized_model() use the separate pack_weights=True default and reject widths outside {2, 3, 4, 8}. For another width, pass pack_weights=False to create an unpacked inference model or to write unpacked tensors. These tensors do not use the standard packed GPTQ format and are not compatible with vLLM's GPTQ kernels.

Group-wise Quantization¶

When groupsize > 0, weights are divided into groups of consecutive columns, and each group has its own scale and zero-point. This improves quantization accuracy at the cost of slightly more storage for the quantization parameters.

Typical values:

groupsize=128 -- good balance of accuracy and compression
groupsize=-1 -- per-channel quantization (no grouping)

Activation Ordering¶

When actorder=True, columns are reordered by their activation magnitude (Hessian diagonal) before quantization. Columns with higher activation influence are quantized first, which can improve accuracy. The permutation is stored so weights can be reconstructed correctly.