Paper Readings · 2026-10-06 · 3,332 words · 15 min read

TORQUE re-implemented: where keeping outliers around a rotation pays

A reading of arXiv 2609.36032v1, re-run on Qwen2.5-0.5B: keeping outliers cut activation error at every budget; the KV-cache gain faded once its bits were paid.

quantisation Hadamard rotation KV cache outliers

Rotation-based quantizers multiply a vector by a random rotation so that its coordinates look Gaussian, then round them with a codebook designed once, offline, for that Gaussian. The paper discussed here cites QuIP#, QuaRot and SpinQuant among the inference uses of the idea and notes that vLLM ships a backend with a Hadamard rotation. arXiv 2609.36032v1, "TORQUE: Optimizing What (not) to Quantize Before and After Rotation", posted on 28 September 2026 by Ran Ben Basat (University College London), Michael Mitzenmacher (Harvard University) and Shay Vargaftik (VMware Research by Broadcom), adds one decision to that recipe: which values to keep exactly instead, before the rotation and after it, paid for out of the same bit budget. We read the PDF, rebuilt the method around a textbook inlier quantizer, ran it on the model and text the paper uses, Qwen2.5-0.5B on the WikiText-2 test split, and measured one thing the paper does not report: perplexity with the quantized cache inside the model. We found no code release in the paper or on its abstract page. Ours is code/torque-outlier-retention.py, and every number we computed is in datasets/torque-outlier-retention.csv.

What the paper claims

The abstract makes three claims. A uniform random rotation makes a normalised vector's coordinates approximately Gaussian, which is what lets a codebook be optimised offline. Keeping large input coordinates at high precision before the rotation stops the rotation from spreading them across every coordinate, and keeping the largest rotated values afterwards lets the rest be quantized over a narrower range. And choosing both jointly, under one expected bit budget, improves the trade-off between reconstruction error and storage, shown under the Gaussian model and on nearest-neighbour retrieval, KV-cache compression and activation compression.

The formal result is narrower than the pitch, and its proof is short and sound: for each count k, keeping the k largest-magnitude inputs minimises the paper's error bound (Appendix A, Proposition 1). That turns a search over subsets into a search over three numbers, which is what makes the method cheap enough to run on every vector.

How the method spends its bits

The encoder does four things to a vector of d coordinates. It stores the k largest inputs with their positions and zeroes them. It scales what remains to norm √d and rotates it. It stores every rotated value whose magnitude exceeds a threshold c, again with its position. And it quantizes the rest, the inliers, at s bits each with a codebook built offline for a standard Gaussian cut off at ±c. In the paper's experiments a kept value costs 16 bits and its position 8, so a kept coordinate costs 24 bits at either stage. Overheads like these are easy to lose behind a format's name, the way a GGUF Q4_K block costs 4.5 bits per weight rather than 4. The expected cost is 24k + d·[p(c)·24 + (1 − p(c))·s], where p(c) is the chance that a standard Gaussian lands beyond ±c, and the error is bounded by ρ·ε(c, s): the share of the vector's energy left after the k inputs are removed, times the inlier quantizer's error per unit of energy. The proof that the k largest inputs are the right ones is a single exchange: swapping a kept coordinate for a larger unkept one lowers ρ and changes no cost.

A worked case from the listing. A 256-coordinate chunk at 4.5 bits has 1,152 bits to spend. The best plan that keeps nothing before the rotation uses c = 2.75 and s = 4.375, a mix of 4-bit and 5-bit codes, for a modelled error of 0.005274 per unit of energy. The plan c = 3, s = 4 spends an expected 1,037.8 bits on the rotated vector, since only 0.27% of Gaussian values land beyond ±3, which leaves room for four kept inputs; its modelled error is 0.008023 times the share of energy those four leave behind. It wins whenever the four largest coordinates carry more than 34.3% of the chunk's energy. In the activations we captured from Qwen2.5-0.5B, the median share in the first 256-coordinate chunk is 46.3% at the input to the attention projections and 40.1% at the input to the MLP down-projection, against 26.0% and 23.7% at the other two points, so the choice really does change from one vector to the next.

The same arithmetic predicts most of what follows. A kept value costs 24 bits however long the vector is: 0.094 bits per coordinate in a 256-coordinate chunk, but 0.375 in one of Qwen2.5-0.5B's 64-coordinate attention heads, the unit in which its key and value cache is stored (what that cache costs in memory is worked out in our KV cache piece).

What the paper measured

The application results are reconstruction error, reported as the reduction in normalised mean squared error (NMSE) at the same bit budget. On Qwen2.5-0.5B with the WikiText-2 test split (section 3.1), the authors record four activation vectors in each of the 24 decoder blocks for 64 tokens and quantize them in 256-coordinate chunks, and they build key and value caches from 16 segments of 255 tokens, quantizing each 64-coordinate head vector on its own. Adding TORQUE cuts activation NMSE by 30.8 to 51.5% for EDEN, 31.8 to 86.2% for RaBitQ and 33.6 to 51.2% for TurboQuant; key NMSE by 13.9 to 44.1%, 16.4 to 94.7% and 19.7 to 44.8%; and value NMSE by 0.8 to 19.6%, 1 to 92% and 1.2 to 19.4%. Those are the unbiased versions of the three quantizers. The biased versions appear only as plots, in Appendix C.1, where the KV caches use retention after the rotation only.

On GloVe-200 (1,183,514 vectors, 10,000 queries) the mean paired NMSE reductions reach 20.9%, 81.5% and 20.3%, and Recall@1 gains up to 1.5 percentage points; on GIST the authors write that lower reconstruction NMSE does not consistently improve Recall@1. Under the Gaussian model (section 3.4) the scalar reductions reach 25.69% for EDEN, 53.73% for RaBitQ and 24.65% for TurboQuant. The price is encoding time: on an RTX 5090 at d = 256 and 4.5 bits, encoding takes 48.4 to 99.7% longer with both stages, and on a CPU it takes 2.08 to 8.85 times EDEN's encoding time once the parameter search is included, while decoding takes 1.12 to 1.41 times EDEN's.

What we rebuilt

The paper's inlier quantizers are EDEN, RaBitQ and TurboQuant, and its main figures use their unbiased forms. Ours is simpler and exact: a Lloyd-Max codebook, the minimum-error scalar quantizer for a standard Gaussian cut off at ±c, computed from closed-form integrals rather than samples. It is deterministic and therefore biased, which places it beside the paper's biased variants in Appendix C.1, TurboQuant-MSE above all, rather than beside its headline figures. Around it sit the paper's own settings: a randomized Hadamard rotation with fresh random signs for every vector, the threshold set C = {1.75, 2, 2.25, 2.5, 2.75, 3, 3.5, 4, 4.5, 5, 6, ∞} and the bit grid s = 1 + j/64 from Appendix D, 16-bit values with 8-bit positions, and plans chosen by its Equation 5. One assumption is ours: a fractional s is realised as a mix of the two neighbouring whole-bit codebooks, because the paper does not say how its inliers reach fractional rates.

The checks came first. The 1-bit codebook reproduces the textbook values, levels at ±√(2/π) and an error of 1 − 2/π; the exact errors match two million Gaussian samples to within 1%; and on rotated Gaussian vectors of 256 coordinates the measured error lands within 1% of the model at 4, 4.25 and 8 bits, with and without retention. The search over plans, as shipped:

def candidates(d: int, b: float, stage: str):
    """(c, k, s, side_bits) plans. The paper's rule: for each (c, s) the largest feasible k wins,
    equivalently for each (c, k) the largest feasible s; we enumerate (c, k)."""
    if stage == "none":
        return [(math.inf, 0, b, 0)]
    charged = stage == "charged"
    side = side_bits_on(d) if charged else 0
    out = []
    if charged:   # the 'off' plan: plain quantizer, one flag bit
        out.append((math.inf, 0, best_s((d * b - 1) / d, 0.0), 1))
    for c in C_SET:
        if stage == "pre" and math.isfinite(c):
            continue
        p = p_out(c)
        kmax = 0 if stage == "post" else d
        for k in range(0, kmax + 1):
            s = best_s((d * b - side - L_RET * k) / d, p)
            if s is None:
                break
            if charged and k == 0 and not math.isfinite(c):
                continue       # same as the off plan but dearer
            out.append((c, k, s, side))
    return out

Enumerating (c, k) instead of (c, s) finds the same optimum: the paper's rule takes the largest affordable k for each (c, s), and the error only falls as s rises, so for each (c, k) only the largest affordable s can win.

Under the Gaussian model the gain is a sawtooth

A Gaussian vector has nothing for the first stage to grab, so the Gaussian model isolates the second stage, as the paper's Figure 5(a) does. The comparison is the same quantizer with and without retention at the same expected budget; between whole bit widths, the plain quantizer mixes its two neighbouring codebooks, which is how the paper says a raw quantizer has to meet a fractional budget.

At whole budgets the gain is small: nothing at 2 and 3 bits, 1.80% at 4, 11.43% at 8. A quarter of a bit above them it jumps, to 16.30% at 4.25 and 24.79% at 7.25, the largest on our grid. The paper's largest figure for TurboQuant is 24.65%, and its plotted curves have the same teeth. The bit accounting explains them. At a whole budget, keeping any rotated value forces the inliers below a whole bit and into a mix with the next-cheaper codebook, which costs almost as much accuracy as the narrower range saves. Just above a whole budget the inliers can sit on a whole bit while the fractional remainder pays for the tail; the paper makes the same point about budgets slightly above an integer.

Mixing two codebooks is not the only way to spend a fractional budget, though. At 4.5 bits, 22 levels fit two values into one 9-bit code (22 × 22 = 484, under 512), and that fixed-rate code has an error of 0.005170, which is 13.9% below the mixed baseline and 2.0% below TORQUE's 0.005274. Across the six half-bit budgets, TORQUE's reduction against such a packed code runs from −6.2% at 3.5 bits to +6.8% at 2.5. Under the Gaussian model, then, most of the fractional-budget gain is a gain over mixed precision rather than over the best fixed-rate code.

Skewed inputs show the first stage at work. On the paper's two synthetic families at 8 bits (our run: 256 vectors of 256 coordinates, five seeds), near-Gaussian inputs gain only from the second stage, 10.2% for a signed lognormal with log-variance 0.1 and 11.2% for a Student-t with 30 degrees of freedom, while a lognormal with log-variance 10 gains 96.7% from the first stage alone and 97.1% from both. The paper reports the same split, up to 99.68% against EDEN, although one parenthesis in that passage reads backwards: it calls the low-θ² and low-h end "heavy-tailed", two sentences after defining larger θ² and h as the direction that makes large coordinates more pronounced. Its plot follows the definition, not the parenthesis.

On Qwen2.5-0.5B the gain follows the outliers

We captured the paper's setting as closely as its text allows. For activations: the same four points in all 24 blocks for 64 tokens, from eight 256-token segments at positions 31, 63 and so on to 255 (the paper does not say which eight tokens it took), chunked 3 × 256 + 128 for the 896-wide inputs and 19 × 256 for the 4,864-wide input to the down-projection. For the cache: keys after the rotary embedding, as they are stored, and values, from 16 segments of 255 tokens, 195,840 head vectors of 64 each. The model ran in float32 on a CPU. The table gives the NMSE reduction against the plain rotated quantizer at the same budget, first with TORQUE's per-vector choices free, then with them paid for.

The activation result holds up, and it is not a budget effect. Keeping inputs before the rotation is worth 15.0 to 24.9% on its own at every budget, whole or fractional, and both stages together reach 16.6 to 40.3%. It also survives the strongest comparison we have: TORQUE beats the packed code on activations at all six half-bit budgets. The key and value caches look like the Gaussian model instead, with small reductions at whole budgets and larger ones at half budgets, where the packed code does better than TORQUE at 3.5, 4.5, 5.5 and 6.5 bits on keys and from 3.5 to 7.5 bits on values. Keeping key inputs before the rotation is worth only 1.5 to 4.7%, although the four largest coordinates of a key vector hold a median 48.1% of its energy, more than in any activation chunk. The price per kept value is the likely reason: in a 64-coordinate head, every kept value takes 0.375 bits from every coordinate.

Two checks on the bookkeeping. TORQUE's realised spend came in at or under budget everywhere; its plans are made from expected costs and round s down to the 1/64-bit grid, and keys at 4.5 bits, for instance, averaged 4.461 bits. And the per-vector choices are not free. The decoder has to learn which plan was used: at fixed width that is a 1-bit on/off flag, 4 bits for c and two counts, 19 bits for a 64-coordinate vector when on and 1 bit when off. The paper's formulation says metadata belongs inside the budget (section 2.2), while its timing appendix reserves 160 bits per vector outside the budget for both methods, retained counts and quantizer identifier included; which accounting its Figure 2 uses is not stated, so we ran both. Paid for, the key and value gains are gone: keys land between −3.4% and +1.0% against the plain quantizer and values between −4.2% and −1.2%, and at whole budgets even the 1-bit flag hurts, because the plain path then has to drop one of its 64 coordinates to a lower bit width. Activations, where the side information (23 bits) is spread over 256 coordinates, keep 10.3 to 34.0%.

Perplexity, which the paper does not measure

The paper stops at reconstruction error. We put the quantized cache inside the model: every key (after the rotary embedding) and every value vector was quantized and dequantized before attention in all 24 blocks while the model scored 64 non-overlapping 256-token segments of the WikiText-2 test split, 16,320 predicted tokens. Unquantized, the model scores a perplexity of 20.11. TORQUE here is the version with its choices free; the paid version was not run inside the model. The fourth column is the paired difference in mean negative log-likelihood per token, TORQUE minus the rotated quantizer, with a 95% bootstrap interval over the 64 segments, computed the way our bootstrap piece describes. The last column is a plain absmax quantizer: each vector scaled by its largest magnitude and rounded, with no rotation.

Two things stand out. TORQUE's lower reconstruction error does carry through, in direction at least: its perplexity is lower at eight of the nine budgets, by 0.004 to 0.262 nats per token, but the interval excludes zero only at 6.5 and 7.5 bits, two of the budgets where its key and value error reductions are largest, and at 4.5 bits it is worse, with an interval that also spans zero. And neither rotated quantizer is the right tool for this model's cache at 4 to 6 bits. The absmax quantizer has 2.71, 2.32 and 2.20 times the rotated quantizer's key error at 4, 5 and 6 bits, yet its perplexity is 59.3%, 62.2% and 43.3% lower: 82.7, 38.1 and 24.1 against 203.1, 100.8 and 42.4, with TORQUE at 156.2, 84.2 and 42.0. At 7 and 8 bits the three end up close together: 24.6, 23.5 and 24.1 at 7 bits, 20.9, 20.8 and 21.1 at 8. Reconstruction error, the paper's metric and the quantity TORQUE minimises, put the absmax quantizer last at 4 to 6 bits, where its perplexity was the best by a wide margin.

We did not isolate why. Two measured facts are consistent with it: absmax reproduces each vector's largest coordinate exactly, since that coordinate sets the scale, and Qwen2.5-0.5B's key vectors hold a median 23.1% of their energy in a single coordinate, while a random rotation spreads the rounding error across all 64 coordinates on average, that one included. Which coordinates the attention scores depend on most is the measurement that would settle it, and we did not make it.

What holds up

The bound and its proof are sound, and the activation result reproduces in direction with a different inlier quantizer: keeping outliers cut activation error by 16.6 to 40.3% at the same budget in our run, against 30.8 to 51.5% in the paper's EDEN runs and 33.6 to 51.2% in its TurboQuant runs, both with unbiased quantizers rather than our biased one. That is the part of the paper we would build on. The key-cache figure needs the most qualification. Ours shows the paper's direction, 1.4 to 20.1% while the choices are free, and it mostly carries into perplexity, but at whole budgets little of it is left, at half budgets a packed code does better on keys at four of the six budgets, and paying for the side information removes it. What we could not check: the encoding-time figures (a NumPy loop says nothing about a fused CUDA kernel), the EDEN, RaBitQ and HIGGS results themselves, and the GloVe and GIST retrieval runs.

What it does not establish

The paper measures reconstruction error, plus Recall@1 for retrieval, where its own appendix notes that lower error does not consistently buy recall. It reports no model-quality measurement for the activation and KV experiments, which run on a single 0.5-billion-parameter model, and it does not say whether its Figure 2 charges the per-vector choices to the budget. Our limits are as plain: one model, our token positions, our inlier quantizer and side-information encoding, 64 segments of text, the paid-for variant left out of the perplexity run, and a perplexity check in which every quantizer is far from the unquantized model below 6 bits, so differences there compare configurations that are all badly damaged.

What to do with it

If you quantize activations behind a Hadamard rotation in chunks of 256 or more, keeping a handful of the largest inputs exactly is worth trying: it is cheap to decode, its gain did not depend on the budget, and it survived every accounting we applied. For a KV cache stored in 64-coordinate heads, count the side information, compare against a codebook packed to the same fractional budget before crediting retention with a gain, and measure perplexity rather than reconstruction error. On a model whose keys carry a few very large coordinates, as Qwen2.5-0.5B's do, try the plain option first: at 4 to 6 bits the absmax quantizer, with more than twice the key error, gave a perplexity 43.3 to 62.2% lower than the rotated quantizer's.