One exponent for thirty-two numbers

Eight bits, and a format that breaks the rules ends on an arithmetic observation. At fifty-three bits the exponent field costs eleven bits out of sixty-four and nobody thinks about it; at eight bits it costs four or five out of eight, which is more than half the format.

What it buys is the ability of each individual value to sit anywhere across seventeen octaves. Thirty-two values drawn from the same place in the same computation are usually within a factor of a hundred of each other, so most of them are carrying a description of a range they do not occupy.

So spend the exponent once for a group. Thirty-two values share one eight-bit scale and each keeps a small signed integer, and the cost per value is the integer plus 8/32 of an exponent:

That is 22% less storage. Whether it is also less accurate turns out to depend on a quantity that appears in no bit count.

What the shared exponent is actually charging for #

The scale is set by the largest value in the block, because it has to be — a scale that overflowed the largest entry would lose it entirely. So it is chosen as the power of two that puts the block maximum at the top of the integer range, and every other entry is then represented as a multiple of a step chosen for somebody else.

An entry a thousand times smaller than the block maximum therefore has ten fewer significant bits than the maximum does. An entry small enough has none at all: below half a step it rounds to zero and is gone.

The bound is one line. The step is the same for every entry, so an entry of magnitude v has an absolute error of at most half a step and a relative error of at most

step / 2|v|

and since the step is at most twice the block maximum over the number of levels, that is at most M/(L|v|) — which contains the entry’s own size only through the ratio M/v. Two entries of wildly different magnitudes, in blocks of wildly different scales, have the same relative error if they sit the same distance below their own block’s top.

That is checked entry by entry over three hundred values, against both forms of the bound. The tighter one, from the step, is attained to within 0.5%; the looser one, from the maximum, is never attained, and the gap between them is the up-to-one-bit that a power-of-two shared scale wastes. A block whose maximum happens to sit just above a power of two uses half its integer range and nothing else.

So the deciding quantity is the dynamic range inside a block, in octaves, and it is a property of the ordering as much as of the values. That is the sentence the rest of the essay is about.

Where it wins and where it loses #

The comparison is a six-bit block format at 6.25 bits a value against E4M3 at 8, on 640 values whose magnitudes are spread log-uniformly over a chosen number of octaves.

The cheaper format is the more accurate one out to about three octaves of within-block range and the less accurate one beyond it, and the crossing is where the arithmetic above says it should be: six significand bits give thirty-one levels, so an entry more than about five octaves below its block’s maximum has fewer than one level left.

That is a usable rule and it is not a rule about the bit count. A four-bit block format at 4.25 bits is beaten everywhere on this data; an eight-bit one at 8.25 wins everywhere and costs more than the per-element format it beats. Six is the interesting width because it is cheaper and better, in a regime, and the regime is stated in octaves rather than in bits.

The two readings, and they disagree by sixty #

The right-hand column of that table is the reason to distrust the middle two.

A relative error in the 2-norm is dominated by the largest entries — which are exactly the entries a shared scale represents well. So across sixteen octaves the block format’s 2-norm error grows by a factor of 1.76, and never looks disastrous. Over the same range the median entry’s relative error grows by a factor of 62, and ends at exactly 1.00.

A relative error of exactly one means the entry is zero and the whole of it is gone. More than half the numbers have been deleted while the norm moved by a factor of two.

The per-element format, over the same range, moves by 5.5 in the median and deletes nothing until its own subnormals run out below 2⁻⁹.

This is the essay’s transferable point and it is not about block formats. A norm is a summary weighted by magnitude, and any quantisation that treats large and small entries differently will look better in a norm than it is. Reporting one number for a whole tensor is reporting what happened to its largest entries; if the small entries are load-bearing — and in a matrix they often are, since rank and conditioning are both decided at the small end — the norm is measuring the wrong thing.

Where the block size comes in #

Thirty-two is a number this essay has been taking as given, and it is a trade with two sides.

A larger block amortises the shared exponent further: at 32 values a value costs the significand plus a quarter of a bit, at 64 an eighth, at 128 a sixteenth. Past about 32 the saving is negligible — going from 6.25 to 6.125 bits a value is a 2% storage improvement — so the storage argument stops paying almost immediately.

The other side does not stop. A larger block has a wider internal dynamic range, because it holds more values and the maximum can only grow, so every additional value increases the chance that some entry sets a scale the others cannot live with. The two effects therefore run in opposite directions with completely different urgencies: the benefit saturates by 32 and the cost keeps accruing.

That is why the block sizes hardware settled on are small, and why nobody proposes sharing one exponent across a whole tensor — which would be free in storage and would be a single global scale, which is to say fixed point, which is what floating point was invented to replace.

A smaller block runs the argument the other way and it is not free either: at 8 values a value costs the significand plus a whole bit, which is a 17% storage penalty at six bits. Somewhere near 32 the two curves cross, and the specifications agree with the arithmetic.

One outlier takes the block #

The cleanest case, and the one these formats are actually deployed into. Activations and weights are not log-uniform; they are mostly of a size with occasional outliers.

Take 320 values between 0.5 and 1.5, with one entry of 64 at the start of every block of 32. At six significand bits there are 31 levels, so the step is 2^ceil(log₂(64/31)) = 4 — larger than the values themselves. Every one of the thirty-one ordinary entries in each block rounds to zero: 310 of 320 entries deleted, by one number in each block that has nothing to do with them.

The threshold is arithmetic rather than a regime. At an outlier of 32 the step is 2 and only the entries below 1 are lost — about half of them. At 64 the step is 4 and all of them go. Two bits of significand also decide it: at eight bits there are 127 levels, the step is 1, and nothing is lost at all.

A per-element format at eight bits keeps every one of them, in every case, because it gives each value its own exponent — which is the thing being economised on.

The order the numbers arrive in is part of the format #

Sort the same 320 values by magnitude and quantise again, changing no value and applying no numerical technique. The deletions go from 310 to 22, and the 22 that remain are the ordinary values that share the first block with the ten outliers, because thirty-two is the block size and ten is not.

On log-uniformly spread data the same experiment takes the median entry’s error from 10.9% to 1.6% and the deletions from 78 to zero.

That is a property no other format on this site has. A per-element format’s error on a vector does not depend on the order of the entries at all — the rounding of each is a function of that entry alone, and the check confirms it agrees to 10⁻¹². A block format’s does, because the block boundaries are at fixed positions and which values share a scale is decided by where they happen to sit.

So the same numbers in a different order are a different format, and permutation — which is free, which changes no value, and which every other essay on this site treats as the thing that does not affect an answer — becomes a numerical decision. The ordering essays made the same discovery about fill in a sparse factorisation, and the parallel is exact: a symmetric permutation changes the cost and not the answer there, and changes the accuracy and not the values here.

What this is not, and the format it nearly is #

One clarification, because the construction here has an older name and a nearly identical rival.

A block format with a single block covering the whole vector is fixed point: one scale for everything, integers below it. Everything this essay measures is the same argument with the block made small enough to track the data’s local scale, and the whole reason floating point exists is that a single global scale cannot. So a block format sits between the two, and how far along it sits is decided by the block size — which is why the previous section’s trade is the format’s central parameter rather than an implementation detail.

The rival is per-channel or per-row scaling, which shares an exponent along a meaningful axis of the data rather than along thirty-two consecutive addresses. It has the same arithmetic and a better claim on the grouping: the values in one row of a weight matrix have a reason to be of a size with each other, and thirty-two consecutive entries of a flattened array have none. Its cost is that the scale is no longer a fixed small overhead per fixed number of values, so the storage accounting stops being one division.

Both are in production and this essay measures the first, because the first is what the hardware specifications standardised and because its cost is stateable in one number. The measurement that sorting removes almost all the damage is, read the other way, a measurement of how much the second kind of grouping is worth: sorting is the best possible grouping and is unavailable, and grouping by a real axis of the data is the achievable approximation to it.

What is asserted, and what refuses #

The cost accounting is right: 4.25, 6.25 and 8.25 bits a value, and a six-bit signed significand holds thirty-one levels either side of zero. Counting the sign bit as a magnitude bit would inflate every comparison here by a factor of two, which is the arithmetic slip this essay is most likely to contain.

Every entry’s error is inside both forms of the bound, over three hundred entries, with the tighter one attained to 0.5% and the looser one never attained.

The crossing is where the block range runs: block wins under two octaves, loses past four, and deletes nothing at the narrow end.

The two readings disagree by more than twenty times, asserted as a ratio of growth factors, and the median is asserted to end at exactly 1.

One outlier deletes more than 80% of the vector at six bits and none at eight, with the step printed in both cases so the threshold is visible as arithmetic.

Sorting confines the loss to one block, asserted with the residue — 22, not zero — rather than rounded up to a cleaner claim.

The claim that the wider format is the more accurate one is refused. 6.25 bits against 8 on narrow data is fed to rejects() and the wider format loses.

Neither format is safe on an outlier, which is the assertion easiest to leave out of a comparison one of the two wins: the per-element format keeps every entry, and at sixteen octaves of spread its own median error still reaches 10%, because below 2⁻⁹ it has run out of subnormals and the entries there are gone too. The block format fails earlier and more completely; it does not fail alone.

And reading the damage off the 2-norm is refused, which is the assertion this essay most needed: assertClose(normError, medianError, 0.5) on sixteen-octave data must fail, and it does, by a factor of nearly thirty.

What is left #

The arithmetic. Everything here is quantisation — taking values in double and storing them. What a matrix product in a block format costs in accuracy, where the accumulation happens and in what width, is the question these formats exist to answer and is not measured.

The real microscaling formats, whose elements are themselves tiny floating-point numbers rather than integers, so an element carries one or two exponent bits of its own on top of the shared scale. Every argument here is about the shared exponent, which they all have; the element encoding changes the constants and not the shape.

Choosing the blocks. The measurement that sorting removes almost all the damage says there is a grouping problem worth solving, and grouping by magnitude is not available in practice — the block boundaries have to match how the hardware reads memory. What is available is choosing which axis of a matrix the blocks run along, which is a real decision with a measurable answer and is not made here.