A bit buys an octave

One exponent for thirty-two numbers establishes what a shared exponent charges for. Thirty-two values keep one scale between them, the scale is set by the largest of them, and every other entry is stored as a multiple of a step chosen for somebody else. An entry far enough below the block maximum has no bits left at all, and below half a step it is not approximated but deleted.

Every measurement in that essay is taken at six significand bits. Six is what a specification chose; it is not a property of the arithmetic, and the arithmetic has a second axis that essay holds fixed. So the question this one asks is the obvious one and its answer turns out to be exact: what does one more significand bit buy, and is it the thing that was going wrong.

The quantity being bought is a range. A block survives an outlier up to some size and is destroyed by anything larger, and the size of the survivable outlier is what a wider significand moves. The answer is a single power of two at every width tried, the return on a bit is one octave — one doubling — and a bit costs one bit per value across the whole vector. Against that sits a repair that costs nothing at all and is not on the same scale.

The threshold is a power of two and the power is the width #

The vector is the one these formats are actually deployed into: values of a size with each other, of magnitude between 0.5 and 1.5, with one large entry per block. It is not log-uniform, because activations and weights are not, and the failure it produces is not a gradual loss of digits.

Take the number of levels first, because everything else follows from it. A b-bit signed significand holds 2^(b−1) − 1 levels either side of zero — 3 at three bits, 31 at six, 255 at nine — and the shared scale is the power of two that puts the block maximum inside that range, rounded up. So the step is the smallest power of two at or above the outlier divided by the number of levels, and the ordinary entries, which sit between 0.5 and 1.5, are deleted when half that step exceeds them.

That gives three regimes and they are separated by exact integers rather than by a regime boundary. A step of 1 leaves half a step at 0.5 and nothing is lost, because no ordinary entry is smaller than 0.5. A step of 2 puts half a step at 1 and takes every entry below 1, which on this data is 150 of 320. A step of 4 puts half a step at 2, above the largest ordinary value there is, and takes all 310 of them.

So the arithmetic predicts a threshold at the outlier where the step reaches 4, which is where outlier / levels passes 2, which is outlier > 2^b − 2. Probed at integer granularity the first outlier that deletes an entire vector is 7 at three bits, 63 at six, 127 at seven and 511 at nine — in each case 2·levels + 1, and the two that separate it from 2^b are the level the sign bit costs, counted twice. Among the powers of two, which is what a slider and a specification both deal in, the threshold reads 2^b at every width from three to nine, with no exception and no rounding.

An outlier of 8 is not extreme. It is the ratio between a large activation and an ordinary one in almost any tensor, and at three significand bits it removes every value that is not itself an outlier. The step here is 4, computed from 8 over 3 levels, and every ordinary entry in the vector is below 2.

What the next bit does, in the same column #

The clean way to read the return on a bit is to hold the data still and widen the format, which is the opposite of how these comparisons are usually run. The same 320 values, the same outlier of 8, and one more bit.

One bit has taken the vector from total loss to half loss, and it has done it by halving the step rather than by improving anything about the entries that survive. The next bit finishes the job.

Three figures, one vector, three widths, and the whole of the return on a bit is visible in the first bar of each: 310, 150, 0. The outlier of 8 is fatal at three bits, half fatal at four and harmless at five, and the reason is that 8 is 2^3, 2^4 / 2 and 2^5 / 4 respectively. Nothing about the data changed and nothing about the block size changed.

The same statement runs the other way, and it is worth drawing at the far end of the range because a law confirmed only where it was discovered is not a law.

Two more bits and two more doublings, at the widest setting the drawing reaches.

Laid out as a table, the widths and their thresholds are one column of powers of two against another, and the half-deleted case sits exactly one octave below the fatal one at every width:

Seven widths, fourteen entries, and every one of them a power of two with no residue. The half-deleted count is 150 of 320 in all seven cases and the fully deleted count is 310 in all seven. A significand bit is worth one octave of within-block range, stated as a measurement rather than as an estimate.

And a factor of two for the entry that survives #

The threshold is one of two things a bit is buying, and the second is the ordinary reason to want precision: the entries that are not deleted are represented better. That number is read off the lower bar of every figure above, in the sorted ordering, where no entry is lost at any width and the error is the format’s own resolution rather than a report of what the outlier did.

Across the seven widths the median entry’s relative error reads 0.131, 0.0652, 0.0236, 0.0130, 0.00619, 0.00321 and 0.00160. The successive ratios are 2.01, 2.76, 1.82, 2.10, 1.93 and 2.01 — a halving per bit, with the scatter that a power-of-two scale forces, since rounding the shared exponent up wastes between nothing and a whole bit depending on where the block maximum happens to sit relative to a power of two.

So both of the things a bit buys are worth a factor of two, in different currencies: one doubles the outlier the block tolerates, the other halves the error of the typical survivor. The comparison that decides anything is against a repair costing no bits at all.

The given order’s error is a function of one ratio, exactly #

Before that, one measurement that says how completely the width is spent on the outlier.

The median entry’s relative error in the given order does not depend on the significand width and the outlier separately. It depends only on their ratio, and the dependence is not approximate. At an outlier of 2^b / 4 the median error reads 0.24381756355881012 at three bits with an outlier of 2, and reads the same seventeen digits at four bits with an outlier of 4, at five with 8, at six with 16, at seven with 32, at eight with 64 and at nine with 128. At 2^b / 2 it reads 0.9273859133654531 at all seven. At 2^b it is exactly 1 at all seven.

That is what it looks like when a format’s entire width is consumed by one entry. The step is 2^(k − b + 2) for an outlier of 2^k, so it is a function of k − b alone, and every ordinary entry’s error is a function of the step. Widening the format does not make the vector’s ordinary values better represented in any way that survives the outlier — it moves the whole picture one octave along and leaves it otherwise identical. The bits are not being spent on the data; they are being spent on the distance between the data and the one number sitting above it.

The repair that costs nothing, and does not care about the width #

Sorting the same 320 values by magnitude changes no value, invents no numerical technique and leaves every scale factor exactly what it was — a power of two, applied exactly. What it changes is which values share a block.

The result is on the lower half of every figure above and it is the same number in all of them: 22. Not 22 at six bits and 11 at seven; 22 at three bits, 22 at four, 22 at five, 22 at six, 22 at seven, 22 at eight and 22 at nine, whenever the given order deletes 310. When the given order deletes 150 or fewer, sorting deletes none at all.

Twenty-two is arithmetic rather than a residue nobody chased down. Sorted by magnitude, the ten outliers come first and fill a third of the opening block; the remaining twenty-two slots hold the twenty-two largest ordinary values, which run from 1.495 down to 1.435 and are quantised against a step of 4. Every one of them is lost. The nine blocks behind hold ordinary values only, their shared step is 0.0625 rather than 4, and their loss is zero — the per-block counts read 22, 0, 0, 0, 0, 0, 0, 0, 0, 0. Twenty-two is thirty-two minus ten, and ten is the number of outliers, and neither number has anything to do with the format’s width.

The mechanism is the quantity the earlier essay identified as the deciding one. The whole vector spans 6.99 octaves. One of its blocks, in the order given, spans the same 6.99 octaves, because every block contains an outlier. Sorted, nine of the ten blocks span 1.51 octaves and the tenth spans all of it. Sorting does not improve the format; it confines the range that defeats the format to one block out of ten, and the count of casualties is then a property of the block size.

What an octave costs when it is bought with bits #

The two repairs can now be priced against each other on the same vector, because the storage accounting for a block format is one division: a b-bit significand with an eight-bit scale over thirty-two values costs b + 0.25 bits a value.

Take the outlier at 64, which is the case the earlier essay measured at six bits, and widen the format until the loss goes away. At three, four, five and six bits the given order deletes 310. At seven bits it deletes 150. At eight bits it deletes none, and at nine none. So the cure is two extra bits, taking the storage from 6.25 to 8.25 bits a value — 32% more memory for the whole vector, to survive one entry in each block of thirty-two.

Sorting, on the same vector at six bits, takes the count from 310 to 22 for nothing.

And the two do not scale alike. The purchased octaves are consumed one per bit by whatever the data’s largest ratio happens to be, so an outlier at 256 defeats the eight-bit format that had just been paid for, exactly as an outlier at 64 defeated the six-bit one; the hero drawing at the top of this page is that case. The free repair does not move at all: 22, at every width, against every outlier large enough to matter. One of these is a rate and the other is a constant, and no amount of the first reaches the second.

That is the sense in which a wider format is the wrong answer to this failure rather than a partial one. It is a real improvement, priced correctly at one octave a bit, applied to a problem whose size is set by the data and not by the format. A format’s width is measured in bits and a tensor’s dynamic range is not, and the two only look commensurable because both can be written as powers of two.

The same rate, on data with no outlier in it #

An outlier vector is one shape, and a law measured on one shape is a law about that shape. The other standard case is log-uniform data, spread over a chosen number of octaves inside each block, which is what the comparison against a per-element format is usually run on.

The narrow format loses this comparison everywhere on the sampled range, which is the result the earlier essay records at four bits. Widening it by four bits reverses the verdict entirely.

The interesting number is neither curve but the octave at which the first entry is deleted, since that is the same event the outlier data measures. Swept at quarter-octave resolution, the first deletion arrives at 2.25 octaves at three bits, 3.25 at four, 4.25 at five, 5.25 at six, 6.25 at seven and 7.25 at eight. Six consecutive widths, one octave apiece, on data that shares nothing with the outlier vector but the block size.

So the rate is the same on both shapes, and it is the shape rather than the rate that decides whether a block format is usable. On log-uniform data the format degrades: entries near the bottom of the range go first and the count climbs from 6 at eight octaves to 304 at sixteen. On outlier data it does not degrade at all. It is exact, then half destroyed, then wholly destroyed, across a factor of four in one parameter, which is why the numbers below the smallest one is the closer relative of this failure than any gradual loss of precision is.

Where the law stops being clean #

Three places, and all three were found by looking rather than by deriving.

The ninth bit buys two octaves rather than one. The first-deletion sweep reads 7.25 octaves at eight bits and 9.25 at nine, not 8.25. Nothing about the format changed; the vector did. Each spread is a fresh stretch of the same draw, the first deletion is decided by the single unluckiest entry among 640, and the ceiling in the scale computation wastes a variable fraction of a bit depending on where each block’s maximum falls relative to a power of two — measured here at 0.82 to 0.86 of the integer range. One octave a bit is the rate; a quarter of an octave of scatter around it is the instrument.

A different quantile gives a different rate, correctly. The octave at which half the vector is deleted moves 4.25, 7.5, 9.75, 12.25, 14.5, 16.75 and 18.75 across the seven widths — about 2.25 octaves a bit, not one. That is not a contradiction. Log-uniform entries spread over a range have half of them in the top half of it, so pushing the deletion line past the median takes twice the range it takes to push it past the smallest entry. The two rates are the same law read at two quantiles, and quoting either as the rate without saying which entry it is about is the mistake a small residual is not a small error is about in another field.

The per-element format is not a safe harbour either. It keeps every entry in every one of these runs, in either order, which is the whole of its advantage. But at an outlier of 512 the outlier itself is past E4M3’s largest finite value of 448 and saturates, at a relative error of 0.125 on the one entry the block format also represents well. A format with its own exponent per value still has an exponent range to run out of, and the entry it loses is the one carrying the magnitude.

What follows for anything that uses this #

A width is a specification and a range is a measurement. Choosing between a six-bit and an eight-bit block format is choosing between two octaves and 32% of the memory, and the choice is only answerable once the largest within-block ratio in the data is known. That number is cheap — one pass, one maximum and one minimum per block — and it is not in any format’s data sheet. The units the matrix is measured in is the same quantity arriving from a different direction, as a choice nobody thought was numerical.

The ordering is worth more than the bits and is usually not available. Sorting is the best possible grouping and cannot be done, because the block boundaries have to match how the hardware reads memory. What is available is choosing which axis the blocks run along, and that recovers most of it when the magnitudes vary one way. The general principle is the order decides the memory’s, applied to accuracy instead of to fill: a permutation that changes nothing about the values changes what the computation costs, and here it changes what the computation keeps.

An ordering chosen before the numbers are looked at is not the same thing. The sorted order here is chosen from the magnitudes, which is exactly what an ordering that does not wait for the numbers declines to do in a sparse factorisation for reasons of cost. Here the cost of looking is a sort and the benefit is 288 entries, so the trade runs the other way — where the layout is free to choose at all.

A deleted entry is not a rounded one. Every number in the middle column of the tables above is a count of entries with a relative error of exactly 1, which no error analysis phrased in terms of unit roundoff describes. It is nearer to what cancellation takes the answer measures — a quantity that is gone rather than approximate — and a solver fed such a vector is solving a different problem, not the same problem slightly perturbed.

Storage saved on the wrong axis is not saved. A block format at 6.25 bits a value against a per-element format at 8 is a 22% saving that becomes a total loss on one shape of data and a clear win on another, and the same reasoning applies to any accuracy request read as a width, which a rank that is a number of digits measures for a very different object. What is bought with bits should be priced in the units the failure is measured in.

And scaling does not rescue it. Multiplying the whole vector by a constant changes every shared exponent and no ratio, so it moves nothing here at all — the same limit a condition number scaling cannot move reports for conditioning. What a shared exponent charges for is the ratio inside the block, and a ratio is what a uniform scaling is guaranteed to preserve.

What is asserted, and what refuses #

The threshold is 2^b at every width from three to nine, with the half-deleted case at 2^(b−1) in every case as well, and both are counted rather than estimated: 310 of 320 and 150 of 320, seven times each.

The sorted count is 22 at every width, and 22 is thirty-two minus ten, which is the block size minus the number of outliers and contains no reference to the significand.

The median entry’s error halves per bit, from 0.131 at three bits to 0.00160 at nine, with the scatter that a power-of-two shared scale produces reported rather than smoothed.

The given order’s error is a function of outlier / 2^b alone, asserted to seventeen digits across seven widths at three different ratios.

And the claim that the wider format is the more accurate one is refused. A six-bit block format at 6.25 bits a value is compared against a per-element format at 8 on data spanning one octave inside a block, the narrower format wins, and the assertion that it cannot is fed exactly that comparison and must reject it. The measurement that the wide-spread comparison produces is real; what it is evidence about is the range inside the block, and reading it as evidence about the bit count is what this page refuses. Where the hardware went records what these formats were built to make cheap, and it was never the dynamic range.