Numerical Linear Algebra
where the algebra stops being the arithmetic
Concept

Block floating-point — where it appears

A format in which a group of values share one exponent and each keeps a small signed integer of its own. The shared scale is set by the largest magnitude in the group, so the cost per value is the integer plus the scale divided by the block size, and every other entry is stored as a multiple of a step chosen for the maximum.

Named by 2 essays across one field — each of them below, with the objects they name alongside it.

Also named here as outliers, quantisation, shared exponent — the same set of essays touches all of them, so they are one junction rather than several.

half a stepthe 32 values of one block, in the order they arrivestep 0.0625one scale, thirty-two valuesoctaves inside the block1.9entries rounded to zero0worst error over its bound131 levels either side of zerothe largest entry chose the step

One exponent for thirty-two numbers

Share the exponent across a block and the cost per value drops from eight bits to 6.25, and the accuracy improves — up to about three octaves of spread inside a block. Past that a single outlier deletes the thirty-one values beside it, and the 2-norm barely notices.

arithmetic · Block formats entries rounded to zero, of 320block, as given310block, sorted by size22E4M3, either order0median entry's relative errorblock, as given1block, sorted0.00321E4M30.022the same numbers, three waysdeleted, as given310deleted, sorted22deleted, per-element08-bit significands, blocks of 32sorting is free and changes no value

A bit buys an octave

The outlier a block survives is exactly two raised to its significand width — 8 at three bits, 32 at five, 128 at seven, 512 at nine. Each extra bit doubles the range the block tolerates and halves the ordinary entry's error. Reordering the same numbers buys every octave at once and costs nothing.

arithmetic · Block formats

Named alongside it

The objects these essays reach for when they reach for this one.

Dynamic rangeFp8OutliersQuantisationShared exponentSignificandExponent rangePermutationRelative error

All concepts