Numerical Linear Algebra
where the algebra stops being the arithmetic
Concept

Significand — where it appears

The digits of a floating-point number, as distinct from its exponent. Their count sets the relative accuracy of every operation, and it is the one parameter on this site that is genuinely a knob.

Named by 14 essays across 2 fields — each of them below, with the objects they name alongside it.

[½, 1)[1, 2)[2, 4)0.5124gap 0.125gap 0.25 — twice as wide8 values per octavespacing doubles at each power of two

What a float can hold

The representable numbers are not a fine fuzz spread evenly over the line. They are evenly spaced inside each power-of-two interval and twice as far apart in the next one up, and almost everything else in this subject is a consequence of that one fact.

arithmetic · Floating point 10¹10²10³10⁴10⁵10⁶10⁷10⁸10⁹10¹⁰110⁴10⁸10¹²10¹⁶condition number κ(A)× short of a double solvebf16fp16fp32bf16fp16fp32the reference is soundreference solve, worst backward error10⁻¹⁶bfloat16 threshold κ256fp32 threshold κ1.7·10⁷eight refinement steps, residual always in doubleflat at 1 means it reached double

Where the hardware went

bfloat16 carries eight mantissa bits, which puts its refinement threshold at a condition number of 256. That is not an exotic matrix. It is an ordinary one, and past it the method still improves the answer by a factor of four hundred while getting nowhere near a usable one.

arithmetic · Mixed-precision 3456789101100.250.50.751bits in the exponent fieldeach curve as a fraction of its own maximumbfloat16fp16rangeprecisionto 617 decadesto 3.9 digitswhat the split buysbfloat16: largest number3.4·10³⁸fp16: largest number6.6·10⁴a bit of exponent doubles the rangea bit of significand adds a third of a digit

The other half of a format

fp16 and tf32 have the same eleven significand bits and their largest numbers are 65,504 and 3.4·10³⁸. For two phases this site simulated the significand alone, so it was obliged to report them as the same format — which is a claim, and a false one.

arithmetic · Floating point -47-37-27-17-7313233301234log₁₀ of the vector's normfp1611 bitsbfloat168 bitstf3211 bitsbinary3224 bitspale: the format's range · blue: √(Σ(xᵢ/m)²)·m · red: √(Σxᵢ²)fp16 and tf32 have the same eleven significand bitsand their bars do not overlap

A norm that overflows before it is a norm

The vector of sixteen thousands has a Euclidean norm of 4,000, which fp16 represents exactly. Written as the square root of the sum of squares it returns infinity, because squaring doubles the exponent — and the expression costs half the format's range on the one computation every iterative method performs at every step.

arithmetic · Overflow 10⁻⁷10⁻⁵10⁻³10⁻¹⁰10⁻⁸10⁻⁶magnitudespacing to the next numberthe smallest normalgradualflush to zerosmallest normal6.1·10⁻⁵smallest subnormal6·10⁻⁸octaves of subnormals10pairs that lie under FTZ10the spacing stops halving and stays putwhich is what makes x − y = 0 mean x = y

The numbers below the smallest one

Below the smallest normal number the spacing stops halving and stays put, all the way to zero. That is what gradual underflow is, and the thing it buys is the sentence every algorithm assumes without being told — x minus y is zero only when x equals y.

arithmetic · Subnormals 2022242628303210⁻¹⁷10⁻¹⁴10⁻¹¹10⁻⁸10⁻⁵10⁻²k, where the entries are near 2ᵏrelative error of the determinantas writtenfused: exactproducts need 54 bitsone rounding, the whole answertrue determinant1naive, k = 300fused, k = 301first wrong at k27sizes returning 06both forms conformand the source does not say which

One multiply the compiler removed

A determinant whose value is exactly 1, computed as exactly 0 by the expression that is written down, and exactly 1 by the same expression with the multiply and the add fused. Both forms conform to IEEE-754, both are legal compilations of the same source, and nothing in the program says which one you have.

machine · Fma contraction 10⁻³10⁻¹10¹10³01magnitude448NaN — no ∞0.0156 — the smallest normaldrawn from the format's own rulespositive finite values126largest finite value448worst round-trip error0the subnormals are the evenly spaced ticks at the lefteverything a byte can be

Eight bits, and a format that breaks the rules

E4M3 reuses the exponent code IEEE reserves for infinities, so it reaches 448 where the same bits under IEEE's rules would reach 240 — and has no infinity left to signal an overflow with. The same computation is a NaN on one conforming device and 448 on another.

arithmetic · Floating point half a stepthe 32 values of one block, in the order they arrivestep 0.0625one scale, thirty-two valuesoctaves inside the block1.9entries rounded to zero0worst error over its bound131 levels either side of zerothe largest entry chose the step

One exponent for thirty-two numbers

Share the exponent across a block and the cost per value drops from eight bits to 6.25, and the accuracy improves — up to about three octaves of spread inside a block. Past that a single outlier deletes the thirty-one values beside it, and the 2-norm barely notices.

arithmetic · Block formats entries rounded to zero, of 320block, as given310block, sorted by size22E4M3, either order0median entry's relative errorblock, as given1block, sorted0.00321E4M30.022the same numbers, three waysdeleted, as given310deleted, sorted22deleted, per-element08-bit significands, blocks of 32sorting is free and changes no value

A bit buys an octave

The outlier a block survives is exactly two raised to its significand width — 8 at three bits, 32 at five, 128 at seven, 512 at nine. Each extra bit doubles the range the block tolerates and halves the ordinary entry's error. Reordering the same numbers buys every octave at once and costs nothing.

arithmetic · Block formats 0510152025303540first barrier depth at which the far well is wrong by half, octavesfp16the band adds 11 octaves · starting at the top adds 15E4M3the band adds 2 octaves · starting at the top adds 10E5M2the band adds 3 octaves · starting at the top adds 14flush to zero13gradual underflow24flush, started at the top29gradual, started at the top39flush to zero5gradual underflow7flush, started at the top13gradual, started at the top17flush to zero13gradual underflow16flush, started at the top27gradual, started at the top30dot: median of nine chains · bar: their rangethe headroom is worth more than the band

The far well in a byte

Half precision's subnormals carried a two-well chain's far well ten octaves deeper than flush to zero, and the essay that measured it predicted that on the eight-bit formats the band would be worth almost nothing and starting high almost everything. Measured on nine chains, the band is worth what it always was, two or three octaves; starting at the top of the range is worth ten and fourteen; and at three significant bits the far well is lost on eight chains of nine with no exponent limit at all, because the precision fails before the range does.

arithmetic · Subnormals 0102030405060first barrier depth at which the far well is wrong by half, octavesfp16E4M3E5M2gradual underflow, from onegradual, from the topcounter, floor and ceiling9 of 9counter, window from the rates9 of 9gradual underflow, from onegradual, from the topcounter, floor and ceiling1 of 9counter, window from the rates8 of 9gradual underflow, from onegradual, from the topcounter, floor and ceiling1 of 9counter, window from the rates1 of 9ring: chains that keep the far well past sixty octavesthe window is the unbounded exponent

Room for the step, not the value

Starting a recurrence at the top of an eight-bit format's range spends its headroom once. An exponent counter beside the stored value was predicted to spend it at every step, remove every range failure and leave only the precision. Rescaled on the value — back to one whenever it nears the bottom or the top — it does not: E4M3 still loses its far well on seven or more chains of nine, because one climbing step multiplies the value by forty and E4M3 clips silently at 448. Rescaled into a window read off the chain's own rates, it reproduces the unbounded exponent bit for bit on all 531 runs of every format, and what is left of E4M3's limit is nothing.

arithmetic · Subnormals 0.010.1110100the earlier recipe, product rounded41 of 41 over · median 62.0the same recurrence, exact24 of 41 over · median 6.3a well-mixed generator21 of 41 over · median 8.0one-pass error ÷ κ²u, one dot a seed; bar: medianfilled: the variance over-estimated; open: underthe line at one is the digits-lost rulethe exact recurrence does not lean

The generator that rounded its own product

A one-pass variance leaned the same way on forty-one seeds, and the low bits of a congruential generator were the suspect. Replacing the lowest five bits made the lean larger, not smaller. The generator was not congruential at all: written in double precision, its product of two 31-bit numbers was rounded before the modulus, its states collapsed onto a handful of low bytes, and those are the bits a running sum near a billion throws away. The exact recurrence does not lean, and random bits are not neutral either: fewer of them than the sum discards all round one way.

arithmetic · Cancellation 020406080100120barrier depth, octaves32 statesa rate is zero from 9140 statesa rate is zero from 11548 states56 states64 stateswindow under one octavewindow width below zerothe window rule first failsthe aiming rule first failsthe window per step: exacta stored rate is zeroE4M3, five chain lengths, every depth from 2 to 120a window needs an octave to hold a power of two

An octave to land in

A window read off a chain's own rates kept an eight-bit recurrence exact to sixty octaves, and its width, falling towards zero, was predicted to mark where it would stop being exact, to within an octave. It stops two to seven octaves earlier, as soon as the window is narrower than the octave a power-of-two rescale moves by. The rule that aims every step, predicted to outlast it, fails too, once the value it aims at no longer fits. A window computed for each step separately never narrows below 4.8 octaves, reproduces the unbounded exponent on every depth at which the format can still store the chain, and makes the fewest rescales of the three.

arithmetic · Subnormals blocks of 232 states×40 states×48 states×56 states×64 states×blocks of 432 states×40 states×48 states×56 states×64 states×blocks of 832 states×40 states×48 states×56 states×64 states×020406080100120barrier depth, octaves×the predicted lineblock window under an octaveblock window emptyfirst stops being exactexacta stored rate is zeroE4M3, every depth from 2 to 120the block fails where its window empties

Twice the block, half the depth

An eight-bit chain kept exact by a window per step was predicted to keep exactness with one exponent shared by two states, and to lose it with four from the depth where four of its steepest steps span more than the narrowest window, 4.8 octaves. Blocks of two fail on four chains of five, at 66 to 114 octaves. Blocks of four fail at 35 to 66, two to three times deeper than the predicted 13 to 26. Every block size fails within three octaves of the depth where its block's window empties, and that depth is 4.0 to 4.6 octaves for each state a block holds.

arithmetic · Subnormals

Named alongside it

The objects these essays reach for when they reach for this one.

Dynamic rangeOverflowExponent rangeFp8Unit roundoffHalf-precisionSubnormal numbersIEEE 754Stationary distributionbfloat16Catastrophic cancellationScaling

All concepts