Numerical Linear Algebra
where the algebra stops being the arithmetic
Concept

Exponent range — where it appears

3 essays name this object, across one field. What follows is each of them, and the objects they name alongside it.
3456789101100.250.50.751bits in the exponent fieldeach curve as a fraction of its own maximumbfloat16fp16rangeprecisionto 617 decadesto 3.9 digitswhat the split buysbfloat16: largest number3.4·10³⁸fp16: largest number6.6·10⁴a bit of exponent doubles the rangea bit of significand adds a third of a digit

The other half of a format

fp16 and tf32 have the same eleven significand bits and their largest numbers are 65,504 and 3.4·10³⁸. For two phases this site simulated the significand alone, so it was obliged to report them as the same format — which is a claim, and a false one.

arithmetic · floating point 10⁻³10⁻¹10¹10³01magnitude448NaN — no ∞0.0156 — the smallest normaldrawn from the format's own rulespositive finite values126largest finite value448worst round-trip error0the subnormals are the evenly spaced ticks at the lefteverything a byte can be

Eight bits, and a format that breaks the rules

E4M3 reuses the exponent code IEEE reserves for infinities, so it reaches 448 where the same bits under IEEE's rules would reach 240 — and has no infinity left to signal an overflow with. The same computation is a NaN on one conforming device and 448 on another.

arithmetic · floating point half a stepthe 32 values of one block, in the order they arrivestep 0.0625one scale, thirty-two valuesoctaves inside the block1.9entries rounded to zero0worst error over its bound131 levels either side of zerothe largest entry chose the step

One exponent for thirty-two numbers

Share the exponent across a block and the cost per value drops from eight bits to 6.25, and the accuracy improves — up to about three octaves of spread inside a block. Past that a single outlier deletes the thirty-one values beside it, and the 2-norm barely notices.

arithmetic · block formats

Named alongside it

The objects these essays reach for when they reach for this one.

SignificandDynamic rangeFp8Half precisionIEEE 754OverflowUnit roundoffbfloat16Block floating pointOutliersQuantisationSaturation

All concepts