Numerical Linear Algebra
where the algebra stops being the arithmetic
Generator

Precision against range, with the 4-bit-exponent formats marked

One function in the lowprec library, called 5 times across 1 essay. Below: what it draws at its defaults, what it draws at every value an essay asks for, the 7 claims it put to the test while drawing them, and where it stands against the rule this site is named for.

At its defaults it draws precision against range, with the 4-bit-exponent formats marked. A scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.

format-plane is one function in lib/figures/lowprec.js — low precision — eight bits, the format that breaks the rules, and one shared exponent. Everything below came out of it during this build, at arguments taken from the essays rather than invented for this page. A figure here is the figure a reader meets in an essay, and if the generator changes, this page changes with it.

At its defaults

Drawn even though every essay passes arguments — which on this site is every essay, at 100% of placements since the standard pass. A default nothing exercises is a trap for the next essay to call this with none, and this is the page where a default that has drifted from the figures around it becomes visible.

Precision against range, with the 4-bit-exponent formats markedA scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.10⁻¹⁶10⁻¹²10⁻⁸10⁻⁴10²10⁸10¹⁴10²⁰10²⁶10³²10³⁸unit roundofflargest finite valueE4M3E5M2bfloat16fp16tf32binary32binary64240 — what IEEE's rules would givethe octave the rule-breaking boughtE4M3 largest finite448under IEEE's rules240E5M2 largest finite5.7·10⁴seven bits and a sign, spent two waysno infinities in one of them

A scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.

expBits: 4

The arguments are the ones Eight bits, and a format that breaks the rules passes. A value nobody placed would be a picture no essay asked for and no claim was ever checked against.

Precision against range, with the 4-bit-exponent formats markedA scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.10⁻¹⁶10⁻¹²10⁻⁸10⁻⁴10²10⁸10¹⁴10²⁰10²⁶10³²10³⁸unit roundofflargest finite valueE4M3E5M2bfloat16fp16tf32binary32binary64240 — what IEEE's rules would givethe octave the rule-breaking boughtE4M3 largest finite448under IEEE's rules240E5M2 largest finite5.7·10⁴seven bits and a sign, spent two waysno infinities in one of them

A scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.

expBits: 5

The arguments are the ones Eight bits, and a format that breaks the rules passes. A value nobody placed would be a picture no essay asked for and no claim was ever checked against.

Precision against range, with the 5-bit-exponent formats markedA scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.10⁻¹⁶10⁻¹²10⁻⁸10⁻⁴10²10⁸10¹⁴10²⁰10²⁶10³²10³⁸unit roundofflargest finite valueE4M3E5M2bfloat16fp16tf32binary32binary64240 — what IEEE's rules would givethe octave the rule-breaking boughtE4M3 largest finite448under IEEE's rules240E5M2 largest finite5.7·10⁴seven bits and a sign, spent two waysno infinities in one of them

A scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.

expBits: 8

The arguments are the ones Eight bits, and a format that breaks the rules passes. A value nobody placed would be a picture no essay asked for and no claim was ever checked against.

Precision against range, with the 8-bit-exponent formats markedA scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.10⁻¹⁶10⁻¹²10⁻⁸10⁻⁴10²10⁸10¹⁴10²⁰10²⁶10³²10³⁸unit roundofflargest finite valueE4M3E5M2bfloat16fp16tf32binary32binary64240 — what IEEE's rules would givethe octave the rule-breaking boughtE4M3 largest finite448under IEEE's rules240E5M2 largest finite5.7·10⁴seven bits and a sign, spent two waysno infinities in one of them

A scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.

expBits: 11

The arguments are the ones Eight bits, and a format that breaks the rules passes. A value nobody placed would be a picture no essay asked for and no claim was ever checked against.

Precision against range, with the 11-bit-exponent formats markedA scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.10⁻¹⁶10⁻¹²10⁻⁸10⁻⁴10²10⁸10¹⁴10²⁰10²⁶10³²10³⁸unit roundofflargest finite valueE4M3E5M2bfloat16fp16tf32binary32binary64240 — what IEEE's rules would givethe octave the rule-breaking boughtE4M3 largest finite448under IEEE's rules240E5M2 largest finite5.7·10⁴seven bits and a sign, spent two waysno infinities in one of them

A scatter of formats with unit roundoff across and largest finite value up, both logarithmic. The two eight-bit formats sit at the lower left, joined to each other and to a marked position below one of them.

What it checked while drawing

Every figure above checked its own claims on the way to being drawn, and a claim that failed would have stopped the picture rather than shipped a wrong one. Those checks used to leave no trace at all: a passing one returned true and the only evidence the figure had checked anything was that nothing crashed. The list below is what they actually said, collected by running this generator with an observer installed — not a description of what it is believed to check.

7 distinct claims across 5 sets of arguments, grouped below by shape — because most of them are one sentence with a different number in it, and how many separate times that sentence was put to the test is the informative part.

an exponent width one of these formats uses

and E5M2 reaches 57344

and its range is 128 times wider

E4M3 reaches 448

E5M2's steps are twice as coarse

some format uses that exponent width

under IEEE's rules the same bits would reach only 240

Against the rule

The rule does not apply to it. It factorises nothing, so there is no residual it could be withholding. That is worth stating rather than leaving blank: a site that reported the rule as satisfied by every generator would be counting mostly generators the rule never reached.

Across the library: the rule bites on 217 of 397 generators — 199 print a residual and 18 are exempt with a published reason; 180 factorise nothing. Read from lib/residual-rule.js, which is the same body the gate enforces from, and the gate's last check fails the build if this page and it disagree about any generator.

Where it is called

Changing this generator changes every figure on this list. That is what makes the list worth publishing rather than keeping in a check script.

The arithmetic underneath

Eight bits, and a format that breaks the rules

E4M3 reuses the exponent code IEEE reserves for infinities, so it reaches 448 where the same bits under IEEE's rules would reach 240 — and has no infinity left to signal an overflow with. The same computation is a NaN on one conforming device and 448 on another.

The whole library · All essays · What must fail