The other half of a format

What a float can hold opens this site with the spacing of the representable numbers and the observation that it doubles at every power of two. Every essay since has treated “precision” as meaning the significand, and lib/float.js has simulated the significand and nothing else — each operation performed in double and rounded to a chosen number of significant bits, exactly as IEEE 754 rounds.

That was written down as a limitation on the day it was built, in the file’s own docstring: no overflow to infinity at 10³⁸, no gradual underflow, no subnormals. Everything this site claims is about precision, and precision is what is here.

The limitation had a consequence that grew rather than staying still, and the consequence is a false statement.

The false statement #

lib/mixed.js carries a table of the hardware formats and it is indexed by mantissa bits:

bfloat16: 8 · fp16: 11 · tf32: 11 · fp32: 24 · fp64: 53

Read that table for what it is and it says fp16 and tf32 are the same format. Everything downstream agreed. Where the hardware went reports “fp16/tf32 κ·u = 0.488” as one row, because at eleven bits they compute the same thing — and that row was correct about the computation it described and wrong about the formats it named.

The two are not the same format. tf32’s largest finite number is 3.4·10³⁸ and fp16’s is 65,504, which is thirty-three decades apart, and the whole reason tf32 exists is that difference. It was introduced so that a value which fits in single precision cannot overflow when it is narrowed for a matrix multiplication, and the price paid for it is thirteen significand bits.

So the maturity phase’s job here was not to add a feature. It was to remove a claim the site was making by omission.

A format is two numbers #

An IEEE-style binary format is specified by two integers: the width of the exponent field e, and the precision p — the number of significand bits including the implicit leading one. Everything else follows from those two and from the standard’s rules.

The bias is 2^(e−1) − 1. The top exponent code is reserved for infinities and NaNs, and the bottom one for zero and the subnormals. So the largest finite number is 2^(emax)·(2 − 2^(1−p)) with emax the bias, the smallest normal is 2^(1−bias), and the smallest subnormal is that divided by 2^(p−1).

Those formulas reproduce the numbers everybody knows, which is the check that entitles this site to say “fp16” rather than “an 11-bit simulation”:

65,504 and 3.4028…·10³⁸ are published constants that came from nowhere near this file, and a formula off by one in the bias reproduces neither. The site checks both, and checks binary32’s largest finite number against Math.fround — the hardware agreeing that the number computed from the formula is representable.

The budget, and why the trade is not symmetric #

The figure is the design decision. A 16-bit format has fifteen bits to split, and every bit given to the exponent is taken from the significand.

What makes the choice interesting is that the two curves have different shapes. A bit of exponent doubles the range, because the range is exponential in the exponent width. A bit of significand adds about a third of a decimal digit, because precision is linear in it.

So the trade is wildly asymmetric, and it explains bfloat16 completely. Going from fp16’s five exponent bits to bfloat16’s eight takes the span from about twelve decades to about seventy-eight — three doublings, one per bit — and costs three significand bits, which is a single decimal digit. For a workload where values span many orders of magnitude and each individual value need only be roughly right, that is not a close call.

Both halves of the trade are exact enough to state as laws. A significand bit is worth log₁₀2 = 0.301 of a decimal digit, which is the “about a third” above to three figures. And the exponent bit’s doubling is a limit rather than an identity — the decades of normal range at p = 11 run

   e = 3    1.81        e = 7    37.93
   e = 4    4.21        e = 8    76.46
   e = 5    9.03        e = 9   153.53
   e = 6   18.66        e = 10  307.65

with successive ratios 2.33, 2.14, 2.07, 2.03, 2.02, 2.01, 2.00, converging to exactly two from above. The excess at the narrow end is the two reserved exponent codes, which are a fixed cost and matter proportionally more when there are few of them.

It is worth being careful about which quantity doubles, because three different things could be called “the range” and they grow at three different rates. Between fp16 and bfloat16 the largest finite number grows by a factor of 2¹¹², the ratio of largest to smallest normal by 2²²⁴, and the span measured in decades by 8.5. Only the last of the three is what “doubles per bit” is about, and it is the one worth carrying, because a format’s usefulness is a question about how many orders of magnitude it covers rather than about how large its largest number is.

Why that distinction is not pedantry #

Three quantities growing at 2¹¹², 2²²⁴ and 8.5 across the same three bits is a wide enough spread that picking the wrong one changes the argument, and the argument is the one the essay is making.

The 8.5 is what a workload cares about. A model whose activations span twenty orders of magnitude fits in bfloat16 and does not fit in fp16, and the question is how many orders each covers — twelve against seventy-eight. Doubling per bit is a strong statement in that currency: three bits is the difference between a format that covers a physicist’s range and one that does not.

The 2¹¹² is what an overflow cares about, and it is the number to quote when the question is how far away is infinity. A computation whose intermediates reach 10²⁰ overflows fp16 at 6.6·10⁴ and has eighteen orders of headroom in bfloat16, and the ratio of the two ceilings is what says so.

The 2²²⁴ is not what anything cares about. It is a ratio of ratios, and no computation is sensitive to it — but it is the number a reader gets by dividing the two “range” columns of the format table, which is exactly how a wrong factor gets quoted.

That is the general form of the caution. Where a table has several columns that could be called by one word, a factor computed from it needs the column named, and this essay’s own table has three such columns side by side.

It also explains why nobody builds a 16-bit format with twelve exponent bits. The extra range is free in the sense that it costs bits, and useless in the sense that no computation needs 10⁶⁰⁰ — and the three significand bits it costs take the format below two decimal digits.

Nineteen bits is the interesting stop because it is the only width in the table nobody inherited — somebody sat down and chose it, and chose to spend the extra three bits on precision rather than on range:

Three of those five formats have the same range, and they span a factor of two in total width. bfloat16 at sixteen bits, tf32 at nineteen and binary32 at thirty-two all carry 8 exponent bits and all reach 3.4·10³⁸. Going from the first to the last triples the significand — 8 bits to 24 — and does not move the largest representable number by a digit. The exponent field of single precision is a sixteen-bit-sized decision that nothing since has revisited, which is the sentence the thirty-two-bit frame above is making and the nineteen-bit frame confirms from the other side: tf32 was designed from scratch in 2018 and chose the same eight.

And the two sixteen-bit formats differ by thirty-four orders of magnitude. bfloat16 reaches 3.39·10³⁸ and fp16 reaches 6.55·10⁴, on the same sixteen bits and the same silicon budget, because one spent three of them on the exponent and the other on the significand. That is the essay’s whole subject in one row of the table: the half of a format nobody quotes decides whether a computation overflows, and the half everybody quotes decides only how accurate the answers that do not overflow are.

Only binary64 buys more range, and it buys a great deal — three more exponent bits taking the ceiling from 10³⁸ to 10³⁰⁸. Exponent bits are worth a doubling of the exponent each, so the returns there are enormous and the returns on significand bits are one bit of accuracy each. That asymmetry is why the trade is not a matter of taste: the two halves of the budget are not priced in the same units, and a format that runs out of range fails completely where a format that runs out of precision degrades.

What the extension does, and what it must not do #

arith(bits) now takes an optional exponent width. With it, the arithmetic context overflows to infinity, underflows gradually, and has a largest and a smallest number. Without it, the behaviour is what it was.

That last clause is the whole engineering constraint. Two phases of figures were drawn against arith(bits) and none of them may move — a change that silently redrew 225 existing figures would be exactly the failure ../CONSOLIDATION.md records about a bad closeGaps breaking 181 figures across four sites.

So it is asserted rather than assumed. Eighteen thousand random values inside fp16’s normal range are rounded by both contexts and required to be bit-identical, and the two are required to differ in exactly the way a format does outside it: 300² is 89,984 without a range, at eleven bits, and is an infinity in fp16.

The default is expBits: null, meaning an unbounded exponent, and the site refuses to let that be described as a format — the assertion that an unbounded context has a range is fed to the build and rejected, because having no largest number is precisely what it was missing.

Where the difference shows up #

Three places, and each gets an essay or a section — starting with what a float can hold.

Overflow, which is where a computation leaves the top of the range. This is the fp16-against-tf32 case and it is a norm that overflows before it is a norm: the naive Euclidean norm squares before it adds, which doubles the exponent, so it works over only about half of a format’s range. In fp16 a vector of sixteen entries each equal to 1,000 has a norm of 4,000 — comfortably representable — and the naive expression returns an infinity. In tf32, at the same precision, it returns 4,000.

Underflow, which is the bottom, and it is a different story with a different lesson. That is the numbers below the smallest one.

And the ranking of the formats. Where the hardware went ranks the five by mantissa bits and concludes that bfloat16 is the least accurate. It is, and it is also the only 16-bit format on which a great many computations complete at all — which is a fact about a different axis and which the ranking cannot express.

Why the two axes are not interchangeable #

A reader could reasonably ask why this needs its own axis at all. Both range and precision are ways of failing to represent a number, and a computation that overflows and a computation that rounds have both lost information. The answer is that they fail differently, and the difference is the reason overflow gets a separate treatment everywhere in this subject.

Rounding is bounded and relative. Every operation returns a result within a factor of 1 + u of the exact one. That is what makes backward error analysis possible: a whole sequence of operations can be described as the exact computation on slightly perturbed data, and the perturbation is proportional to the data. Every claim in the error field rests on it.

Overflow is neither. The result is not within any factor of the exact one — it is an infinity, so the relative error is infinite. And infinity is absorbing: infinity minus infinity is a NaN, and a NaN propagates through everything it touches. So a single overflow anywhere in a long computation does not degrade the answer, it destroys it, and there is no analysis that recovers from it.

That asymmetry is why the standard treats them differently. Rounding happens silently on essentially every operation and no sane system reports it. Overflow raises a flag, because it is an event rather than a background condition.

And it is why the fix is different in kind. There is no way to compute more accurately than the format allows; the only response to rounding is a longer significand or a better-conditioned algorithm. But overflow can always be avoided by scaling, at the cost of one extra pass over the data — which is what the next essay’s scaled norm does and what every serious library routine does internally. A failure that can be engineered around and a failure that cannot are different kinds of failure, and collapsing them into “the format was not good enough” loses the only actionable half.

There is one more consequence and it is the practical one. Because rounding is relative, the natural way to think about precision is in significant digits, and the natural way to compare formats is by unit roundoff — which is what every essay on this site did before this one. Because overflow is absolute, the natural way to think about range is in decades of magnitude, and comparing formats by it means comparing 12 against 78. The two comparisons produce different orderings of the same five formats, and there is no single number that reconciles them, which is exactly why the figure above draws two curves and not one.

What is asserted here #

The format constants are the hardware’s. 65,504, 2⁻¹⁴, 2⁻²⁴ for fp16; 3.4028…·10³⁸ and 2⁻¹²⁶ for binary32, with the hardware’s own fround confirming the largest finite number is representable; and binary64’s largest and smallest against the language’s own constants.

fp16 and tf32 have exactly the same unit roundoff, asserted as an equality, and ranges more than 10³³ apart — the pairing this whole extension exists for, stated as two assertions rather than as a sentence.

bfloat16 is the less precise of the two 16-bit formats and the one that cannot overflow where single precision does not. Both halves, because the first alone is the ranking that was already there and the second is what it was missing.

Bounding the exponent changes nothing inside the range, over eighteen thousand values, bit for bit.

The refusals: that fp16 and tf32 compute the same thing, fed a norm that is finite in one and infinite in the other; that overflow is a large rounding error, fed 300² in fp16; and that an unbounded context has a range.

A note on what is still missing #

Three things, and naming them is cheaper than discovering them later.

The fp8 formats are not here. E5M2 follows IEEE’s rules and would fit the machinery exactly; E4M3 does not — it has no infinities and extends its exponent range to reach 448, which the formulas above do not produce. Adding it would mean either special-casing it or reporting a number that is not the one on the hardware, and reporting the wrong number quietly is the failure this site is organised against.

Rounding modes are not simulated. Everything here rounds to nearest, ties to even. Round-toward- zero and the directed modes exist, they change results, and stochastic rounding — which is genuinely used in low-precision training — changes them in a way that matters for exactly the accumulation problems the order they are added in is about.

And the block formats are not here at all. The formats current accelerators are moving to share one exponent across a block of values, which is neither a scalar format nor an array of them, and the whole notion of “the range of the format” needs restating for them. That is a subject rather than an omission.

What is here is the pair of numbers that specifies an ordinary IEEE format, the five formats that pair produces, and the two failures — off the top and off the bottom — that a significand-only simulation could not express.

What this changes about the site’s own claims #

Two of them, and both are corrections rather than additions.

The mixed-precision field’s ranking is now incomplete rather than wrong. Where the hardware went measures iterative refinement at three formats and finds the threshold at κ·u = 1, with bfloat16 failing first because its unit roundoff is largest. Every number in it is correct. What it could not say is that the same experiment run on data spanning more than twelve decades would not produce three results and one failure — it would produce two results, one failure, and one infinity, and the infinity would belong to the format with the second best precision of the three.

And the precision slider means something narrower than it appeared to. Nine figures on this site put the mantissa on a slider and watch a quantity slide with it, which is the site’s precision as the knob thread and is exactly as sound as it ever was — those figures are about the significand and they measure the significand. What they cannot show is a computation failing rather than degrading, and a reader who took the slider as a model of “using a smaller format” was being given half the model. The formats on the slider have no bottom and no top.

Nothing in the existing figures moves. The extension is opt-in, the eighteen thousand-value check above is what guarantees that, and the figures that want the other axis are the three in this essay’s field. But the claim the site was implicitly making — that narrowing a format is a matter of losing digits — was too small, and the correction is the phase’s own subject: a measurement is about the axis it was taken along, and an axis nobody measured is a claim nobody checked.

A quantity that uses all of it #

The exponent range is the half of a format nobody budgets for, and there is one ordinary quantity that consumes all of it at ordinary sizes.