The numbers below the smallest one

A norm that overflows before it is a norm is about the top of the range, where the failure is loud: the answer is an infinity, the infinity is absorbing, and everything downstream becomes a NaN. Nobody misses it.

The bottom is a different story with a different lesson, and the lesson is one that took the standards committee years of argument to settle. It is also the only part of IEEE 754 that hardware manufacturers still routinely turn off.

What is down there #

What a float can hold establishes that the representable numbers are spaced by a gap that halves at every power of two going down. At some point the exponent field runs out. For fp16 that is 2⁻¹⁴ = 6.1·10⁻⁵, which is the smallest normal — the smallest number with the standard’s implicit leading one.

A format could stop there. Everything below the smallest normal would round to zero, the exponent field would be fully used, and the representable set would be a tidy geometric arrangement with a sudden hole in the middle.

IEEE 754 does not stop there. It reserves the bottom exponent code for numbers with an implicit leading zero instead of a one, which gives the subnormals: multiples of 2⁻²⁴ running from 2⁻²⁴ up to the smallest normal. Those are p − 1 further octaves of numbers, at a spacing that no longer halves, with fewer and fewer significant bits as they get smaller until the last one has a single bit.

The band’s width is a fact about the significand rather than about the exponent, and it is worth stating because it is the opposite of what the name suggests. The subnormals run from 2^(emin−p+1) to 2^(emin), which is p − 1 octaves whatever the exponent field does. So a format with more precision has more to lose at the bottom, and the site checks that identity against the measured span at every position of the slider.

The sentence it buys #

The reason all this exists is one property, and it is a property nobody thinks about because it is always true where it has been tested.

x − y = 0 implies x = y.

Every algorithm that tests a difference against zero relies on it. Every convergence test of the form “stop when the iterate stops moving” relies on it. Every deduplication, every equality-by-subtraction, every if (a != b) guarding a division by their difference relies on it.

Without subnormals it is false, and it is false in a specific and easily constructed way. Take x equal to the smallest normal and y the next representable number above it. Their difference is one ulp at that magnitude, which is 2⁻²⁴ in fp16 — below the smallest normal, so with flush-to-zero it becomes zero, while x and y are demonstrably different numbers.

The site counts them rather than describing them. Over the ten distinct pairs immediately above the smallest normal in fp16:

All of them. Not a rare event on adversarial data — every single pair in that band, on data constructed by adding one ulp.

Why anybody turns it off #

The obvious question, given the table above, is why flush-to-zero exists at all. Two answers, and both are about hardware rather than about mathematics.

Subnormals are slow. They are the case the fast path does not handle: an arithmetic unit built around a normalised significand has to detect the subnormal case, shift, and often trap to microcode or to software. On many processors an operation producing a subnormal result costs tens or hundreds of times what a normal one costs. A loop that wanders into the subnormal band can slow down by two orders of magnitude with no other symptom, which is a genuinely alarming performance cliff and one that has caused real production incidents.

And the values are usually meaningless anyway. A quantity that has decayed to 10⁻³⁸ in a single-precision computation is, in most physical settings, zero — and the argument that its last three significant bits are worth preserving is hard to make.

So compilers offer a fast-math flag that flushes, GPUs flush by default in some modes, and a great deal of production code runs with the property above simply not holding.

The counter-argument is what makes it worth an essay rather than a footnote. The performance cliff is a reason to avoid producing subnormals, not a reason to redefine them as zero. The two responses are different: rescaling the computation so it does not enter the band keeps the identity and removes the slowdown, while flushing removes the slowdown and breaks the identity. The first requires knowing something about the data and the second does not, which is exactly why the second is the one that ships.

“The width of the flat section is the precision” is exact rather than approximate, and the whole slider says so.

The top of the band has not moved and the bottom has fallen by a factor of eight, which is three halvings for three bits.

The band spans exactly p − 1 octaves. Three at four bits, four at five, and so on to ten at eleven — eight widths, eight exact hits, no exception. So the subnormal range is not merely related to the precision, it is the precision minus one, and a format’s cushion at the bottom is decided by the field everybody quotes for accuracy.

The top of the band is fixed at 6.1·10⁻⁵ and the bottom halves with every bit. 7.63·10⁻⁶, 3.81·10⁻⁶, 1.91·10⁻⁶, 9.54·10⁻⁷ and on down — each exactly half its predecessor. That is the clean decomposition: the exponent field decides where the band starts and the significand decides how far down it reaches, and the two halves of a format’s budget each own one end of it.

And flush-to-zero destroys every pair in the band, at every width. The last column is 3 of 3, 4 of 4, 5 of 5, 6 of 6, 7 of 7, 8 of 8, 9 of 9 and 10 of 10 — not most, not the closest ones, all of them. Two distinct representable numbers in the subnormal range subtract to exactly zero under flush-to-zero at any precision the figure can draw, which is the sharpest statement of what gradual underflow buys: not accuracy near the bottom, but the property that x ≠ y implies x − y ≠ 0.

Gradual, and what it means quantitatively #

The word in the standard is gradual underflow, and it is doing precise work.

Without subnormals, the smallest representable positive number is 6.1·10⁻⁵ in fp16, and the next thing below it is zero. A value at 3·10⁻⁵ has nowhere to go: it becomes zero, and its relative error is 1. The error jumps from one part in two thousand to one hundred per cent in a single step, which is the cliff.

With subnormals, a value at 3·10⁻⁵ rounds to a multiple of 2⁻²⁴ and keeps nine of its eleven bits. A value at 3·10⁻⁶ keeps six. A value at 10⁻⁷ keeps one. Precision degrades one bit at a time across the band, which is what “gradual” means, and the relative error rises smoothly from u to 1 across ten octaves instead of jumping at one.

The site checks the arithmetic of that band directly: the subnormals are evenly spaced, so a value at one and a half times the smallest one rounds to twice it, and below half of the smallest one there is nothing left. Both are assertions rather than descriptions, and the second is the boundary of the representable set — the point at which the whole apparatus does finally stop.

An honest accounting of what it buys #

The strongest claims made for gradual underflow are stronger than the evidence, and it is worth separating them.

The identity above is real and it is checked. x − y = 0 if and only if x = y, ten pairs out of ten, in both directions.

The smoothness of the relative error is real and it is what the arithmetic of the band delivers.

The claim that it “saves” computations is mostly not. A computation whose intermediate values have decayed into the subnormal range is a computation in trouble for reasons that a few extra bits at the bottom will not repair. The band is ten octaves wide; the loss going into it is already catastrophic on any relative measure. Subnormals turn a discontinuity into a slope, and a slope down to zero is still a slope down to zero.

What they genuinely prevent is a spurious zero — a difference, a residual, a denominator that becomes exactly zero when it should be small, and then divides. That failure is not gradual and does not announce itself, which is precisely the pattern this site keeps finding: the dangerous failures are the ones that produce a well-formed answer of the wrong kind.

The argument that settled it #

The subnormals were the most contested item in the standard, and the argument that carried them is worth knowing because it is not the one a reader would guess.

The case against was straightforward and was made by people who build arithmetic units: an extra case in the hardware, an extra branch on the critical path, and a benefit confined to values so small that nobody should be relying on them.

The case for was not “the numbers are useful”. It was that without them, a fundamental algebraic identity is false, and the identity is one that every programmer assumes without knowing they have assumed it. That is the x − y = 0 test above, and the argument’s force is that a violation of it does not look like a numerical problem. It looks like a logic error: a branch taken that should not have been, a division by a denominator that was tested and found nonzero and then computed as zero somewhere else, an iteration that reports convergence because two distinct iterates subtracted to nothing.

The generalisation is the useful part. The standard is full of choices that could have gone either way on accuracy grounds and went the way they did on predicate grounds — signed zeros so that 1/(+0) and 1/(−0) differ and branch cuts behave; the exact rounding of the arithmetic operations so that a comparison means what it says; the requirement that the operations be correctly rounded rather than merely close, so that two implementations agree bit for bit.

Every one of those is a choice to make the arithmetic predictable rather than to make it accurate, and they are two different objectives. This site spends most of its length on accuracy, so it is worth naming the other one once: a great deal of what makes floating point usable is not that the answers are close but that the same expression gives the same answer, and that the ordinary laws a program’s control flow depends on continue to hold. Subnormals are one of those laws, bought with an awkward case in the hardware.

Which also explains why the flag exists. Turning off gradual underflow costs almost nothing in accuracy — the values involved are tiny — and costs a predicate. A user optimising for a benchmark’s running time is not measuring predicates, and a user whose code silently takes a wrong branch six months later is not measuring anything.

Detecting it, which is harder than it should be #

One practical note, since the failure is silent by construction.

There is no portable way to ask “did this computation produce a subnormal”. IEEE 754 defines an underflow exception flag, most hardware raises it, and most high-level languages give no access to it — JavaScript, which this site is written in, has none at all, which is why the measurements here are made by comparing two simulated contexts rather than by reading a flag.

What is available in practice is cruder. Compare two runs, one with the fast-math flag and one without, and look at whether the answers differ; a difference means the computation entered the band. Watch for the performance cliff, since a loop that suddenly runs a hundred times slower has almost certainly found the subnormals. And rescale on principle where the dynamic range of the data is known: a computation whose values are held near unity does not visit either end of the format, which removes this failure and the one a norm that overflows before it is a norm is about at the same time.

The first of those is worth measuring before it is relied on. A difference does mean the computation entered the band; the converse is what a detector is used for, and it does not hold. A 64-term dot product at eleven significand bits and five exponent bits — where the smallest normal is 6.1·10⁻⁵ and the smallest subnormal 6.0·10⁻⁸ — with the data scaled down a decade at a time:

At a scale of 10⁻⁴ the dot product returns exactly zero where the answer is 6.7·10⁻⁷ — a total loss — and the two runs agree. Both contexts round every product to zero, so there is nothing for the comparison to see. The detector fires in one decade of scale, on the case where the loss is half a per cent, and is silent across four decades where the loss is everything.

The reason is one line and it is structural rather than bad luck: flush-to-zero and subnormal support differ only for values inside the band. Below the band both round to zero and the runs agree exactly. So the comparison detects being in the subnormal range, which is the mild condition, and not having fallen through it, which is the failure this essay is about. Read as a positive signal it is sound; read as an all-clear it is worse than nothing, because the silence is loudest exactly where the answer has gone. It is the same asymmetry a bound that is proved is about: a test that can confirm and cannot refute is a test whose silence carries no information.

The third response has no such gap, and the reason it does not is that it detects nothing. Holding the data near unity puts every product back inside the range at every scale, which is why it keeps being the answer — and why the other half of a format treats the exponent as the thing being budgeted.

That last one is the recurring answer in this field and it is worth stating plainly, since two essays have now arrived at it. Most range failures are scaling failures — the scaling that buys ten orders is the measurement, and a condition number scaling cannot move is the boundary of it. Overflow at the top and underflow at the bottom are the same phenomenon seen from opposite ends — data whose magnitudes are not where the format is — and the fix in both cases is to move the data rather than to widen the format. The formats have narrowed by a factor of three in significand bits over the last decade and by a factor of two in exponent bits — eight bits and a format that breaks the rules prices the narrowest of them — and the practice of scaling, which double precision made unnecessary for thirty years, is becoming load-bearing again.

What is asserted here #

The subnormal band is p − 1 octaves, checked against the measured span between the smallest subnormal and the smallest normal at every significand width on the slider. Two routes: one counts octaves from the two constants, the other is the arithmetic identity.

With subnormals, no two distinct neighbours subtract to zero. Ten pairs, none of them.

And without them, every one of the ten does. Both directions, because the first alone would be consistent with a format in which the second was also true.

The subnormals are evenly spaced, checked by rounding one and a half of the smallest one to exactly two of them — which is the property that distinguishes the flat section of the staircase from the doubling section above it.

And below half the smallest subnormal there is nothing, which is the bottom of the representable set and the assertion that the band does end somewhere.

The refusals: that flush-to-zero leaves subtraction faithful, fed a pair one ulp apart at the smallest normal; that overflow is a rounding error near the answer; and that fp16 and tf32 are one format because their significands match. All three are the same shape of mistake — a property of the significand mistaken for a property of the format — and all three are refused.

The two ends, side by side #

The phase’s arithmetic field ends with a symmetry that is not a symmetry, and it is worth setting out because the two failures get treated as one thing and are not.

At the top, the failure is loud, absorbing and unrecoverable within the format. An infinity propagates, a NaN propagates further, and no later step repairs either. What can be done about it is everything: rescale, and it does not happen. The scaled norm is three lines long and removes the failure entirely.

At the bottom, the failure is silent and partial. A value loses bits one at a time, and the worst outcome — a difference that should be small becoming exactly zero — is a well-formed number that no flag reports. What can be done about it is less: subnormals soften the landing and do not stop the descent, and rescaling helps only if the small values are small for a reason the scaling can express.

So the two ends of a format’s range are not mirror images. One is a cliff with a guard rail three lines long, and the other is a slope with a well-formed wrong answer at the bottom of it. The one that gets the attention is the first, and the one that has quietly been switched off across most of the hardware anybody runs is the second.

A quantity that reaches them without trying #

The subnormal range is usually met as a curiosity at the bottom of a format. There is one ordinary quantity that walks straight into it at ordinary sizes: the determinant of a matrix, which is a product of n numbers and therefore scales like the nth power of them.