The thread: Precision as the knob — page 2
The tolerance that buys no agreement
Ask for four more orders of accuracy and you get them — the answers improve by a factor of 1.5 million. The ratio between the best and the worst run is 1.34, 1.48, 1.71 and 1.17 across the same sweep. The band falls and it does not close.
Neither sparse nor denseThe offset that moved the slope
The accuracy is supposed to lift a storage curve and leave its growth alone. Measured at seven tolerances, the strong partition adds 10.9 numbers per unknown per doubling at two digits and 36.5 at twelve — the growth rate more than triples, so ten decades of accuracy cost 33 per cent more storage at 64 unknowns and 118 per cent at 512.
The arithmetic underneathA bit buys an octave
The outlier a block survives is exactly two raised to its significand width — 8 at three bits, 32 at five, 128 at seven, 512 at nine. Each extra bit doubles the range the block tolerates and halves the ordinary entry's error. Reordering the same numbers buys every octave at once and costs nothing.
The arithmetic underneathA bound that is proved
Every error statement in the essays before this one is a measurement of one run. Interval arithmetic makes a different kind of claim — the answer lies in this set, for this input, with no probability attached — and its failure mode is that it returns nothing at all. On a Hilbert system it proves a bound 23 times the error it bounds, and one size later it refuses.
When the index is a tupleThe repair that costs exactly itself
A ridge on every subproblem is the standard cure for a swamp and it works — the terms stop at 1.61 instead of climbing past 6.7, and a run that never finished finishes in 798 sweeps. On a tensor that does have an answer the error it costs is the ridge itself, to within a factor of two, at every setting from 10⁻¹⁰ to 10⁻¹. And the terms it returns are still the right terms.
The arithmetic underneathProving the answer is in the box
Every other method here computes a number and estimates how wrong it is. This one returns a verdict: there is exactly one solution in this box, or there is none, or — the honest third outcome — nothing can be said. Two of the three are proofs about infinitely many points from finitely many operations.
The arithmetic underneathThe units that overflow before the answer does
A change of variable that is exact in the algebra requires γ² times a matrix to be a number the format can hold. In binary64 that is a bound nobody meets by accident. In binary32 it arrives at 10¹⁹ and in fp16 at 256, and past it there is no answer rather than a poor one.
The arithmetic underneathNine steps of pessimism
A proved bound is 8 to 26 times the error it bounds, at every precision from 16 to 40 significand bits. A carried interval is (√2)ᵐ times too wide after m re-enclosures. The two cross between eight and nine, so the method everybody warns against is the tighter of the two for a short computation.
The answer that depends on the machineThe reading that never moves
Across thirty runs — five grids from 36 to 196 unknowns, six working precisions from 53 significand bits down to 8 — the residual conjugate gradients stops on stays between 1.10·10⁻¹³ and 9.95·10⁻¹³. Over the same thirty runs the error of the answer spans a factor of 2.39·10¹¹, and the step count more than doubles. The one number the run publishes is the only one that responds to neither axis.
The arithmetic underneathThe far well in a byte
Half precision's subnormals carried a two-well chain's far well ten octaves deeper than flush to zero, and the essay that measured it predicted that on the eight-bit formats the band would be worth almost nothing and starting high almost everything. Measured on nine chains, the band is worth what it always was, two or three octaves; starting at the top of the range is worth ten and fourteen; and at three significant bits the far well is lost on eight chains of nine with no exponent limit at all, because the precision fails before the range does.
The arithmetic underneathRoom for the step, not the value
Starting a recurrence at the top of an eight-bit format's range spends its headroom once. An exponent counter beside the stored value was predicted to spend it at every step, remove every range failure and leave only the precision. Rescaled on the value — back to one whenever it nears the bottom or the top — it does not: E4M3 still loses its far well on seven or more chains of nine, because one climbing step multiplies the value by forty and E4M3 clips silently at 448. Rescaled into a window read off the chain's own rates, it reproduces the unbounded exponent bit for bit on all 531 runs of every format, and what is left of E4M3's limit is nothing.
The arithmetic underneathThe formula sets the power, the sum sets the constant
The one-pass variance was found losing three to sixty-nine times more than the digits-lost rule allows, and the explanation offered — operands built from a thousand roundings — was never priced. Measured against the exact variance of the stored data, the rule's ratio is the variance's own condition number squared, and the formula decides what power of it the error grows with: two for every one-pass variant, one for Welford's update, none for two passes or a shift. The summation decides only the constant in front — about 0.4√n for running sums, one for pairwise or compensated ones — and the sixty-nine was neither: it belonged to the random numbers.
The answer that depends on the machineA later look is a worse stop
Every stopping test for reduced-precision conjugate gradients had read the residual, and the residual flattens tens of steps before the error stops falling. The method carries a cheap lower bound on its own energy error — the sum of each step's step length times its squared residual over a window of steps that follow — and a clause that stops when that bound falls below the working precision's share of the iterate stops twenty runs on the scaled Laplacian within 1.22 times their least error, a third of the residual clause's overrun. The prediction that it would stop within thirty steps of the least on the median run misses by three. And the window turns out to be a pure delay: in reduced precision the error creeps back up after its least, so the longer the window waits, the worse the stop.
The answer that depends on the machineA short window from the start
Conjugate gradients carries a cheap lower bound on its own energy error, summed over a window of later steps, and a clause that stops when it falls below the working precision is safe — but the window is a pure delay. A window that starts at twenty and halves each time the estimate falls by a decade was predicted to keep the long window's safety and lose most of its delay. It loses the delay: under half window 20's overrun. It does not stop every run within 1.12 of its least error, as predicted; its worst is 1.126. And a plain window of five, which never needed to shrink, beats it on both counts at once — 1.097 at worst, 239 steps past the least in all against 335. The thing the shrinking was meant to avoid, stopping before the least, costs at most a few per cent, because near its least the error is flat.