The two numbers a caller has

When the matrix is wrong too ends on a choice and hands it to somebody who is not in the room. Ordinary least squares assumes the matrix is exact and moves the whole correction onto the right-hand side; total least squares assumes both were measured and moves the smallest correction that touches either. Neither is the better method. Each is the estimator that is right under one account of where the error came from, and the account is a fact about the instrument that produced the data rather than about the arrays that arrive at the solver.

So the decision needs a quantity the caller does not have. What the caller does have is two numbers, both cheap, both familiar, and both computed from A and b alone: the residual ‖Ax − b‖, and the condition number κ(A).

That the first of them ranks the two answers backwards is settled and is not this essay. ‖Ax − b‖ is the quantity ordinary least squares minimises, which is what the right angle in the picture everybody draws means, so no other vector can make it smaller: not the total-least-squares answer, and not the coefficients the problem was built from. The ordering is therefore reversed wherever total least squares is the more accurate, by construction rather than by an unlucky draw, and that is the sharpest instance the site has of a small residual not being a small error.

The question left over is whether a caller who already knows the ordering is reversed can use the number anyway. Read the residuals, find them separated by some amount, and infer that the two answers differ by something like that amount in the other direction: it is the obvious salvage, and the whole of this essay is that the arithmetic of it does not work. The residual’s separation and the accuracy’s separation are two different sizes, and they differ by between one and two orders of magnitude. The second number is worse than that. κ(A) does not merely fail to track the difficulty of a total-least-squares fit; across a sweep on which the answer loses a factor of sixty-two, it falls.

The separation is two per cent and the stake is seventy #

Take the figure above at its own numbers. Total least squares is the more accurate answer here by a factor of 1.701: its distance from the true coefficients is 0.03071 against 0.05224. That is a substantial difference, and it is the difference the choice of method is about.

The residuals on the same problems are 3.398 and 3.467. The ratio is 1.0202, so the two bars a caller can actually see are 2.02 per cent apart, and they are ordered the wrong way. Set the two separations beside each other: 70.1 per cent of accuracy is at stake and 2.02 per cent of residual is visible, a ratio of 34.6 to one.

This is a different complaint from the ordering one and it survives the ordering being known. A caller told that the larger residual marks the better answer has been told the sign of a quantity whose magnitude is a thirty-fifth of what it is a proxy for. Reading the 2 per cent as though it were the size of the disagreement understates the disagreement by a factor of thirty-five; reading it as a mere flag, and switching methods whenever the two residuals differ at all, is a rule that fires on every problem in this family, because ordinary least squares minimises the residual and the strict inequality therefore holds everywhere.

The separation is small for a reason that is worth naming, because it explains why no amount of care in reading it will help. The residual at the least-squares answer is a minimum, and a smooth function near a minimum is flat: moving the coefficient vector away from the minimiser changes ‖Ax − b‖ only in second order, while it changes the distance to the truth in first order. That is the same geometry the valley with no bottom measures directly, where a degree-nine fit’s coefficients move by a third of their own size before the residual changes in the sixth significant figure. Here the valley is being used as a discriminator between two points inside it, which is the one job its shape guarantees it cannot do.

And the resolution is worst where the noise is smallest #

Drop the total noise from 20 per cent to 5 and the picture does not merely shrink. The two residuals become 0.859 and 0.8602, which is a separation of 0.14 per cent — one part in seven hundred — and the accuracy difference is 14.4 per cent. The stake is now a hundred and four times the signal.

That number rises as the noise falls, which is the opposite of what a caller would want. At 50 per cent noise the residual separation is 11.48 per cent against an accuracy difference of 79.5 per cent, a ratio of 6.9. At 20 per cent it is 2.02 against 70.1, a ratio of 34.6. At 5 per cent it is 0.14 against 14.4, a ratio of 104. The instrument is least informative exactly where the arithmetic of reading it is hardest, since a 0.14 per cent difference between two residuals is four significant figures into numbers a caller would ordinarily print to three.

There is a second thing true at 5 per cent noise and it is the more important one. At that level the whole comparison has stopped meaning anything. The essay that set the method up found the threshold at a total noise of 0.08: below it the bias ordinary least squares carries when the matrix is noisy is second order in the noise and falls beneath the variance the two methods share, and the sweep across the location of the noise flattens into seed-to-seed spread. The figure above sits inside that regime, and so does a measurement that is worth recording because the site’s own machinery refuses to draw it.

With all of the noise in the matrix and only 5 per cent of it, ordinary least squares is the more accurate method. The advantage runs 0.914 the wrong way — the ordinary estimate is 9.4 per cent closer to the truth than the total-least-squares one on the problem total least squares was designed for. The figure will not draw that combination: its accuracy assertion is calibrated at a 20 per cent noise level and rejects the frame rather than presenting a reversal as a finding. The reversal is real and it is not a mechanism; it is what a family looks like when the quantity being estimated is smaller than the spread of the estimate.

The separation is not even ordered by the answer #

A caller could still hope for something weaker than a magnitude: that a wider residual separation at least means a worse mistake. It does not, and the two figures below settle it against each other.

Move the noise from 60 per cent in the matrix to 40, holding the total at 20 per cent. The accuracy ordering flips, which is the crossing the essay before this one measured, and is expected: ordinary least squares is now the more accurate, by a factor of 1.292, at 0.03592 against 0.0464. The residual ordering does not flip, because it cannot. What is new is the size: the residual separation at 40 per cent is 2.26 per cent, which is larger than the 2.02 per cent measured at 60 per cent, where total least squares was ahead by 70 per cent.

So the wider gap belongs to the case in which the method with the smaller residual is the correct choice, and the narrower gap to the case in which it is the wrong one. Read as a magnitude, the residual separation points backwards from the ordering that is already backwards.

Swept finely across the share of the noise at a fixed total of 20 per cent, the residual separation runs 2.41, 2.26, 2.19, 2.19, 2.02, 2.13, 2.22, 2.24, 2.84 and 3.57 per cent, from all the noise in the right-hand side to all of it in the matrix. It falls, reaches a minimum near the share where the accuracy ordering crosses, and rises again — a U, on an axis along which the accuracy advantage is monotone, running from 0.284 to 2.475. A monotone quantity cannot be recovered from a U-shaped one. The map is two to one, and the two branches are the two sides of the decision.

That ambiguity has an exact instance. At a share of 0.4 the residual separation is 2.2580 per cent and ordinary least squares is ahead by 29.2 per cent. At a share of 0.75 the residual separation is 2.2446 per cent and total least squares is ahead by 149.0 per cent. The two separations differ by 0.6 per cent of themselves — six parts in a thousand, well below what any of these medians resolves — and the two verdicts are opposite and differ by a factor of 3.22 in relative accuracy. The residual does not have the resolving power to distinguish those two problems, and they are the two problems the whole question is about.

One problem is not the family #

Every number so far is a median over forty seeds, which is how this site draws a comparison whose per-problem answer is noisy. A caller has one problem. It is worth asking what the two bars look like on it, and the answer removes what remains of the case for reading them.

Across the forty problems behind the 60-per-cent figure, the residual ratio runs from 1.014171 to 1.026111 with a median of 1.020998. Across the forty behind the 40-per-cent figure it runs from 1.016183 to 1.029790 with a median of 1.022719. The two ranges overlap on all but a sliver at each end, and the difference between their medians — 0.17 percentage points — is about an eighth of the width of either band. A single problem’s residual ratio therefore carries essentially no information about which side of the crossing that problem is on. The separation between the two shares is buried in the variation within each of them.

The accuracy on a single problem is not stable either, and that is the honest other half. At the 60-per-cent share the per-problem advantage runs from 0.211 to 6.145, and twelve of the forty problems have ordinary least squares ahead even though the median says otherwise by a factor of 1.70. At the 40-per-cent share it runs from 0.108 to 5.237 and sixteen of forty have total least squares ahead. The choice of method is a decision about a family, made once, on grounds that are not in any single problem — which is what where the answer stops being in the data says about a truncation index and what when the answer is a choice says about a regularisation parameter. The failure is not that the caller reads the residual badly. It is that the residual on one problem is a draw from a distribution whose two candidate parents are indistinguishable.

The second instrument moves the wrong way #

The residual is at least uninformative in a direction a caller can be warned about. The condition number is not, and this is where the essay’s refusal is.

κ(A) is an amplification factor: perturb the input by a known amount, look at how far the output moves, and the largest ratio available is the number. It is the right instrument for an ordinary least-squares solve, where it is the quantity the normal equations square and the reason that route is not taken. Reaching for it on a total-least-squares problem is the natural move, and the essay before this one recorded that it governs nothing there: the conditioning is the gap between σ_n(A) and σₙ₊₁([A b]), two singular values of two different matrices, which close on each other as the noise rises until no finite answer exists.

What that essay did not measure is the direction. Sweeping the noise on a 24×3 fit, with the same amount in the matrix and in the right-hand side at each stop, the gap closes from 1.4002 to 0.2389 and the total-least-squares error rises from 0.0117 to 0.722, a factor of 62. Over the same six stops κ(A) reads 3.54, 3.51, 3.43, 3.19, 2.78, 2.46. It moves by a factor of 1.44 — and it moves downward, monotonically, at every stop.

So a caller watching κ(A) as the data degrades does not see a flat line and conclude nothing. They see the problem becoming better conditioned, by nearly a half, while the answer loses close to two orders of magnitude. The reason is not subtle once stated: adding noise to a matrix built from powers of an evenly spaced variable moves its smallest singular value up faster than its largest, because random perturbation fills in the directions the design left thin. κ(A) is measuring that, correctly, and it is a fact about A rather than about the fit. The refusal published with this essay is fed the claim that the total-least-squares error grows with κ(A) and is required to fail.

More rows widen the gap and κ does not notice that either #

The same sweep at five times the observations separates the three quantities further, and it does it by moving two of them and leaving the third alone.

At 120 rows the gap opens: 2.9295 at the least noise against 1.4002 at 24 rows, because σ_n(A) grows like the square root of the number of rows while the noise the augmented matrix picks up does not keep pace. The error falls with it at every stop, from 0.0117 to 0.00509 at the smallest noise level. And the sweep now spans a factor of 201 in the total-least-squares error, from 0.00509 to 1.021, which is more than two orders.

κ(A) over those six stops reads 3.71, 3.68, 3.60, 3.33, 2.81, 2.42 — a factor of 1.53, and downward again. Between the two sizes it barely moves at all: 3.54 against 3.71 at the least noise, 2.46 against 2.42 at the most. A quantity that changes by 5 per cent between two problems whose errors differ by a factor of 2.3, and by 1.5 across a sweep on which the error changes by 201, is not a weak predictor. It is unrelated to what it is being asked about, which is what the verdict wrong-blame names: the measurement is correct and the thing it is being blamed on is not the thing that moved.

There is a second reading in the same two figures, and it is the one that matters for the choice of method rather than for its conditioning. At 120 rows total least squares is the more accurate answer at the four lowest noise levels and the less accurate at the two highest; at 24 rows it wins only at the two lowest. More data moves the crossing, in the direction the earlier essay predicted from the bias being invariant in the number of rows while the variance falls — and nothing a caller computes reports which side of that crossing a given fit is on.

What does predict it, and who can compute that #

The gap is not merely better than κ(A); it is close to sufficient, and putting a number on that is what makes the comparison worth drawing rather than asserting.

First-order perturbation theory for this estimator says the error should go as the noise level divided by the gap. Multiplying the measured error by the measured gap and dividing by the noise level should therefore give a constant. At 120 rows the six stops give 0.745, 0.810, 0.729, 0.733, 0.825 and 0.848 — a spread of a factor of 1.16 across a sweep on which the error itself moves by 201. At 24 rows the same combination gives 0.816, 0.787, 0.736, 0.622, 0.529 and 0.345, a factor of 2.4, with the drift beginning where the noise stops being small relative to the design and first-order theory stops describing it.

So one quantity accounts for the error to within 16 per cent over two orders of magnitude, and the familiar one accounts for none of it while pointing the other way. That is the same division three errors and one number draws between the parts of a forward error that are properties of something and the parts that are decisions reported by nothing.

Two ingredients go into the predictor and a caller has an uneven claim on them. σₙ₊₁([A b]) is computed by the method: it is the singular value the estimator zeroes, it comes out of the same decomposition that produces the answer, and obtaining σ_n(A) alongside it costs one more decomposition of a matrix one column narrower. The gap is therefore available at essentially no extra cost to anyone who has already run the method, and it is not returned, because the interface of a solver is a vector. That is the collection’s recurring shape — the sketch that is spent is the same defect in the other direction, where the evidence is consumed rather than discarded — and it is the cheapest of the findings here to act on.

The noise level is not available. It is the standard deviation of an error that was never observed, and estimating it from the fit is circular: the residual is what a caller would estimate it from, and the residual is the quantity this essay has spent four figures showing is not sensitive to what is being asked. Told a noise level ten times too small, the discrepancy principle in choosing without knowing goes from an error of 0.112 to 10,449, which is the price of guessing this particular input wrongly in a neighbouring problem.

Where this leaves the choice #

The residual is not a weak version of the accuracy comparison. It separates the two answers by 0.14 to 14.68 per cent across the whole family while the accuracy separates them by 14.4 to 239 per cent, it is ordered backwards wherever the choice is live, and its magnitude is U-shaped where the accuracy is monotone. There is no monotone function of it that recovers the ordering.

κ(A) is worse than silent. It falls by a factor of 1.44 to 1.53 across sweeps on which the error rises by 62 and by 201, so a caller monitoring it is watching a number improve while the answer decays. It is measuring the conditioning of a linear solve with A, which is a real quantity and a differently-posed question from the one being asked.

The gap is computable and unreported. Both singular values it is built from are available to anybody who has run the method, and returning them costs one decomposition of a narrower matrix. A routine that returned the answer alongside σ_n(A) and σₙ₊₁([A b]) would let a caller see the sensitivity that governs it.

And the choice of estimator remains a modelling decision. Nothing measured here selects it, because the quantity that selects it is where the noise came from, and that is a property of the experiment rather than of A and b. It is decided in practice by which routine has no adjective in front of its name, which is a choice presented as an algorithm and not the last time this collection meets one.