The weights the fit chooses for itself

A unit is a statement about the noise found that total least squares has no answer of its own. It minimises one Frobenius norm over every column of $[A\ b]$, so the unit each column is written in decides how the correction is shared out, and rescaling one column carries the fit continuously between two named estimators: the reverse regression of that column on the rest, and the mixed fit that treats it as exact. Dividing every column by its own noise level is the unit that means something, and it was the best estimator on all three placements of the noise the essay tried. But it needs the noise levels, and the essay got them from replicate readings — a second experiment.

It then named the shortcut every practitioner reaches for, and predicted its failure. “A tempting shortcut is to iterate: fit, estimate each column’s noise from the correction the fit applied to it, reweight, and fit again. For Gaussian noise the ratio of the noise levels is known not to be identifiable from the data alone. The prediction with a sign is that on these fits the iteration converges, and to weights that depend on the units it starts from: started from $c = 0.01$ and from $c = 100$ on the same data, the converged weight on the first column differs by more than a factor of ten, and neither start reaches the error of the replicate weights with three readings.”

The iteration fails, as predicted, but not in the predicted way. It does not converge anywhere. The way it fails says exactly what “not identifiable” means.

What one step of the shortcut does #

The fits are the earlier essay’s: sixty rows, two measured regressors with true coefficients 1.5 and −0.7, and noise drawn in each of the three columns of $[A\ b]$ at three placements — the same in every column; largest in the first, at four times the second’s and twice $b$’s; and almost all in $b$, twelve times the regressors’. Weights $s_j$, one per column, mean “column $j$ is divided by $s_j$ before the fit”. Only their ratios matter, so they are normalised to one on the second regressor, and a start at unit factor $c$ means weights $(1/c, 1, 1)$: column one written in units $c$ times smaller.

One step: fit total least squares with the current weights, convert its correction back to the original units, take each column’s root-mean-square correction, and make those the next weights.

That step has a closed form, and it explains everything that follows. Total least squares in weighted coordinates replaces $[A\ b]S^{-1}$ by its nearest rank-deficient matrix, and the correction it subtracts is rank one: $\sigma\,u\,v^{\mathsf T}$, with $\sigma$ the smallest singular value and $u$, $v$ its singular vectors. Back in the original units, column $j$’s correction is $\sigma\,u\,v_j s_j$, and its root-mean-square size is $\sigma\,|v_j|\,s_j/\sqrt{m}$. So a step does nothing but multiply each weight by $|v_j|$, the size of its own column’s component in the smallest right singular vector — and then renormalise.

No run settles #

Run it from the five starts on forty fits at each placement and not one of the six hundred runs converges. Every one leaves a millionfold range of weights within two to seven steps and keeps going — to $10^{60}$, where the measurement stops. Turn the dial: from $c = 0.01$, $0.1$ and $1$ the path runs out toward large weights on column one and on $b$; from $10$ and $100$, toward small weights on both regressors.

The ends are named estimators. A weight on column one that goes to infinity means column one is assumed to carry all the noise, which is the reverse regression; weights on both regressors that go to zero mean they are assumed exact, which is ordinary least squares of $b$ on $A$. On all six hundred runs the final fit is one of the two to a part in a thousand: the reverse regression from every start with $c \le 1$, ordinary least squares from every start with $c \ge 10$.

So the prediction’s second clause holds in the most extreme form available — the weight on the first column from the two starts differs not by a factor of ten but by a factor of $10^{120}$ — and its first clause is false. There are no converged weights to compare. There are two attractors at the edge of the weight space, and the start decides which one a run reaches.

The weights that would be consistent, and why nothing stays there #

A weighting at which the shortcut would stop does exist. It is the one where every column’s component in the smallest singular vector has the same size, so that multiplying each weight by $|v_j|$ changes nothing but the normalisation. The forward map runs away from it, so it is found by running the map backwards — dividing each weight by $|v_j|$ — which converges in ten to twenty-one steps on every fit.

It is unstable, and the rate is the same everywhere. Start one part in a million away and the forward map doubles the departure at each step — a median growth of 2.14, 2.09 and 2.04 a step on the three placements — until, after about twenty steps, the run is gone. The reason is the feedback in the step’s closed form. A column given a slightly larger weight is assumed to be slightly noisier, so the fit puts slightly more of its correction there, its root-mean-square correction grows, and the next weight is larger still. Nothing pushes back, because nothing in the data says how noisy any column actually is.

An iteration that walks out of the set found alternating least squares lowering its objective at every sweep and still failing to converge, because the minimum it was approaching was not in the set it searched. The reweighting is a cleaner case: it has a fixed point inside its set, and leaves it. An iteration that only multiplies found Newton–Schulz converging exactly when every singular value was below $\sqrt 3$ and diverging otherwise; the reweighting is an iteration that only multiplies whose multiplier is above one in every direction that matters.

Damping slows it and does not stop it #

The usual repair for an iteration that overshoots is to take smaller steps: move each weight only part of the way toward the correction’s size, $s_j \leftarrow s_j\,(\text{rms}_j/s_j)^{\alpha}$ with $\alpha$ below one. That repairs overshoot. It does not repair a fixed point that repels, because it changes the multiplier’s size and not which side of one it is on: a departure that doubled at full step grows by $2^{\alpha}$ at a damped one, which is still more than one for any positive $\alpha$.

Measured from the start at $c = 1$ on the forty fits with equal noise, the damped runs behave exactly so. At full step the median run leaves a millionfold range after six steps; at half a step, after ten; at a quarter, after eighteen and a half. Not one of the hundred and twenty runs converges, and every one ends at the reverse regression, as the undamped run from the same start did. Damping buys time and changes nothing else — which is what a smaller step does to an iteration whose problem is the direction it is pushed in rather than how far.

The two edges, and why they are those two #

The runs end at the reverse regression and at ordinary least squares, never at the mixed fit that treats only column one as exact, and the step’s closed form says why. Each step multiplies a weight by its column’s share of the smallest singular vector. A start that weights column one heavily makes that column look noisy, the fit puts its correction there, the share grows, and column one’s weight runs to infinity while the other two shrink together relative to it: all the noise is in column one, which is the reverse regression. A start that weights column one lightly makes it look exact; the fit corrects the second regressor and $b$; but the second regressor’s weight is fixed at one by the normalisation, so it is $b$’s weight that runs away upward relative to both regressors, which is the same as both regressors’ weights running to zero: all the noise in $b$, which is least squares. The mixed fit would need column one’s weight to fall while $b$’s and column two’s stay level, and nothing in the feedback holds two weights level.

The projection and the right angle drew least squares as the projection of $b$ onto the columns of $A$, with all the correction perpendicular to them in $b$. The reverse regression is the same picture with a regressor playing the part of $b$. Total least squares sits between such pictures, and the earlier essay showed rescaling a column carries it from one to the other. The shortcut is a rescaling chosen by the fit’s own output, and the feedback carries it all the way.

The self-consistent weights are the geometry’s #

The fixed point’s location says why. Found on all a hundred and twenty fits, the self-consistent weights have medians of 0.515, 0.530 and 0.477 on column one and 0.746, 0.708 and 0.782 on $b$, on the three placements. The true ratios on those placements are one and one, four and two, and one and twelve. The three clouds of self-consistent weights lie on top of one another, a long way from all three truths.

That is non-identifiability made visible. The data are sixty rows of three numbers each, and Gaussian noise of any allocation among the columns, added to points on a plane, produces a cloud whose shape is the same to second order — a plane with scatter about it. What the fit can see is the plane and the size of the scatter, not which column put it there. The weights at which the corrections agree with themselves are a property of the cloud’s shape, which here is set by the spread of the regressors and the coefficients, and they come out the same whether the noise is in the regressors or in $b$.

The valley with no bottom found a polynomial fit whose coefficients could move by a third of their size while the residual changed only in its sixth digit: a direction in which the data hold no information, so that the fitted answer along it is decided by rounding, the starting point or the algorithm. The noise ratio is a direction of exactly that kind, one level up. Every allocation of the noise among the columns fits the data equally well to second order, so the data cannot prefer one; whatever a procedure returns along that direction was put there by the procedure. The reweighting returns an edge, because its feedback pushes it along the flat direction until something stops it, and nothing does until a weight is infinite.

The theory says the ratio cannot be identified from second moments. The measurement shows what a procedure that tries anyway does: it finds the one weighting the cloud’s shape singles out, cannot stay there, and runs to an edge.

What each landing costs #

The prediction’s third clause was that neither start would reach the error of weights estimated from three replicate readings. It holds on two placements and fails on the third, and where it fails is instructive.

With the noise the same in every column, the replicate weights give a median error of 0.127 and the known weights 0.114. The reverse regression gives 0.161, least squares 0.317. Neither landing is close. With the noise largest in the first column, the reverse regression gives 0.091 — exactly the known weights’ error, because this placement puts almost all the noise where the reverse regression assumes it — and three replicates give 0.084. With the noise almost all in $b$, ordinary least squares gives 0.043, better than three replicates’ 0.046, because least squares assumes exactly that.

So a run of the shortcut is right when its start happens to put it in the basin of the estimator whose assumption matches the truth, and it has no way to know. From the same data, started on the other side, it returns 0.148 on the third placement and 0.264 on the second, three to six times the known weights’ error. The self-consistent weights, if a procedure could be made to stay there, would give 0.098, 0.106 and 0.106: good on the first placement by coincidence, and two and a half times the known weights’ error on the third.

What the replicates were for #

The two numbers a caller has showed that the residual and the condition number of $A$ cannot say whether total least squares or least squares is the right method. This is the same conclusion reached from the inside: the fit’s own corrections cannot say how the noise is shared either, and an iteration that pretends they can is not a weaker method but a coin with two faces. When the matrix is wrong too introduced total least squares as the method for data in which $A$ is measured as well as $b$; this essay adds the price of that method’s honesty. It needs to be told the noise, and only an experiment can tell it.

That is what replicate readings, repeated measurements of a standard, or an instrument’s specification are for. They are not a refinement of the fit. They carry information the fit does not contain. The earlier essay found three replicates enough to get within a tenth of the known weights’ error; this one finds that zero replicates, plus any amount of cleverness with the residuals, is a choice between two named estimators made by the starting units.

What sixty rows do not show #

Every fit here has two regressors, sixty rows and Gaussian noise. With non-Gaussian noise the noise ratio becomes identifiable in principle, through higher moments, and an estimator that used them could do what this iteration cannot; none is measured. The step is the root-mean-square of each column’s correction, plain or damped; other summaries of a correction — its largest entry, its median — would change the multiplier’s form and are not tried, though the argument that a column weighted up is corrected more applies to any of them. Influence is decided before the data is a reminder that a fit’s geometry, set by its regressors, governs much of what it does; whether it governs the self-consistent weights is the first question below. And the explanation of the self-consistent weights as the cloud’s shape rests on their being nearly the same across placements; how they move with the regressors’ spread is not swept.

Still open: the cloud’s shape, and noise that is not Gaussian #

What sets the self-consistent weights. If they are a property of the cloud’s shape, they should move with the spread of the regressors and not with the noise. The prediction with a sign is that doubling the spread of the second regressor while keeping every noise level fixed moves the self-consistent weight on column one by more than a factor of 1.5 on all three placements, while multiplying every noise level by three moves it by less than a tenth.

Noise with a skew. With noise drawn from a skewed distribution, third moments carry information about which column is noisy. The prediction is that a reweighting that matches each column’s third central moment of the correction to the data’s, instead of its root-mean-square, converges from every start on noise drawn as centred exponentials, and to weights within thirty per cent of the true ratios on at least two placements of three — the first estimator here that can see the noise from the data alone, and only because the noise is not Gaussian.