General form regularisation — where it appears
Named by 5 essays across one field — each of them below, with the objects they name alongside it.
A rule that has to be told how good its answer will be
The L-curve's corner reads a noise share of 0.10 to 0.21 across fifteen pairings of signal and penalty, and the share the best λ sits at runs from 0.0037 to 0.43 — a factor of a hundred and fifteen. A rule aimed at the right share is within a few per cent of the oracle on every one of them. The right share is about a third to four-fifths of the relative error that λ will achieve, which is the number the answer was wanted for.
A second penalty is not a second parameter
Penalise ‖x‖ and ‖L₁x‖ at once and there are two λ to choose. Over ten draws on five signals the best pair beats the better single penalty by between 0.00% and 3.1%, and one of its two parameters is exactly zero on 30 to 70 per cent of draws. Choosing the wrong one of the two costs up to 54%. The surface is a choice between two curves with a knob nobody needs.
A third penalty on a flat floor
Two penalties at once were worth three per cent at most, and the explanation offered was that one penalty already does the work — which predicts that a third buys less still and that the best choice sits on a face of the parameter cube. The third buys a median of exactly nothing on all five signals and 0.66% on its best draw. But on twelve draws of forty the best triple does use all three, and on every one of them the nearest point with a penalty switched off is within that same 0.66%. The minimum is not on a face; it is on a floor so flat that where it lands is noise. Choosing the right single penalty is worth a factor of two.
The floor is the part the noise buried
A third penalty bought nothing, and the prediction was that a signal made of two structures — a smooth arch joined to a staircase — would be where mixing finally pays, by more than ten per cent. It pays 0.76% at most, over twenty-four draws at three positions of the join. The flat floor every mixture lands on has a size that can be read off before any penalty is chosen: the share of the answer carried by coefficients the noise has buried. The best of a thousand-odd mixtures sits within nine per cent of it on every draw of every signal with a jump or a spike, and the best single penalty within ten per cent of it at every noise level from one per cent to ten to the minus five. What gets under it is a penalty told where the join is — by ten to thirty-one per cent, on one join of three.
The search finds a join, not the join
A smooth arch joined to a staircase is reconstructed 10 to 31 per cent better by a pair of penalties split at the join — the second difference before it, the first after — than by any uniform mixture, when the join is at three quarters and the penalty is told where it is. A search that tries every join and keeps the one generalised cross-validation likes best was predicted to find it within two grid points on every draw and keep two thirds of the gain. Left to itself, GCV collapses on three draws of twenty-four, choosing a first-difference strength of 10⁻⁴ and twenty times the error. With both strengths floored at a thousandth it finds the join within two points on seven draws of eight — and keeps a third of the gain, because the strengths it picks are not the error's. And where the join was at a quarter, so that splitting there buys nothing, it puts the join past the staircase's last step instead, and that split beats the best uniform pair by up to fifteen per cent: a structure nobody had told it about.
Named alongside it
The objects these essays reach for when they reach for this one.
Parameter choiceRegularisationTikhonov regularisationL-curveFilter factorsGeneralised cross-validationNormal equationsOracleSingular value decompositionCondition numberDiscrepancy principleNoise floor