Bias variance — where it appears
Named by 4 essays across 2 fields — each of them below, with the objects they name alongside it.
One arc, and what each filter pays to be on it
Conjugate gradients and Tikhonov stop at the same effective dimension, and that could have meant two curves crossing once or one curve. It is one curve over a stretch — on six problems, Tikhonov and truncation reach the iteration's error at the iteration's dimension to within 7.2 per cent from 0.7 of the answer to its top — and the two separate on either side. But the curve is shared by a trade, not by an identical answer: at the same dimension Tikhonov carries 22 to 42 per cent more noise than the iteration and up to five per cent less bias.
The overshoot was the lead
Past its best, conjugate gradients' error rose more slowly than Tikhonov's at the same effective dimension — on the narrow blur at 0.1% noise Tikhonov was 2.35 times worse at 1.3 of the answer — and the reading was that the iteration spends its dimension where the data has content. Count admitted directions instead of summing factors, so that a factor of 1.81 counts once, and the lead is gone: 0.96 on that problem, and within 0.17 of one on all six from 1.1 to 1.3. Tikhonov now carries less noise than the iteration there. Below half the answer, where the three filters also disagree, the count changes nothing.
The degree that is safe to overshoot
The rules that choose a Tikhonov parameter miss by factors of millions on one draw in twenty. Transplanted to the degree of a polynomial fit, in a basis orthonormal on the data, the same rules never cost more than 2.7 times the best degree's error in three hundred draws. The reason is the shape of the valley they search: six degrees too few costs from 44 to 16,000 times the best error, forty degrees too many costs about twice it. The one rule with a tail, the discrepancy principle, has its threshold half a standard deviation above the residual it is waiting for.
The rule that divides by one minus the leverage
On a smooth function the rules that choose a polynomial fit's degree were forgiven by a lopsided valley: too few degrees cost thousands of times the best error, too many about twice. On |x| and on a function with a pole just outside the interval, the left wall was predicted to flatten, the best degree to climb, and the information criterion's tight threshold to become a liability. All three happen. The wall falls to a twentieth more, the best degree on |x| runs to the largest the basis has, and the criterion is the costliest rule on |x| at one per cent noise. But the rule that actually fails is leave-one-out: at small noise it stops at degree 26 and costs twelve times the best error, because by degree 26 the leverage of the end points is 0.97 and its formula divides every residual by what is left.
Named alongside it
The objects these essays reach for when they reach for this one.
Conjugate gradientsCross-validationDeconvolutionDiscrepancy principleEffective dimensionFilter factorsGeneralised cross-validationIterative regularisationLeast squaresLeverageOrthonormal basisSemi-convergence