A second penalty is not a second parameter

The L-curve plots one norm against one residual, so it has a corner in the way a plane curve can. Penalise two norms at once — ‖x‖ for size and ‖L₁x‖ for roughness — and the object becomes a surface in three quantities with two parameters indexing it, and the question the earlier measurement ended on is what such a surface would have instead of a corner.

The answer is that it does not need one, because the second parameter is not doing anything. That is a measurement and not an argument, and it took ten draws on each of five signals to make.

Solving it without squaring it #

The problem is

and it has an obvious solution route that these essays do not take. Its normal equations are $(A^{\mathsf T}A + \lambda_1^2 I + \lambda_2^2 L^{\mathsf T}L)x = A^{\mathsf T}b$, which is symmetric, positive definite for any positive λ and solvable by a Cholesky factorisation in a third of the work of anything else. It is also the route that squares a condition number, which is the whole subject of the road that squares the problem and the reason this field keeps refusing it.

The route taken instead writes the objective as one least-squares residual. With A = UΣVᵀ and $z = V^{\mathsf T}x$, the first two terms are $z^{\mathsf T}(\Sigma^2 + \lambda_1^2 I)z - 2z^{\mathsf T}\Sigma c$ up to a constant that does not depend on z, where c = Uᵀb. So the whole objective is the squared residual of a stacked system whose top block is the diagonal $(\Sigma^2 + \lambda_1^2 I)^{1/2}$ and whose bottom block is $\lambda_2 LV$ — an (n + m)-by-n least-squares problem with a diagonal top, factorised by Householder reflections. The first parameter enters as a diagonal shift and costs nothing; the second enters as a scaled block and costs a factorisation.

That arrangement is what makes a sweep over a grid of pairs affordable at all. Two hundred and fifty-six pairs on a 64-point problem take about six tenths of a second, and the shares reported below need two further solves at each pair — one with the signal’s coefficients and one with the noise’s — because the noise’s contribution to a norm is not recoverable from the solution that contains both.

One detail of that arrangement is worth keeping because it is the reason the sweep over the first parameter is nearly free. The diagonal top block means $\lambda_1$ never touches the factorisation: changing it rescales $n$ numbers and changes the right-hand side, and the Householder work is all in the $\lambda_2 LV$ block. So a surface of $k$ by $k$ pairs costs $k$ factorisations rather than $k^2$, and the asymmetry between the two parameters that the measurement below finds is already present, in a much less interesting way, in what it costs to sweep them.

What the surface looks like #

The grid runs from 10⁻⁶ to 1 in each parameter with zero prepended, so its outer row and outer column are each penalty used alone and its corner cell is the unregularised solve.

On the two-bump signal under ‖x‖ and ‖L₁x‖, the best pair reaches 0.00343 at $\lambda_1 = 1.9\cdot 10^{-2}$, $\lambda_2 = 5.2\cdot 10^{-2}$. ‖x‖ alone reaches 0.00483 and ‖L₁x‖ alone reaches 0.00351. So the mixture is worth 2.3 per cent over the better of the two and 41 per cent over the worse.

That is the largest gain in the table, and it is the signal this figure was chosen for. On the other four the gain is smaller, and on three of them the best pair is on an edge — one parameter at exactly zero, the mixture not used at all.

The optimum is well localised — one to eight of the 196 cells are within one per cent of it — so this is not a flat surface with an arbitrary minimum. It is a surface whose minimum keeps sitting on its boundary.

It is worth saying what “on an edge” means here, because it is a statement about a grid and not about a limit. The grid prepends an exact zero to each axis, so the outer row and column are the single-penalty problems solved exactly rather than approximated by a very small parameter. A minimum in that row is a minimum at λ = 0 and not a minimum at the smallest λ the grid happens to carry, which is the distinction that makes the count below mean anything. Refining the grid would move some of those optima to small positive values, and it would not change what those values buy — which is the quantity the next section measures and the one the argument turns on.

What the mixture buys, and what the choice buys #

Ten draws of the noise on each of five signals, with the better single penalty and the pair both scored against that draw’s own best:

The mixture is worth a median of 1.0002, 1.0000, 1.0314, 1.0000 and 1.0000 on the five signals, and its best draw anywhere in the table is 1.171. Choosing the right single penalty is worth 1.008, 1.011, 1.298, 1.542 and 1.011, and its worst draw is 1.68.

So on the one signal where the mixture does anything, the choice does ten times more; on the signal where the choice matters most, the mixture is worth nothing at all. And the best pair has one parameter at exactly zero on 30, 60, 30, 70 and 50 per cent of draws — between a third and two-thirds of the time the sweep over a surface returns a point that a sweep over a line would have found.

For comparison, the quantity these rules are actually struggling with is larger by an order of magnitude. The corner reads the norm it is drawn in measured the L-curve at 1.38 to 25.1 times the oracle across the same signals, and the earlier measurement found a rule that is nearly exact given a target nobody can supply. Against a rule-versus-oracle gap of 25, a mixture-versus-choice gain of 1.03 is not a place to spend a second parameter.

Why one penalty is enough #

The reason is visible in the shares, and it is that measurement’s reading applied twice.

A penalty’s oracle sits where the amplified noise is a particular fraction of the norm that penalty measures — a fraction proportional, to within a factor of three, to the accuracy the answer will have in that norm. Both norms here are norms of the same vector, filtered by the same filter. So a filter that puts the ‖x‖ share where the ‖x‖ oracle wants it has already put the ‖L₁x‖ share somewhere, and the question is whether that somewhere is where the ‖L₁x‖ oracle wants it.

Measured on the five signals it very nearly is. At the ‖x‖-only optimum the two shares are 0.0123 and 0.0772 on the step signal; at the ‖L₁x‖-only optimum they are 0.0097 and 0.0571. On the two-bump signal the two optima give (0.0028, 0.0166) and (0.0028, 0.0167) — indistinguishable. The two penalties are not asking for different filters. They are asking for the same filter and disagreeing slightly about where to stop.

That is the mechanism behind a result this field already had from another direction. Four knobs and one floor found a truncation, a Tikhonov parameter, a step count and a randomised rank reaching 0.1445, 0.1406, 0.1426 and 0.1449 on one problem — four methods sharing no arithmetic and landing within three per cent of each other. The floor is a property of the data, not of the method, and a method with two knobs instead of one reaches the same floor from a slightly different direction.

There is a second way to see that the two penalties are not independent, and it is the one that would generalise. The filter a penalty produces is determined by the operator and the penalty together — when L is the identity it is the familiar $\sigma^2/(\sigma^2 + \lambda^2)$ in the operator’s own singular basis, and for a derivative penalty it is the same expression in the generalised singular basis of the pair (A, L). Two penalties give three bases and one filter, and the filter has to be a single set of weights on a single set of directions whatever basis it is written in. A second parameter can only reshape that one set of weights, and the reshaping it offers is small because the two penalties agree about which directions are the dangerous ones: both of them damp the small singular values hardest, because that is where the noise is amplified, and they differ only in how hard.

That is why the gain is largest on the two-bump signal. It is the smoothest of the five, so its own coefficients fall fastest, so the region where the two penalties disagree about the weighting covers more of the directions that carry signal. On the four-spike signal, whose coefficients barely decay at all, the mixture is worth 1.0000.

What a surface would have instead of a corner #

The question the earlier measurement left was geometric: a curve has a corner, and a surface has what.

It has an edge and a vertex, and neither of them is what an L-curve’s corner is. The corner of an L-curve is a curvature maximum — the place where the plot stops being governed by amplified noise and starts being governed by discarded signal — and it is read off a one-dimensional curve because there is one direction along which to turn. A surface parameterised by two λ has a two-dimensional tangent plane at every point and a curvature that is a tensor rather than a number, so the detector has to be told which direction to look along before it can report anything.

The earlier measurement supplies the answer to that, and it is not encouraging. A corner reads the plotted norm, and the share it reads is right only when that norm’s own target happens to equal 0.14. With two plotted norms there are two shares and two targets, and satisfying both at once is a pair of conditions on a pair of parameters — solvable in principle, and requiring both targets, which is two numbers nobody has instead of one.

So the honest account is that the surface has no corner because it does not need one. It has an optimum, the optimum is on an edge more often than not, and getting to the edge means choosing between two curves rather than searching a sheet.

A practical reading follows from that and is worth having. Anyone who has written a general-form Tikhonov code has faced the temptation to expose both parameters, on the grounds that the user can always set one to zero. The measurement says the exposure is a liability: the second parameter enlarges the search by a factor of the grid size, it is worth at most three per cent on the best of five signals, and the user who sweeps it will spend that search rediscovering that one of the two should have been zero. One draw in twenty is the general warning — a search with more places to go finds worse answers more often — and a two-dimensional sweep over a surface whose optimum is usually on an edge is exactly the shape that invites it.

Which of the two to choose, then #

The measurement says the choice is what matters and says nothing about how to make it, which is worth stating plainly rather than leaving implied.

What can be said is negative and useful. The choice cannot be made by comparing the two L-curves’ corners, because the corner’s cost depends on the penalty in exactly the way the earlier measurement described — under ‖x‖ it costs 25.1 on the two-bump signal and under ‖L₂x‖ it costs 2.36, and that difference is about where the targets sit rather than about which penalty gives the better answer. On that signal ‖L₂x‖ does give the better answer, so the two agree; on the oscillating signal under ‖L₁x‖ they do not, and the penalty whose corner is cheaper is not the penalty whose oracle is better.

Thirty-two coefficients instead of a noise level found the missing input of one rule sitting in the data’s own tail, which is the shape a repair would take here too — some statistic of the data that says which norm the answer is smooth in. Nothing in these measurements supplies one.

Nor can it be made from the residual, which is the one quantity a code has in abundance. Both penalties reach essentially the same residual at their own optima, because both are stopping when the residual has come down to the noise; that is what the discrepancy principle is, and it is why the discrepancy principle is the most stable of the three rules across penalties and the least discriminating between them.

What the null result is worth #

A measurement that says a parameter does nothing reads as an absence, and it is worth saying what is present in it.

The general-form Tikhonov problem — one penalty, an arbitrary L — is standard and has been since the 1970s, and its whole point is that L encodes what “smooth” means for a particular problem. A second penalty is the natural next thought and it appears in the literature under several names, usually motivated by a case where the answer is known to be both small and smooth. What the surface says is that “both” is not a property a filter can express: the filter has one weight per direction and both penalties are asking about the same directions, so the second penalty does not add a dimension to the space of achievable answers, it adds a coordinate to a path through it.

The demonstration is stronger than a plausibility argument because the sweep is exhaustive. Every pair on the grid is solved and scored against the exact answer, so “the mixture buys 3 per cent” is not a claim about a search that might have looked harder; it is the best the family contains, over 225 pairs, at a resolution of a fifth of a decade. The oracle is what makes that statement available, and this is the kind of statement it exists for — not that a rule failed to find something, but that there was not much there.

And it closes a question this field opened from the other end. The corner was diagnosed as a fixed-share detector, which invited the thought that a surface might carry more information to detect with. It carries less: two shares whose targets are two unknown numbers, indexing a family whose useful range is one-dimensional.

What this does not settle #

One operator, one noise level for the full grid, five signals, and two derivative penalties paired with ‖x‖. The pairing ‖L₁x‖ with ‖L₂x‖ is not swept, and two penalties that are both derivative operators may interact differently from one that is and one that is not.

The grid is 15 by 15 over six decades plus zero, so a parameter is located to about a fifth of a decade and “exactly zero” means the prepended zero column rather than a limit. A finer grid would move some edge optima to very small interior values and would not change what those values buy, which is the quantity reported.

Ten draws a signal is enough for the medians quoted and is not enough for the maxima; the 1.171 is the best of fifty draws and nothing more.

Every share above is computed with the noise vector in hand, by solving the same system three times. It is a diagnostic and not a rule, in the same standing that measurement’s estimated share has and one step further from usability.

Still open: which curve, and the surface’s own conditioning #

A rule for the choice of penalty. The measurement here says the choice is worth up to 54 per cent and the mixture is worth 3, so the whole of the remaining value in the general-form family is in a decision nobody has a rule for. Whether the two penalties’ residuals at their own discrepancy stops differ enough to rank them — they very nearly do not — or whether some cheaper statistic does, is the direct continuation and it is one sweep.

Three penalties. If the argument above is right, a third penalty should buy less than the second, and the optimum should sit on a face of the cube rather than in its interior a larger share of the time. That is a prediction with a sign, and it is the cheapest test of whether the mechanism proposed here is the mechanism.

The conditioning of the stacked solve. The top block is $(\Sigma^2 + \lambda_1^2 I)^{1/2}$, so the first parameter conditions the problem before the second one is applied, and at small $\lambda_1$ the stacked matrix inherits the operator’s own decay. Whether the factorisation route is measurably more accurate than the normal-equations route at the λ these sweeps actually visit — or whether the regularisation has already made the difference invisible — is the measurement this essay assumed rather than made.

The surface under a Krylov method. A parameter chosen on a smaller problem found rules transferring from a projected problem to the full one with a known bias. A two-penalty problem projected onto a Krylov space has a surface of its own at every step, and whether its optimum sits on an edge as reliably as this one does would say whether the finding is about the penalties or about the dimension.