Influence is decided before the data

A regression report prints a leverage beside every observation, and it is read the way the other columns of that report are read: as something the data did. A large residual means the model missed that point. A large deleted residual means dropping it would move the answer. A large leverage is taken in the same spirit — this observation was unusual, something happened there.

Nothing happened there. The leverage of the ith observation is

hᵢ  =  aᵢᵀ (AᵀA)⁻¹ aᵢ

— a row of the design matrix, the Gram matrix of the design matrix, and the same row again. The response vector does not appear. Every leverage in a fit is fixed at the moment the experiment is laid out, before an instrument is switched on, and the measurements that arrive later cannot move one of them by a bit.

That much is a definition and nobody disputes it. What follows from it is stronger, and it is what this essay measures: the leverages are not merely independent of the response, they are constrained in total. Each lies between zero and one, and they sum to the number of columns. A fit with p columns has exactly p units of influence to hand out among its m observations, and the only thing a design decides is who gets them.

The consequence is that an absolute leverage means almost nothing. Half a unit of influence is an enormous share of a three-column budget spread over sixty rows and an ordinary share of an eight-column budget spread over thirty-two, and the number 0.5 is identical in both cases. What carries information is the ratio of a leverage to the average the budget forces, and that average is p/m for every design that has ever been fitted.

The budget is the number of columns, to the last bit the format has #

The first row of that figure is scaled until its leverage equals one half, so the axis of every panel in this essay is an input rather than an outcome. Everything else about the design is left alone, which means the sum is free to come out wherever the arithmetic puts it.

It comes out at the number of columns. Measured across four shapes — forty observations of six columns, sixty of three, thirty-two of eight, twenty-four of four — the sums read 6.000000000, 3.000000000, 8.000000000 and 2.000000000 to the nine decimal places the figures print, and the last of those is a sixteen-row two-column fit rather than one of the first three resized. Four different numbers of rows, four different numbers of columns, four different sets of random entries, and the total influence is the column count every time.

At full precision the agreement is what a sum of m floating-point numbers can be and no better. The seven designs drawn or quoted here return sums whose distance from the integer is between exactly zero and 1.78·10⁻¹⁵ — two of the seven land on the integer to the last bit, and the worst of them is off by less than one unit in the last place of the number six. The theorem is exact, the summation is not, and the gap between those two facts is a rounding error rather than a finding.

The reason the sum is the column count is the reason the object is called a hat matrix at all. H = A(AᵀA)⁻¹Aᵀ is the orthogonal projection onto the column space, its trace is the dimension of the space it projects onto, and the trace of a matrix is the sum of its diagonal. So Σhᵢ = tr(H) = p, and there is no room in that derivation for the data, for the noise, or for anything an experimenter could have done differently once the design was fixed.

The same half-unit is a ten-fold outlier and a boundary case #

Every panel in this essay holds its first observation at h = 0.5000. Nothing else is held fixed, and the interpretation of that number moves by a factor of five across the designs drawn.

On sixty observations of three columns the average leverage is 0.0500, so the constructed row carries ten times the average share. On thirty-two observations of eight columns the average is 0.2500 and the same row carries twice it. Between them the ratio reads 3.333 at forty rows and six columns, 3.000 at twenty-four and four, 5.000 at sixty and six, 4.000 at sixteen and two, and 2.500 at fifty and ten. Those are not seven independent measurements. The ratio of a leverage to the average is h·m/p, and with h pinned at one half it is m/2p exactly, which is what the seven numbers are.

The common working rule flags an observation whose leverage exceeds twice the average, and the two extremes of that list are what the rule does with one number. At sixty rows and three columns the threshold is 0.100 and a row at 0.5 clears it fivefold; five further rows on that design clear it too. At thirty-two rows and eight columns the threshold is 0.500, and the same row does not clear it at all — it sits exactly on the line, and no other row on that design comes near.

So a report that lists leverages and marks the ones above a cut is reporting a fact about its own design that it is presenting as a fact about its observations. That is the same substitution a numerical rank threshold makes when a default cut is quoted as though it were a property of the matrix, and the same one a null-space basis makes when a one-line convenience decides a condition number nobody chose.

The second-largest leverage on that design is worth reading off the panel. It is 0.4565, reached by a row nobody scaled, which is 1.83 times the average. The constructed outlier and an untouched neighbour differ by nine per cent. A budget of eight units over thirty-two rows simply does not leave enough room for one row to stand out, whatever the entries are, and that ceiling was set by the shape of the matrix rather than by anything in it.

A column costs one unit, and it is taken from everybody #

Because the total is the column count, adding a column adds exactly one unit of influence to be shared out. That is a strong statement about what a modelling decision does, and the two sixty-row panels here are the controlled version of it: the same sixty rows, three columns against six, budgets of 3.000000000 and 6.000000000, averages of 0.0500 and 0.1000.

The row held at one half moves from ten times the average to five times it, without moving. Doubling the number of parameters halves the ratio the diagnostic reports for an observation that is unchanged in every entry. An analyst who adds three columns to a model and finds that a previously alarming observation is no longer flagged has learned nothing about the observation.

The reverse of the same conservation law is that raising one row’s leverage lowers everybody else’s, and by exactly the amount raised. On the forty-row six-column design the first row takes 0.500000 of the six units, leaving 5.500000 for the other thirty-nine — an average of 0.141026 against the nominal 0.150. Scaling that row by two takes its leverage from 0.500000 to 0.800000, and the total still reads 6.000000000: the extra three tenths of a unit came out of the remaining rows and out of nowhere else.

That is what makes the word influence accurate rather than metaphorical. Influence in a least-squares fit is a conserved quantity. It cannot be created by measuring something interesting and it cannot be destroyed by measuring something dull, and the only operation that moves any of it is a change to the design.

Nothing in the number is the response, and this can be checked rather than argued #

The claim that the response does not enter is a reading of a formula, and the site’s habit is to give such a reading a test it could fail. The test is cheap: hold the design and vary the response as widely as the arithmetic allows.

Three responses were generated on the fixed forty-by-six design — coefficients drawn afresh and scaled by forty, noise levels of 0.001, 0.1 and 5.0, so the signal-to-noise ratio spans five orders of magnitude between the extremes. The largest difference between any two of the three leverage vectors is zero. Not small: the forty doubles are bitwise identical across all three fits, which is what it looks like when a quantity is computed from a matrix that did not change.

A second invariance is less obvious and more useful. Leverage is a property of the column space of the design rather than of the design’s columns, so any reparameterisation that spans the same space must leave it alone. Right-multiplying the design by a random invertible six-by-six matrix with a condition number of 7.37 moves no leverage by more than 7.2·10⁻¹⁶. Rescaling the columns individually by factors from 10⁶ down to 10⁻⁴ — which takes the design’s own condition number from 1.93 to 1.27·10¹⁰, nine and a half orders of magnitude — moves no leverage by more than 8.3·10⁻¹⁷.

That last measurement is the sharpest available separation between two things that get discussed together. Leverage is not conditioning. A design can be made arbitrarily ill conditioned without disturbing a single leverage, because the ill-conditioning lives in the coefficients and the leverage lives in the rows. The valley with no bottom is about the first: a direction the data never determined, read along the parameters. This essay is about the second, read along the observations, and the two quantities are orthogonal in the strong sense that one of them can be moved ten orders of magnitude while the other does not move at all.

The lower panel is the half that does depend on what was found #

Every figure here has a second panel, and it is a different kind of quantity. The deleted residual — what the fit would have predicted at observation i had observation i not been in it — is

e₍ᵢ₎  =  eᵢ / (1 − hᵢ)

which is a product of two independent things. The numerator is the ordinary residual and it exists only after the measurements arrive. The denominator is one minus the leverage and it was settled with the design.

The closed form is not obviously the same thing as refitting, so each panel checks it by refitting: the model is solved m times, once with each observation withheld, and the prediction error at the withheld point is compared against the division. Across the seven designs the worst relative disagreement runs from 1.2·10⁻¹³ on the sixty-by-three fit to 6.4·10⁻¹² on the twenty-four-by-four one, with the forty-by-six panel at 1.3·10⁻¹². Sixty refits and sixty divisions give the same sixty numbers, and this is the site’s two routes to one number habit applied to a shortcut every statistical package ships. Checking a closed form against the computation it replaces is the same discipline as checking a solver against an answer that is known, with the expensive route standing in for the exact one.

So the deleted residual factorises into a term nobody controls and a term nobody can change once the experiment has run. That is the useful shape of it. The response half is what the world did; the design half is what the experimenter did, and only the second was ever a decision.

Where the design half stops being the interesting one #

An argument that a diagnostic is really about the design would be worth little if it did not say where the response takes over, and on an ordinary fit the response takes over almost everywhere.

On the forty-by-six design the raw residuals span a factor of 96.8 between the smallest and the largest in absolute value, while the multiplier 1/(1 − h) spans a factor of only 1.946 — from 1.0277 at the least influential row to exactly 2.0000 at the constructed one. The largest deleted residual on that fit belongs to observation six, which has an unremarkable leverage; observation zero, which holds the largest leverage on the design, is nowhere near the top of the lower panel. Ranking observations by deleted residual on that fit is very nearly ranking them by residual.

The crossover can be named. For the design factor to match the spread the response already produces, 1/(1 − h) would have to reach 96.8, which is h = 0.98967. Below that the deleted residual is dominated by what was measured; above it, by where it was measured. Ordinary designs live far below it, which is exactly why the leverage column of a regression report looks uninformative most of the time — and why the one occasion it matters is the occasion nobody is watching for.

That asymmetry is worth stating in the site’s usual terms. A small residual is not a small error, and a high-leverage observation is the cleanest way to produce one: the fit is dragged onto the point, the point is fitted well, and the residual is small because the observation bought its own agreement. A diagnostic that ranks by residual will place such a point last.

The figure above is the same number wearing different clothes. Removing an observation from a factorisation takes the square root of 1 − h, so the divisor of the statistician’s shortcut is the discriminant of the numerical analyst’s rotation, and both are undefined at the same place. That essay is about what 1 − h does to an algorithm’s accuracy; this one is about what h was before either of them was computed, and the two never touch except at that shared number. It is the same relation a rank-one update has to the matrix it routes through: the quantity that governs the arithmetic belongs to an object that is not the answer.

Where the budget runs out #

The bound at the top of the range is not decorative and it is reachable. Three measurements walk up to it.

A saturated design. With m = p the design is square and invertible, the fit interpolates, and the projection onto its column space is the identity. Every leverage is then exactly one — measured at six columns and at ten, every entry reads 1.000000000000 and the sums read 6.000000000 and 10.000000000. Every observation determines its own fitted value entirely, no observation can be deleted, and every deleted residual is a division by zero. The budget is spent and every row holds a whole unit of it.

One row of slack. At seven observations of six columns the leverages run from 0.540020 to 0.987176 against an average of 0.857. At twelve observations of ten columns they run from 0.681043 to 0.933832 against an average of 0.833. Every row on both designs is above any threshold anyone would set, which is correct and useless: when the budget is nearly as large as the number of rows, every row is influential and the diagnostic has nothing to distinguish.

And the figure refuses to draw it. The hat-matrix panel accepts a design only when there are at least four observations for every column. Handed twenty observations of eight columns — an average leverage of 0.400, where the usual cut of twice the average is 0.800 and almost nothing can reach it — it fails rather than drawing a panel whose axis carries no information. That refusal is fed its counterexample on every build, so the claim that the diagnostic needs a design to be a diagnostic at all is a computation rather than a remark.

The floor has a story too, and it depends on the model rather than on the data. Designs of random entries put their smallest leverage wherever they like: 0.0113 on the sixteen-by-two panel, 0.0019 on the sixty-by-three one. A model with an intercept cannot do that. A constant column forces hᵢ ≥ 1/m for every row, and on three designs carrying one the smallest leverages measure 0.039747 against 1/m = 0.025000, 0.017141 against 0.016667, and 0.053812 against 0.041667. Every observation in a model with an intercept holds at least one m-th of a unit whatever else the design does, which is the sense in which fitting a mean gives every point a vote.

What follows, and all of it precedes the data #

The habit this produces is not a formula. It is a change in when the question gets asked.

An experiment can be audited for influence before it is run. The design matrix exists as soon as the settings are chosen, so the whole leverage vector — every ratio, every row that will dominate a direction of the fit — is computable while the plan can still be changed. The response is not needed and would not help. This is the only diagnostic on a regression report of which that is true.

Report the ratio, never the leverage. A leverage of 0.4 is a crisis on a design with a hundred rows and three columns and unremarkable on one with twenty rows and eight, and h ÷ (p/m) says which in one number. A report that prints the raw value is asking every reader to divide by an average it did not print.

Read a high leverage as a statement about coverage. An observation with h near one is holding a direction of the fit up by itself, which means the design sampled that direction once. The remedy is another observation in the same region, and it is available before anything is measured. Adding a row is cheaper than imposing the constraint that would fix the direction afterwards — and a row given unbounded weight is a row driven to h = 1, which is the same limit reached from the other side.

Do not read a leverage as a condition number. The two are independent, measured here to 8.3·10⁻¹⁷ across ten orders of magnitude of rescaling. A well-conditioned design can hand one row half its budget, and a design at κ = 10¹⁰ can spread its leverage perfectly evenly.

And know which factorisation produced it. The leverages here are computed as ‖R⁻ᵀaᵢ‖² against the Cholesky factor of AᵀA, one triangular solve per row, which is cheap precisely because forming the Gram matrix has already been paid for by the solve. Where that route was not taken — where the fit went through a QR factorisation of A directly, which is what a library does — the leverages are the squared row norms of Q and are cheaper still. Two routes, one number, and the arithmetic of the second is much better behaved.

There is a general shape here that this field keeps producing. A choice made for one purpose settles a number belonging to another: an elimination order chosen for convenience decides how much memory a factorisation needs, and a set of experimental settings chosen for coverage decides which observations will be allowed to dominate the answer. In both cases the number is fixed early, is computable early, and is looked at late — or not at all, which is the usual outcome, because it does not appear on the report until the data has arrived and by then it reads as a property of the data.

It never was. The response enters the deleted residual and it enters nothing else on this page. Half the influence structure of any fit is a decision, taken before the first measurement, by whoever chose where to look.