Sparsity, and what elimination costs

The factor is not sparse

A sparse matrix has a factor that is not sparse, and the gap between them is the entire reason iterative methods exist. The entries elimination creates can be counted before any arithmetic runs, from the graph alone.

A sparse matrix is not a compressed dense matrix. Its zeros are structural — they are where they are because the problem has no coupling there — and a matrix arising from a physical discretisation has a fixed number of entries per row however large it gets. The five-point Laplacian on a k×k grid has k² variables and about 5k² nonzeros, forever.

That is the good news, and it is the reason problems with millions of unknowns are posed at all. Storing such a matrix costs O(n). Multiplying by it costs O(n).

Solving with it does not, and the reason is the single fact this field exists to state: eliminating a variable couples every variable it touched to every other variable it touched, and each of those couplings is an entry that was zero in A and is not zero in L.

The 12×12 grid Laplacian and its Cholesky factor, ordered by naturalTwo square sparsity plots side by side. The left shows the nonzeros of the matrix; the right shows the nonzeros of its Cholesky factor, with the entries created by elimination marked in a second colour.the matrix, lower triangle408 entriesits Cholesky factor1739 entries · 1331 created‖A − LLᵀ‖/‖A‖1.4·10⁻¹⁶fill, symbolic1331fill, numeric1331n = 144 · density 3.2% · bandwidth 12same matrix, renumberedthe answer is identical to rounding
Fig. 1 The 12×12 grid Laplacian beside its Cholesky factor. The left panel is the matrix’s lower triangle; the right is the factor, with the entries elimination created drawn apart from the entries the matrix already had. Drag the ordering — the left panel never changes.

The rule, in one sentence

Elimination of variable k adds an edge between every pair of remaining variables that were both connected to k.

That is the whole of it. Think of the matrix as a graph: a vertex per variable, an edge wherever there is a nonzero off-diagonal entry. Eliminating a vertex removes it and turns its neighbourhood into a clique. Every edge that clique adds is a fill entry.

The consequence is immediate and it is what makes the subject hard. A vertex with d neighbours creates up to d(d−1)/2 new edges, so eliminating a single well-connected variable can add more entries than the matrix contained. And those new edges raise the degrees of the vertices they join, so the next elimination is more expensive than it would have been. Fill compounds.

Counting it without doing any arithmetic

The rule above mentions no numbers, and that is not an accident of the exposition — it is a property of the algorithm, and it is what makes sparse direct solving practical.

The fill pattern can be computed from the graph alone: no additions, no multiplications, no rounding, nothing that depends on the values of the entries. A real sparse solver does exactly this first, in a phase called symbolic factorisation, and it is why such a solver can allocate precisely the right amount of memory before it touches a number.

The site computes it both ways and requires them to agree.

Symbolically, by the clique rule, on a boolean matrix. Numerically, by running Cholesky and counting what came out nonzero. On the 12×12 grid both give 1,739 entries in the factor, against 408 in the matrix’s lower triangle. Not close to each other — the same integer, and the assertion is written as an equality of integers rather than as a tolerance, because an integer is the one kind of measurement on this site that cannot be approximately right.

That agreement is a genuine second route in the sense two routes to a number means. The two computations share nothing but the matrix: one is a walk over a graph, the other is a floating-point factorisation.

Where the two routes could legitimately differ

Worth stating, because a check that can never fail is not a check and this one has a known escape.

The symbolic count is an upper bound. It asks where a nonzero could appear; the numeric count asks where one did. If a fill entry happens to be exactly zero because the contributing terms cancelled, the numeric count is lower and the two disagree without anything being wrong.

On the model problems that does not happen, and the check is written as an equality for that reason. On a matrix with structural symmetries it can, and a solver that allocated from the symbolic count would simply have reserved slightly too much — which is the right direction for an error in a memory estimate to run.

Why the count is about Cholesky and not about LU

The essay has been quietly assuming a symmetric positive definite matrix, and the assumption is doing more than tidiness.

For a symmetric positive definite matrix, Cholesky requires no pivoting. That is a theorem, and it is what makes the symbolic phase possible: the elimination order can be chosen entirely on structural grounds, before any value is examined, because no numerical consideration will ever demand a different one. Structure and stability are decoupled.

For a general matrix they are not, and the field becomes considerably harder. The swap that is not optional establishes that elimination without a row interchange fails on matrices it has no business failing on, so a sparse LU has to pivot for stability — and the pivot it wants for stability is rarely the one the ordering wanted for sparsity. A row swap chosen at step k changes the fill pattern of every subsequent step, which means the symbolic prediction is no longer a prediction.

The practical resolutions are all compromises. Threshold pivoting accepts any pivot within a factor of the largest, using the freedom to prefer the sparse choice. Static pivoting commits to the structural ordering and perturbs a pivot that turns out to be too small, then repairs the damage with iterative refinement — which is buying the accuracy back doing structural work rather than precision work.

So the clean two-route agreement in this essay is a symmetric-positive-definite result. It is stated here in the case where it is exactly true, and the general case pays for its generality with a symbolic phase that can only bound what the numeric one will do.

What it costs, fitted rather than quoted

The usual way to state this is a complexity class, and this site’s habit is to measure the exponent.

Across grid sizes from 5×5 to 12×12, fitting the logarithm of the nonzero count against the logarithm of the dimension:

  • the matrix grows with an exponent of 1.04 — linear, as promised, since the stencil is fixed;
  • its factor grows with an exponent of 1.49.

The second number is the field. n1.5n^{1.5} for a two-dimensional problem is the textbook result and here it is a fitted slope over eight points rather than a citation. The assertions bracket it: above 1.05, so the factor genuinely grows faster than the matrix, and below 2, so it is genuinely better than dense.

The practical translation is unpleasant. A grid ten times finer in each direction has a hundred times the variables and about a thousand times the factor. Memory, not time, is what stops a sparse direct solver, and it stops it suddenly.

Fill growth under natural: the factor rises as n^1.49A log-log plot of nonzero count against matrix dimension. The matrix's own count is a straight line of slope one; the factor's is steeper; a dense factor is steeper still.10²10².³10²10³10⁴n (dimension)nonzerosdense factorfactor, n^1.49matrix, n^1.05grid Laplacians from 5×5 to 12×12fitted, not quoted
Fig. 2 The same measurement as a slope. The matrix’s line has slope one and the factor’s is steeper; a dense factor’s is steeper still. Fill is the gap between the first two lines, and no ordering closes it.

Why this is the reason iterative methods exist

The comparison that decides how large problems get solved.

A conjugate gradient iteration costs one matrix-vector product, which for a sparse matrix is O(n). The rate the condition number predicts measures the count at forty for the model problem at n = 40. So the iterative route costs O(kn), with k in the tens or hundreds and — importantly — storage of a fixed handful of vectors.

The direct route costs a factorisation whose storage is the number this essay measures. At n = 144 that is 1,739 entries against the matrix’s 408: a factor of four, which is nothing. At n = 10⁶ on a three-dimensional problem it is the difference between a machine that can hold the problem and one that cannot.

So the decision between iterating and factorising is a memory decision before it is a speed decision, and it is made from the fill count, which is available in advance from the symbolic phase. That is a much more definite basis than the iteration count, which the rate the condition number predicts shows can only be bounded to within an order of magnitude.

The direct method’s compensating virtue

It would be one-sided to leave it there, because sparse direct solvers are not a legacy technology and there are cases where nothing else will do.

A factorisation is reusable. The expensive phase depends only on the matrix, so a system solved against a thousand right-hand sides pays for the factor once and then pays a triangular solve each time. An iterative method pays in full every time.

It has no convergence question. There is no parameter to tune, no preconditioner to design, no stagnation to diagnose. The answer arrives, backward stable, with the residual the site prints beside every factorisation.

And it does not care about conditioning in the way an iteration does. A κ of 10¹⁰ makes an iterative method very slow and makes elimination merely inaccurate in the way the condition number is an amplifier describes, which is a bound nothing can improve on anyway.

The two approaches meet in the middle, at incomplete factorisations: a factor computed with the fill deliberately discarded, used not as a solver but as a preconditioner. That is changing the condition number on purpose, and it is exactly this essay’s quantity being traded against that essay’s.

Incomplete Cholesky on the 12×12 grid: κ 67.8 → 6.84A semi-logarithmic plot of relative residual against iteration for conjugate gradients with and without an incomplete Cholesky preconditioner, the preconditioned curve falling faster.061218243036424810⁻¹¹10⁻⁹10⁻⁷10⁻⁵10⁻³10⁻¹iteration‖r‖ / ‖b‖plain CGIC(0) CGwhat the preconditioner didκ(A)68κ(L⁻¹AL⁻ᵀ)6.8‖A − LLᵀ‖/‖A‖0.0842D Laplacian, n = 144√κ ratio predicts 3.15×
Fig. 3 The middle ground. The incomplete factorisation whose condition-number reduction that figure measures is this essay’s Cholesky with the fill thrown away — one restriction, and a factor that costs what the matrix costs to store.

What a sparse format actually stores

The figures on this site draw dense arrays, because at the sizes a picture can show, a real sparse format would be machinery with nothing to teach. What is simulated honestly is the count, and it is worth one paragraph on what that count corresponds to.

Compressed sparse column, the usual format, stores three arrays: the nonzero values, the row index of each, and a pointer per column saying where that column’s entries begin. So an entry costs a double and an integer, and a column costs one more integer. The nnz this essay counts is the length of the first two.

Two consequences follow, and both are why sparse solvers are written the way they are. The pattern must be known before the values are stored, since inserting an entry into the middle of a packed array means moving everything after it — which is precisely why the symbolic phase exists rather than being an optimisation. And the indirection costs: every access is a lookup through the index array, so a sparse matrix-vector product runs at a fraction of the throughput a dense one of the same operation count would, because the hardware cannot predict where the next value is.

That second point is a reason iterative methods are less dominant than the operation counts suggest. An iteration’s cost is one sparse product, and a sparse product is memory-bound rather than compute-bound — while the dense triangular solves inside a direct method run at full speed. Counting floating-point operations understates the direct method’s case, sometimes by a large factor.

What is asserted here

The two fill counts agree exactly, on five different structures, as integers.

The factor is denser than the matrix, at every position of the ordering slider — which is the claim that there is a subject here at all.

And the factorisation is exact, ‖A − LLᵀ‖/‖A‖ below 10⁻¹³ whichever ordering was used. That one is on the badge and it is doing real work: the entire next essay is about orderings that differ by a factor of six in cost, and the only thing that makes that comparison interesting is that they are all equally correct.

The matrix grows linearly and its factor does not, as fitted exponents with brackets on both.

The refusals: the claim that fill is the same under any two orderings must throw, and so must the claim that a good ordering eliminates fill rather than limiting it. Both do.

Three dimensions, where the argument turns

Everything measured here is two-dimensional, and the exponent changes in a way that decides how the subject is practised.

For a two-dimensional grid the factor grows as n1.5n^{1.5} and the work as n2n^2 — bad, and survivable. For a three-dimensional grid with the same nested-dissection reasoning the factor grows as n4/3n^{4/3} and the work as n2n^2, which sounds comparable and is not, because n itself is now k³ rather than k². A modest 1,000³ grid is 10⁹ unknowns, and a factor with n4/3n^{4/3} entries is 10¹² numbers.

That is the crossing point for the whole field. Two-dimensional problems are routinely solved directly, and the memory is affordable. Three-dimensional problems of any resolution are not, and this is why essentially every large-scale simulation code — fluid dynamics, electromagnetics, structural analysis at scale — is iterative. Not because iteration is more accurate, and not because it is faster per unknown, but because the factor does not fit.

The exponent in the figure is fitted on two-dimensional grids and says so. Extending it to three would need three-dimensional problems, and the honest reason this site does not draw them is that a spy plot of a 1,000-variable matrix is a grey square.

The second one is the one people believe

That second refusal is worth drawing out, because it catches a misreading that the next essay could otherwise create.

Shown the arrowhead matrix — where one ordering gives a completely dense factor and another gives no fill at all — the natural conclusion is that fill is an artefact of bad ordering, and a good ordering removes it. It does not. On the 12×12 grid, minimum degree is the best of the four orderings drawn and its factor still holds 1,026 entries against the matrix’s 408. Nested dissection, which has the better asymptotics, gives 1,413.

Every ordering fills in. The best available ordering fills in. The question the field answers is not how to avoid fill but how much of it is unavoidable, and finding the ordering that minimises it is NP-hard, so even that question is answered heuristically.

Which makes the arrowhead a teaching example rather than a representative one, and the next essay opens with it for exactly that reason — and then has to spend most of its length walking the conclusion back.

What the count is not

One clarification, because “the factor is denser than the matrix” invites a conclusion about accuracy that is not there.

Fill costs memory and it costs arithmetic. It does not cost accuracy. A factorisation under the worst ordering on this page and one under the best are both backward stable, both reconstruct the matrix to about 10⁻¹⁶, and both return the same solution to within rounding. The badge on the hero figure carries that number at every position of the slider precisely so the comparison in the next essay cannot be misread as a comparison of quality.

That separation — a decision that costs a great deal and affects no answer — is unusual on this site. Most of what is measured here is a trade between cost and accuracy: precision against speed, incomplete factorisation against exact, sketching against a full pass. Fill is a pure cost with no accuracy dimension at all, which is what makes ordering the one decision in the subject that can be made on structural grounds alone and be certain of losing nothing.

Gaussian elimination on a 4×4, one step at a timeFour copies of the same matrix: as given, and after each of the three elimination steps. The pivot in use is outlined and the entries reduced to zero are greyed.21-13-3-121-212-443-12as givenrows in the order 1 2 3 443-1201.251.252.502.51.5-30-0.5-0.52after step 1pivot 443-1202.51.5-3000.5400-0.21.4after step 2pivot 2.543-1202.51.5-3000.540003after step 3pivot 0.5‖PA − LU‖/‖A‖0largest multiplier0.75row order 4 3 2 1the pivot is chosen
Fig. 4 Where the entries come from. The foundation’s elimination figure shows the arithmetic of a single step; every entry it modifies in a position that held a zero is one unit of what this essay counts. The two figures are the same operation at two magnifications.
Elimination with and without pivoting, ε = 10⁻¹⁷The same two-by-two system solved twice. With a row swap the answer is exact; without one the upper triangular factor contains an entry of order one over epsilon and the second component of the answer is wrong.[ ε 1 ; 1 1 ] x = [ 1 ; 2 ], exact answer (1.000000, 1.000000)with partial pivoting1101U after elimination1.0000001.000000computed xbackward error 0forward error 0without10⁻¹⁷10-1·10¹⁷U after elimination0.0000001.000000computed xbackward error 0.25forward error 0.71no error is raisedgrowth 10¹⁷
Fig. 5 Why the clean symbolic prediction is a symmetric-positive-definite result. A general matrix must pivot for stability, the pivot chosen for stability is rarely the one the ordering wanted, and a row swap at step k changes the fill pattern of every step after it.
The arrowhead matrix, eliminated from each endThree sparsity plots. The first shows an arrowhead matrix with a dense first row and column. The second shows its Cholesky factor, completely dense. The third shows the factor obtained after moving the dense row to the end, which has no fill at all.the matrix43 entriestip eliminated first253 entriestip eliminated last43 entries‖A − LLᵀ‖/‖A‖, tip first1.4·10⁻¹⁶‖A − LLᵀ‖/‖A‖, tip last0dense factor is n(n+1)/2 = 253 · sparse factor is 2n − 1 = 43one row swapped to the endnothing numerical chose between them
Fig. 6 The extreme form of the same count. One matrix, one row moved from the front of the elimination order to the back, and the factor goes from every position in the lower triangle to exactly the entries the matrix had.
Conjugate gradients at κ = 104, against the bound κ permitsA semi-logarithmic plot of the relative A-norm error against iteration count. The measured curve falls below a smooth dashed curve showing the classical condition-number bound.0408012016020024010⁻¹⁶10⁻¹³10⁻¹⁰10⁻⁷10⁻⁴10⁻¹iteration‖e‖_A / ‖e₀‖_Ameasuredκ bound119 steps40×40, spectrum spread evenly in logbound permits 1417
Fig. 7 The alternative the fill count sends people to. An iteration costs one sparse product per step and stores a handful of vectors, so the comparison is the factor’s size against the step count — and the first of those is known before either is run.