A matrix decomposition hands over orthonormal factors and a diagonal middle at once. For three indices the two come apart, and there is no arrangement that has both — so the question stops being which decomposition to use and becomes which of the two properties the computation needs.
A singular value decomposition delivers two things at the same time, and because it delivers them
together nobody has to decide which one they wanted.
The factors are orthonormal: U and V have unit columns and no two of them are correlated, so the
decomposition is a change of basis that no norm can see. The middle is diagonal: one number per
direction, and the whole matrix is a sum of r independent rank-one pieces that do not interact.
Every use of the decomposition on this site leans on one or the other. Truncation leans on the second,
because dropping a diagonal entry drops a whole term. Stability leans on the first, because an
orthogonal change of basis does not amplify anything. For three indices the two come apart, and this
essay is about which of them survives.
Fig. 1 How much of a core’s energy sits on the superdiagonal a diagonal reading would need. On a tensor with no
structure at all it is 0.4 per cent.
The higher-order SVD’s factors are orthonormal by construction — they are left singular vectors — and
its core inherits a real property from them, which is not diagonality and is not nothing.
Fix a mode k and slice the core perpendicular to it: for a three-index core that is r_k matrices, one
for each value of the k-th index. Those slices are mutually orthogonal, in the sense that the inner
product of any two of them, taken entry by entry, is zero. And their norms are decreasing, and they are
the singular values of the k-th unfolding.
That is called all-orthogonality, and it is the honest generalisation of the middle factor is
diagonal: for a matrix, the slices perpendicular to either mode are single rows or columns of a
diagonal matrix, and two of them are orthogonal for the trivial reason that their supports are disjoint.
For a tensor the supports are not disjoint and the orthogonality is a real statement.
It is checked here rather than cited. The largest inner product between two slices of a core, relative
to the core’s own energy, is below 10⁻⁹ on every family measured.
Fig. 2 The same measurement at rank four, where the compressible families concentrate more of their energy on
the diagonal and the incompressible one does not move.
One detail of that measurement is worth a sentence because getting it wrong makes the property look
false. The inner product between two slices has to be scaled by the core’s norm rather than by the two
slices’ own norms. A core with a decaying spectrum has trailing slices at the rounding level, and the
cosine between two vectors of norm 10⁻¹⁶ is a cosine between two roundings — O(1), and meaningless. The
first version of this measurement reported a worst pair of 7·10⁻⁴ for exactly that reason.
Fig. 3 The orthogonality field’s version of the same discipline: an inner product that is a measurement rather
than an adjective, reported entry by entry.
A core is not diagonal, and the question worth asking is not whether but by how much.
Take the leading r × r × r block of the core and ask what fraction of the tensor’s energy sits on its
superdiagonal — the r entries with all three indices equal. Everything else would have to be discarded
by a decomposition that insisted on a diagonal middle.
family
on the diagonal
error of a diagonal core
error of the full core
smooth
97.7%
0.152
7.9·10⁻⁶
hilbert
94.5%
0.235
1.8·10⁻⁶
wave
26.6%
0.857
1.3·10⁻¹⁵
noise
0.4%
0.998
0.842
The smooth families put most of their energy on the diagonal and still pay a factor of 20,000 in error
for the last few per cent of it. The wave family — which is exactly multilinear rank two, and which the
full core reproduces to the rounding level — puts a quarter of its energy there, so a diagonal reading of
a tensor the format handles perfectly loses 86 per cent of it.
The last row is the one that settles the general question. A tensor with independent normal entries has
0.4 per cent of its energy on the superdiagonal of its own core, because there is no reason for it to
have any.
Fig. 4 At rank ten, where the diagonal window is wider and the share on it falls for every family — because the
entries being added are off it.Fig. 5 And at rank two, where it is highest, which is the shape of the whole measurement: the diagonal is a
line through a cube and its share of the volume shrinks.
It is worth writing down what is being asked for, because the request sounds modest until it is counted.
A decomposition with orthonormal factors and a diagonal core of length r says
T = Σ_j σ_j · u_j ⊗ v_j ⊗ w_j
with the u’s, the v’s and the w’s each orthonormal sets. Count the parameters. Each factor matrix is
n × r with orthonormal columns, which is nr − r(r+1)/2 free numbers, and there are d of them, plus r
singular values. For n = 12, d = 3 and r = 12 that is 3·(144 − 78) + 12 = 210.
A general 12 × 12 × 12 tensor has 1,728 numbers. So the set of tensors admitting such a decomposition
has dimension 210 inside a space of dimension 1,728, which is a measure-zero subset of it — and a
measure-zero subset is exactly what “almost no tensor has one” means.
For a matrix the same count gives 2·(144 − 78) + 12 = 144, and a 12 × 12 matrix has 144 numbers. The
dimensions match, which is the counting version of the statement that every matrix has an SVD. There is
nothing subtle in the difference: for d = 2 the parametrisation is exactly the right size and for d = 3
it is a fraction of the right size, and the fraction shrinks with d.
Fig. 6 What a measure-zero condition looks like when it is measured rather than argued, from earlier in this
field: a property that occurs with probability zero and a property that occurs with probability π/4.
Naming the alternatives makes the trade explicit, because both exist and each keeps one of the two
properties.
Keep orthogonality, give up diagonality. That is the higher-order SVD: orthonormal factors,
all-orthogonal core, and r^d numbers in the middle. Its truncation is quasi-optimal, its cost is d matrix
decompositions, and the previous essay measures both.
Keep diagonality, give up orthogonality. That is the CP decomposition: a sum of r rank-one terms,
which is a diagonal core of size r, and factor matrices that are not orthogonal and generally cannot be.
Its rank is the object the first two essays of this field are about — the one whose best approximation
need not exist and whose value depends on the field.
There is no third option that keeps both, and the reason is a counting argument rather than an accident.
An orthogonal decomposition with a diagonal core of size r has r(n₁ + n₂ + n₃ − 2) parameters after the
orthogonality constraints are imposed, and a general tensor has n₁n₂n₃; the first is far smaller than the
second for any r that keeps the factors orthonormal, so almost no tensor has one.
Fig. 7 What the other choice costs, from the field’s essay on it: an iteration over the diagonal-core model,
whose factors grow without bound when the target’s best approximation is not there.Fig. 8 And what it buys, which is the reason anyone takes it: factors that are recoverable rather than
arbitrary.
The trade is only interesting because different uses want different halves, and it is worth going
through the uses this collection has.
Compression wants orthogonality. A projection onto orthonormal factors has an error that is
computable from the discarded singular values, and the bounds of the previous essay are made of them. A
non-orthogonal factorisation has no such accounting: its error is whatever the fit achieved.
Interpretation wants diagonality. A rank-one term is a component, and a component is only a
component if it stands alone. A core with entries off the diagonal says that direction three of mode one
interacts with direction two of mode two, which is a true statement about the tensor and an unusable one
about the data.
Arithmetic inside the format wants orthogonality. Adding two representations, applying an operator,
truncating the result — all of it is stable when the bases are orthonormal and none of it is when they
are not, which is the same reason the orthogonality field exists at all.
So the answer to which decomposition is decided by what is being done, and the two are not competitors
for a single job. Where they are competitors, the measurement above says the cost of insisting on
diagonality is between a factor of twenty thousand and a factor of a hundred.
Fig. 9 Why the third of those uses is not negotiable, from the orthogonality field: two algorithms with the
same algebra, one of which loses its basis.Fig. 10 And the same loss printed entry by entry, which is what an all-orthogonality measurement is a three-index
version of.
One further property comes across and one does not, and they are easy to confuse.
The core’s slice norms are ordered — the first slice perpendicular to mode k has the largest norm, the
second the next largest, and so on — and those norms are the singular values of the k-th unfolding. So
each mode has a spectrum, and the picture of a decaying sequence that every truncation argument on this
site relies on is available.
What is not available is a single spectrum for the tensor. There are d of them, they are different
lengths, and there is no canonical way to merge them: dropping the last direction of mode one and
dropping the last direction of mode two remove different amounts of energy, and neither is comparable
with the other except through the sum in squares that the bound uses.
The consequence is that the phrase the k-th singular value of a tensor names nothing, which is why the
truncation is specified by a triple of ranks rather than by a single cut-off. Every figure in this field
that draws a spectrum draws d of them for that reason.
Fig. 11 Three spectra on one pair of axes, each normalised to its own leading value, because there is no scale
on which comparing them across modes would mean anything.Fig. 12 And the decision each of them requires, from the spectra field — taken three times here, and once for a
matrix.
It is worth going back and asking what the matrix case is actually doing, because the answer is not that
matrices are better behaved.
A matrix has two modes. Its core, in the language above, is r × r, and all-orthogonality of an r × r
core forces it to be diagonal: the slices perpendicular to mode one are its rows, the slices
perpendicular to mode two are its columns, and requiring all pairs of rows and all pairs of columns to be
orthogonal leaves only diagonal matrices.
So diagonality is not an extra property the SVD provides. It is a consequence of all-orthogonality,
available only when d = 2, and it disappears the moment there are three sets of slices to be mutually
orthogonal rather than two.
That is the cleanest statement of what this field’s break actually is. Nothing was lost; a coincidence
stopped applying. And the coincidence is the reason every decomposition anyone learns first has both
properties without either being requested.
Fig. 13 The consequence of the coincidence, from the spectra field: an error that is a single discarded number,
because there was only one number per direction to discard.Fig. 14 And what replaces it here: an error made of three tails, whose sum in squares is the bound.
Two arguments elsewhere on this site are quietly the same argument as this one, and putting them beside
it is worth doing because none of the three is about tensors.
The hierarchy field’s recompression essay finds that a sum of two rank-k blocks is a rank-2k block
exactly, so a format built on rank-k blocks is not closed under its own addition and every operation
must be followed by a truncation. That is the same shape: the representation has a property the
arithmetic does not preserve, and what is preserved instead is a bound on how much is lost.
The least-squares field’s normal equations essay finds that a route which is algebraically identical
squares the condition number. Again the same shape: two constructions agree on what they compute and
differ on what they preserve.
And the orthogonality field’s two Gram–Schmidts is the original. Same algebra, different arithmetic,
and the difference is measured as ‖QᵀQ − I‖ rather than as a residual, because the residual does not see
it.
What this page adds to the family is that the property being lost is not lost to rounding. Every
measurement here is exact to the rounding level; the core is genuinely not diagonal, in exact arithmetic,
for reasons of dimension. That makes it the cleanest member of the group: nothing about precision,
nothing about an implementation, and a fact that survives being computed in any arithmetic at all.
Fig. 15 The hierarchy field’s version, from the essay that measures it: a rank that doubles under addition, and
the truncation that has to follow it.Fig. 16 And the original, from the orthogonality field: two orders of the same arithmetic, and a measurement
that separates them.
The claim under test is the name. This decomposition is called the higher-order SVD, and the natural
reading of that name is that its core plays the role a matrix’s middle factor plays.
So the assertion that the off-diagonal share of a smooth tensor’s core is below 10⁻⁸ is fed the measured
value. It fails — the share is 2.3 per cent on the family where it is smallest, and 99.6 per cent on the
family with no structure.
Refusing it matters because the name is the whole of the mistake. Nothing in the construction claims
diagonality and nothing in the theorem claims it; what claims it is the analogy the name invites, and an
analogy is not a thing a gate can fail. Feeding the assertion the number is the only available way to
make the claim testable at all.
The file’s other refusals guard the two neighbouring readings. One is fed a rank-two truncation of a
noise tensor and required to refuse the claim that it is exact — the untruncated decomposition is, and a
truncation of it is not. The other is fed the ratio of the truncation error to the lower bound and
required to refuse the claim that the projection attains it, which is the previous essay’s subject.
Fig. 17 Rank three, between the two settings the argument is made at.Fig. 18 And rank eight, where the diagonal window is wide and the compressible families have started to fall.Fig. 19 The family whose core is exactly rank two and whose diagonal reading still loses 86 per cent of it.Fig. 20 And the family where nothing works, which is the row of the table that settles the general question.Fig. 21 The three matrices behind that row, each of rank two — an array that separates at every index and still
has a dense core.Fig. 22 What the dense core costs as indices are added, which is why the next essay exists.Fig. 23 And the format that keeps orthogonality, gives up the dense core, and cuts between the indices instead
of across them.Fig. 24 The orthogonality field’s own object, for scale: a transformation that cannot stop being orthogonal,
which is the property this page says survives.Fig. 25 And the nearest orthogonal matrix to a given one, from the essay that computes it — the matrix question
whose tensor analogue does not exist.Fig. 26 The other half of the trade, from three essays back: the geometry of the diagonal-core model, and where
it stops having a nearest point.