Numerical Linear Algebra
where the algebra stops being the arithmetic
Concept

Latency — where it appears

The fixed cost of a message regardless of its size, which makes the number of communication rounds a separate currency from the number of words.

Named by 2 essays across one field — each of them below, with the objects they name alongside it.

rounds on the critical pathHouseholder sweep48reduction tree4Cholesky QR4words sentHouseholder sweep1170reduction tree1170Cholesky QR2160two counts, two rankingsrounds, sweep ÷ tree12words, Cholesky ÷ tree1.8arithmetic, tree ÷ sweep1.5the rounds separate the threeand the words do not

The message and the word

Three factorisations of one matrix on sixteen processors: 48 communication rounds, 4, and 4. The words sent are 1,170, 1,170 and 2,160 — so the method with the fewest rounds sends the most words, and the count that separates the three is the one no operation count can see.

cost · communication 10²11.251.51.7522.25processorstraffic saved, as a factorthe √4 the law promisesbreak-evenmeasureda limit is not a sizesaving at p = 640.88saving at p = 5761.4what the law promises2memory, as a factor4a loss at sixty-four processorsand 72% of the law at five hundred

Memory bought with messages

Holding four copies of the data instead of one is supposed to cut a matrix multiplication's communication by √4. Measured on a machine of 64 processors it costs 14% more traffic; at 576 it saves 44%, which is 72% of what the law promises. The memory is exactly four times, and that part is not asymptotic.

cost · communication

Named alongside it

The objects these essays reach for when they reach for this one.

All reduceAsymptotic analysisBlock transferBroadcastCommunication avoidingCommunication lower boundCondition numberData movementHouseholder reflectionMatrix multiplicationMemory hierarchyNormal equations

All concepts