The fall sets the undershoot

A turn is seen one trace late carried a trace estimator’s measured spread along a sequence of operators whose spectra drift and then turn back. The estimator is Hutchinson’s, run as a two-stage rule: a spread decides how many random probes the next trace needs for one standard error to be 3 per cent of it. The trend carry takes that spread from the previous trace and multiplies it by the ratio of the last two, so that a steady drift is extrapolated one step. At a turn the extrapolation is wrong for exactly one trace, and which way it is wrong depends on the turn: where the spread had been falling and turns up, the carry promises one more fall, uses a spread a quarter too small, and covers about nine points under the table.

That essay put the error in a formula — about the square of the last ratio, because the carry extrapolates one step of fall where one step of rise arrives — and used it to predict a case it had not run:

A sequence whose decay moves by 0.02 a trace and then reverses by 0.06 would make the error at the trough about $0.88 \times 0.69 \approx 0.6$ of the true spread; the prediction with a sign is that coverage on the trace after that turn falls below 55 per cent, that a fresh pilot on that trace alone — which no history-reading rule can know to take — would restore it, and that the sequence’s average still stays within two points of the table.

Every clause of that holds. The arithmetic inside it, though, is not the square the essay had stated as its rule; it is something the prediction’s own product computed without naming, and naming it is what turns one confirmed number into a rule for every turn.

A trough with a cliff on one side #

The operators are the earlier essays’: symmetric, 60 × 60, fixed random eigenvectors, eigenvalues $d^k$ for $k = 0, \ldots, 59$, so the decay $d$ is the only thing that changes along a sequence and the exact trace and the exact spread of a Rademacher probe are both known — the estimator counting what cannot be looked at introduced, with the convergence rate a rate that belongs to the matrix found fixed, so that one number, the spread, is all a probe count needs. The spread grows as the decay falls, because a faster-decaying spectrum concentrates the trace in fewer eigenvalues and a random probe’s estimate of a few dominant terms is noisier relative to the total.

The sequences are the earlier essay’s trough with its second half made steeper. The decay rises by 0.02 a trace from 0.74 to 0.9 at the ninth trace, so the spread falls by about twelve per cent a trace toward its minimum. Then it falls by 0.02, 0.04, 0.06 or 0.08 a trace, so the spread turns and climbs — by 14, 27, 40 and 51 per cent on the first trace after the turn. The fall of 0.02 is the earlier essay’s symmetric turn; 0.06 is the prediction’s.

The rules are four. A fresh pilot of eight probes on every trace. The plain carry, which uses the previous trace’s measured spread. The trend carry, which multiplies that spread by the ratio of the last two. And the prediction’s oracle: the trend carry, with a fresh pilot of eight probes on the one trace after the turn, as if something had told it the turn was there. Coverage is how often the trace estimate lands within 3 per cent of the truth, over 400 seeded sequences; the normal table says 68.3 per cent, and 400 draws scatter a single trace’s coverage by 2.3 points.

Every clause holds #

From the third trace to the turn the trend carry is within half a per cent of the true spread, which is what the earlier essays found for a steady drift; the plain carry runs five to fourteen per cent high, one step behind a spread that keeps falling. On the trace after the turn, with the decay falling 0.06, the trend carry uses 0.617 of the true spread. The prediction said about 0.6.

Its coverage on that trace is 48.5 per cent: under 55, as predicted, by more than two draw-noise errors. A fresh pilot on that one trace covers 71.0 per cent, inside the table’s draw noise, and the oracle’s clause holds too — at every fall, as it happens: 70.3, 66.8, 71.0 and 65.8 per cent, all within two draw-noise errors of the table. Averaged over the third to sixteenth traces, the trend carry covers 67.7 per cent against the table’s 68.3, inside the two points the prediction allowed.

The plain carry is caught as well. It had been running high, which a falling spread forgives, and meets a rise it was not extrapolating: 0.710 of the true spread on the trace after the turn and 51.5 per cent coverage, three points better than the trend carry and still under 55. A rule that reads only its own past cannot see the turn before the trace that reveals it, and that is true of the simplest such rule as much as of the extrapolating one.

The square was a coincidence of symmetry #

The earlier essay’s account of the error was that the carry extrapolates one step of fall where one step of rise arrives, so it is off by the last ratio twice: once for the fall it predicted and once for the rise it missed. On its symmetric turn that is exactly right, because the rise after the turn undoes the fall before it — the decay retraces its steps, and the spread’s one-step ratio after the turn is the reciprocal of the one before. A sharp turn breaks that.

The carry’s spread on the trace after the turn is the last spread times the last ratio, $s_9 \cdot r_9$, where $r_9 = s_9/s_8$. The true spread is $s_{10} = s_9 \cdot r_{10}$. Their ratio is $r_9 / r_{10}$: the last ratio before the turn over the first after it. Before the turn the true spread falls by a factor of 0.875 a trace; after it, it rises by 1.143, 1.274, 1.395 or 1.509 depending on the fall. So the undershoot should be 0.765, 0.687, 0.627 and 0.580, and the measured medians are 0.753, 0.676, 0.617 and 0.571 — within two per cent at every fall, the small remaining gap being the carried spread’s own sampling error. The square of the last ratio, 0.765, is right only at the symmetric turn, where $r_{10} = 1/r_9$ to two digits, and it overstates the fraction by a third at the fall of 0.08.

The prediction’s own arithmetic was already the quotient: $0.88 \times 0.69$ is the last ratio times the reciprocal of a rise of about 1.45. The rise is 1.395 rather than 1.45, so the product is 0.627 rather than 0.607, and the prediction landed within half a hundredth of the measurement for a reason its stated rule could not supply. Written as the quotient, the rule is no longer about turns at all: it is the error of a one-step extrapolation whenever the next ratio differs from the last, and a turn is the case where they differ most.

The table prices the dip #

Coverage is a noisy consequence of the spread used, and the earlier essay found it the worst quantity to measure a turn with: a single trace’s coverage over 400 sequences scatters by 2.3 points. The same symmetric trough it ran, with the same seeds and the same operators, covered 59.5 per cent there and 54.3 per cent here, because a fourth rule drawing probes from the same random stream shifts every rule’s draws. But the spread used is a median of 400 and barely moves, and once it is known the coverage it costs can be computed.

If a rule fixes its probe count from a spread that is a fraction $\rho$ of the truth, its standard error is $1/\rho$ times what it planned, and the 3 per cent target is $\rho$ standard errors wide instead of one. The normal table then says coverage is the probability that a standard normal lands within $\pm\rho$. At the four falls that is 54.9, 50.1, 46.3 and 43.2 per cent; the trend carry measures 54.3, 49.3, 48.5 and 45.5. Every one is within two draw-noise errors of the table at the fraction used, which is the sense in which the miss a normal table already priced applies here: the turn costs exactly what an undersized spread costs, and nothing about the turn adds to it.

That also says how much sharpness matters, and the answer is less than the spread suggests. From the symmetric turn to the fall of 0.08 the undershoot grows from a quarter of the spread to more than two fifths. Coverage falls by under nine points, from 54.3 to 45.5, because the table is flat near its middle: the first quarter of the spread lost costs fourteen points of coverage, and the next sixth costs nine more. The prediction’s 55 per cent line is not a property of the sharp turn; the symmetric turn already crosses it.

The same two steps price the plain carry, which is the check that the rule is about extrapolation and not about one rule’s habits. The plain carry uses the previous trace’s spread unchanged, so on the trace after the turn it uses $s_9$ where the truth is $s_9 \cdot r_{10}$: a fraction $1/r_{10}$ of the true spread. At the four falls that is 0.875, 0.785, 0.717 and 0.663, and the measured medians are 0.866, 0.777, 0.710 and 0.656 — the same one per cent short, for the same reason, the carried spread’s own sampling error. The table at those fractions gives 61.4, 56.3, 52.2 and 48.8 per cent; the plain carry measures 63.7, 54.8, 51.5 and 52.8, each within two draw-noise errors. So the plain carry loses less at the turn than the trend carry does, by exactly the factor $r_9$ the trend carry multiplied in, and the earlier essays’ whole comparison of the two carries reduces to which ratio each assumes the next trace will have: one, or the last.

What the average absorbs, and what the oracle costs #

One trace in fourteen is a fourteenth of the average. The trend carry’s mean coverage is 67.9, 67.3, 67.7 and 67.6 per cent at the four falls; the plain carry’s 68.1 to 66.4; the fresh pilot’s 66.9 to 67.1. The oracle, with its single extra pilot, is the closest to the table at 67.8 to 68.8 per cent. A turn, however sharp, is a one-trace event to a rule that recovers a trace later, and the trend carry does recover: on the eleventh trace it uses 1.091 of the true spread at the fall of 0.06, now extrapolating the rise it has just measured into a rise that has slowed, and is within two per cent by the thirteenth. The overshoot has the same form as the undershoot one trace on: the carry now uses $r_{10}$ where the truth is $r_{11}$, and $r_{10}/r_{11}$ is 1.026, 1.075, 1.131 and 1.187 at the four falls. The measured medians are 1.012, 1.054, 1.091 and 1.142 — four per cent low at most, the carried spread’s sampling shortfall now entering twice, through the spread and through the ratio. The turn’s cost to a sequence is therefore two traces of error with opposite signs, and only the first costs coverage; the second costs probes, because an oversized spread asks for more of them than the target needs.

The cost tells the rest. After the turn the spread rises on every trace, and a rising spread is expensive whatever the rule: from 0.264 at the turn to 0.480 on the last trace at the fall of 0.02 and to 0.958 at 0.08, and the probes a trace needs grow with its square. At the fall of 0.06 the trend carry spends 4,589 probes over the sequence against 4,060 for the plain carry, which lags behind the rise and so under-spends, and 4,388 for a fresh pilot every trace, whose eight extra probes a trace are cheaper than the trend carry’s overshoot on the traces after the turn. The oracle spends 4,720: its fresh pilot at the turn costs 160 probes on that trace where the trend carry spent 59, and buys back twenty-two points of coverage on the one trace that needed them. Across the falls the oracle costs two to four per cent more than the trend carry. The prediction called it the pilot “no history-reading rule can know to take”; what the measurement adds is its price, and that the price is small because only one trace pays it.

A drift put back at the table’s price built the trend carry to remove a steady drift’s lag, and a spread carried from the trace before built the plain carry it improves on. The quotient rule is the account of both at once: the plain carry’s error on any trace is $1/r$ of the next ratio, the trend carry’s is the last ratio over the next, and either is priced in coverage by the same table.

What sixteen operators do not show #

The spectra here are geometric, so the spread is a smooth function of one parameter and its one-step ratios move slowly except at the turn. A spectrum that changes shape — an eigenvalue crossing others, a cluster splitting — would change the ratio from one trace to the next without any turn in the decay, and the quotient rule would price that as well, but these runs do not test it. The turn is a single trace; a turn spread over two or three traces would give the trend carry a smaller error on each and might cost less coverage in total. And the oracle’s fresh pilot is eight probes; whether four would restore the trace, or whether sixteen would be needed at a sharper fall, is the question of what the pilot is buying, which a rule that reads only its own probes asked of the first trace and nobody has asked of the turn.

Still open: a turn spread out, a pilot priced by the turn, and the contour #

A turn over three traces. Let the decay’s fall after the turn ramp from 0.02 to 0.06 over three traces instead of jumping. The prediction with a sign is that the trend carry’s undershoot on each of those traces is the last ratio over the next, none below 0.8 of the true spread, and that the coverage lost summed over the three traces is less than the 19.8 points the single sharp turn costs.

A pilot sized by the measured surprise. After the trace that reveals the turn, the measured ratio of the carried spread to the new one is known. The prediction is that a rule which takes a fresh pilot only when that measured surprise exceeds 1.25 takes it on fewer than one trace in ten along these sequences, and keeps every trace’s coverage within two draw-noise errors of the table from the second trace after the turn on.

The contour. Counting what is inside a circle estimates a trace whose exact value is an integer, and a spectrum drifting across the contour changes it between traces. That is a turn in the trace rather than in its spread, and the prediction stands as the earlier essay made it: the ratio of the last two spreads stays near one across it, so neither carry sees the jump and both miss its trace by the jump’s size.