A pre-registered test of whether rank persisted in a public forecasting competition — and what I found wrong with my own registration
My first essay here was about a trading system I built, tested, and shut down when the test said its confidence score carried no information. That was an Assay on my own work. This is the first one on somebody else's results: the M6 financial forecasting competition, using the organizers' public data.
The question is narrow. When a team ranks near the top of M6 in the first six months, does it stay near the top in the next six? If rank is mostly luck, it shouldn't. If it's skill, it should.
On forecasting, rank carried over, strongly. On investment decisions, it did not. And the registration I froze before starting did not describe the test I actually ran. All three are below.
M6 ran from February 2022 to February 2023. Teams forecast the relative returns of 100 assets — 50 S&P 500 stocks and 50 international ETFs — as probabilities over five return quintiles, and separately submitted portfolio weights. Forecasts were scored by ranked probability score (RPS); portfolios by information ratio (IR); and an overall rank (OR) combined the two. Prizes were awarded on all three tracks.
The organizers built M6 around a question: are forecasting skill and investment skill the same thing? Their published results, in the International Journal of Forecasting, report a limited connection between the forecasts teams submitted and the investment decisions they made. That finding is theirs. What I did is smaller and different: test, against a claim frozen in advance, whether each track's ranking is distinguishable from luck when you check it on a later, unseen period.
Before testing anything, count the teams. The organizers' submissions file contains 251 team IDs. Twenty-five of them are flagged inactive in that same file and never received a score. That leaves 226, which is the number the organizers report as having entered. Of those, 163 joined at the start and appear on the official global leaderboard; most of the other 63 joined in a later quarter.
So "the M6 field" is 251, 226, or 163 depending on which file and which sentence you read, and 88 of the 251 IDs never appear on the leaderboard any headline would cite. None of this is a defect in the competition — eligibility rules are normal. But a claim about "M6 performance" that doesn't say which population it means is underspecified. Mine was.
Split the twelve monthly evaluations in half: months 1–6 and months 7–12. For each team with at least three scored months in each half — 197 teams — average its monthly score in each half and rank the teams. Then ask how well first-half rank predicts second-half rank, and compare that to what you'd see if the team labels were shuffled 20,000 times.
A first-half ranking is a claim about skill. The second half is the holdout that grades it.
| Track | Rank correlation, half 1 → half 2 | 95% interval | Chance of this under luck |
|---|---|---|---|
| Forecasting (RPS) | 0.695 | 0.58 to 0.79 | < 1 in 10,000 |
| Investment (IR) | 0.055 | −0.10 to 0.21 | 22% |
| Overall (OR) | 0.644 | 0.54 to 0.73 | < 1 in 10,000 |
197 teams. Permutation test, 20,000 shuffles, seeded; the smallest attainable value is 1 in 20,001. Intervals are team-level bootstrap. OR was not in the registration and is reported separately.
Forecasting rank persisted. Investment rank did not, at least not distinguishably from luck: a correlation of 0.055 is what shuffled labels produce about a fifth of the time.
A correlation of 0.7 on this kind of data made me suspicious, and there was an obvious way it could be fake. A quarter of all team-months sat at almost exactly the same RPS, about 0.160 — exactly what equal probabilities for every quintile score. If a large block of teams did that every month, they would rank consistently in the middle of the pack, and the correlation would be measuring persistent disengagement rather than persistent skill.
So I removed them. 109 of the 197 teams looked static by that standard. On the 88 that remained, the forecasting correlation was 0.648. It barely moved.
The bulk correlation also isn't quite the claim. The claim is about the top. So I took the first half's top decile, 19 teams, and asked where they landed in the second half, compared with 20,000 random groups of 19. On forecasting, the leaders averaged rank 56 of 197; a random 19 averages 99. On the overall rank, they averaged 37. Neither result is close to what luck produces.
The discipline in an Assay is that the claim is frozen before the data is examined. I did freeze one, on August 16, before looking at any scores:
"Top-ranked M6 performance is distinguishable from luck after multiplicity correction across the full 251-team population."
When I prepared this for publication and reread the registration file next to my draft report, they didn't match. The draft said the split-half design, the three-months-per-half threshold, and the permutation test had been registered. The file says none of that. It specifies comparing the top team or top decile against an equal-skill null with a family-wise or false-discovery correction, across all 251 teams, on RPS and IR.
I had settled on the split-half design before running anything, but I never wrote it down. So by my own rules it is a design chosen after registration. Overall rank, which I added because it's the combined prize track, isn't in the registration at all.
Two things follow. The registration itself says any revision after the fact caps the verdict at "supported with conditions," so that's where the verdict sits. And the result is reported the way the schema demands: the registration file is published unmodified next to the report, the deviation is logged as a major finding, and overall rank goes in a separate table marked post-registration.
It's a fair question whether the deviation matters. I think the split-half test is a stricter answer to the registered question than the registered method: it asks whether a ranking predicts an unseen future period, not whether one full-year rank looks surprising in hindsight. But "I think the test I picked later was better" is exactly the argument pre-registration exists to stop people from making. The right move is to say it, show both documents, and let the verdict carry the condition.
There was one more catch, earlier. The first population count found zero teams missing from the leaderboard — because the organizers' "NA" marker had been parsed into a numeric null before a string comparison ran, so the check could never match. The number was wrong and looked clean. It was caught by inspecting the column rather than trusting the count, and it's disclosed in the report.
Established: in M6, a team's forecasting rank over the first six months predicted its forecasting rank over the next six, far beyond what luck produces, and the effect survives removing teams that submitted defaults. The first-half forecasting leaders stayed well above a random draw. Investment rank did not carry over distinguishably from luck.
Not established: that investment skill doesn't exist. A single month's information ratio is a noisy number, and the interval on the investment correlation runs up to about 0.2 — modest persistence isn't ruled out. Nor does this say why forecasting rank persists and investment rank doesn't; the organizers' own analysis goes further on that than this does. It says nothing about any individual team's process, and it tests one competition over one year.
Verdict: Supported with conditions — forecasting rank persistence is real under a test chosen after the claim was frozen; investment rank persistence is not supported.
M6 is public, closed, and already well studied. That's the point. It's a case where anyone can check my work: the registration, the reproduction script, the seed, and the input hashes are published, and the script refuses to run if the data doesn't match. On September 24, I re-ran it from a fresh clone of the organizers' repository and got every number in the report back exactly.
The two Assays so far point in opposite directions — a null on my own system, a supported-with-conditions result on someone else's — and the most useful finding in this one is about my own paperwork. That's the standard I'd want from anyone grading a model I cared about.
Assay is a fixed-scope, independent examination of whether a model's results support the claim being made about it.