Companion to "The Confidence Score Wasn't Confidence." All values below trace to the frozen evidence package (tag evidence-package-v1, branch tip e3c4ec0, implementation commit 65c8dc25f04b), whose manifest lists content hashes for every data snapshot, study-plan document, and the golden-case file.
Entry reference (T0). For every signal, T0 is the open of the next 1-minute bar strictly after the signal timestamp, computed against a full exchange calendar. Signals arriving outside regular trading hours roll to the open of the next regular session. T0 is recomputed for all signals — traded and rejected alike — from a single historical-bar source; live captured entry prices are never reused.
Horizons and endpoint conventions. Four, session-aware, measured from T0. One hour after T0, measured at the close of the bar sixty minutes after T0 (clamped to session close where necessary and flagged); the close of T0's session; the close of the following session ("next-session close," internally 24h); and the close of the fifth session (internally 5d). "Session close" as implemented is the close of the minute bar stamped at 16:00 ET — the auction-period minute — not the official consolidated closing price. The two usually agree to the cent; on one earnings-eve close in the golden set they diverged by $0.52 (~17 bps). This convention was identified empirically during golden-case reconciliation and is documented rather than changed, per the freeze discipline.
Outcome measure. With d = +1 for BULLISH and d = −1 for BEARISH, the signed return over horizon h is
R_signed = d × ( P_h / P_T0 − 1 )
and the SPY-adjusted figure is
R_adj = d × [ ( P_h / P_T0 − 1 ) − ( S_h / S_T0 − 1 ) ]
where S is SPY over the identical T0→h window. This is a β = 1 subtraction, not a fitted regression. The book-mix context: the full population is 2,575 bullish / 1,240 bearish (67.5% long) and the ≥0.95 gate is 530 / 74 (87.7% long); the equal-weight mean trailing 60-session realized beta of the signal universe is 1.152 (n = 2,991; 824 unavailable); the rank correlation between confidence and beta is +0.176; and SPY's total return over the study window was +53.0%. A long-dominated, above-market-beta book with confidence positively correlated to beta, measured against a β = 1 benchmark in a rising market, means the subtraction under-corrects: the measured SPY-relative figures are, if anything, flattered, and a beta-matched benchmark would be unlikely to improve the picture.
Price adjustment. Historical bars carry cumulative split-and-dividend adjustment factors applied identically to each stock and to SPY. The dividend step at an ex-date multiplies all earlier dates by (1 − D/P_prev), where P_prev is the close of the final extended-session bar of the prior trading day — a second internal daily-close convention, distinct from the 16:00-bar session close used for markout endpoints, also identified via golden reconciliation and documented here. The stored t0_price is on a globally adjusted basis: it reflects every in-range ex-date after T0 through the market-data cutoff, even ones outside the signal's own horizon windows. Return computations are unaffected except where an ex-date falls inside a window (the factors cancel otherwise). A split-only sensitivity confirms the adjustment layer barely moves the rank statistics: the split-only versus total-return Spearman differs by at most 0.0009 at any horizon, and gate means by at most 0.09 bps.
Signal population: 2024-02-18T05:00:20Z through 2026-06-08T05:09:50Z. Raw market-data cutoff: 2026-06-08 23:59 UTC (latest cached SPY minute); the latest scheduled fifth-session endpoint is 2026-06-15, with later horizons handled as terminal-print rows. Ledger snapshot: ledger-pre-pr25-pnlfix-20260605 (2026-06-05 20:23:54 UTC). All timestamps parsed timezone-aware UTC; session arithmetic uses the exchange calendar; beta uses 09:30–16:00 America/New_York daily closes keyed by exchange-local session date.
| Stage | N |
|---|---|
| Production raw rows in window | 29,996 |
| Enrichment rows in window | 9,434 |
| After exact processing-key dedup (constraint-enforced at write; zero removed) | 9,434 |
| Same-event consolidation | none applied |
| Frozen directional signals | 5,990 |
| Equities | 3,815 |
| Crypto (counted, excluded from markouts) | 2,175 |
| Final adjusted T0-resolved equities (per horizon) | 2,976 |
The exact processing key is (platform, post_external_id), enforced as a unique constraint with upsert semantics on both raw and enriched tables; a pre-LLM existence check uses the same key and fails open on database errors. No same-event consolidation is applied to the enrichment table; the 60-minute ticker/direction window governs the downstream alert cache only. The raw-to-enriched gap reflects posts that never became directional outputs, not duplicate removal. One post-freeze crypto backfill row (signal 55861, created after the snapshot) makes the live table read 5,991; it is not in the frozen cohort.
Provenance of the 962-duplicate incident. The frozen enrichment population shows zero duplicate groups under the bare external identifier and under (platform, external identifier). Sixteen semantic (source, ticker, posted_at) groups and thirteen content-hash groups exist — distinct external posts relaying the same content — and none has members on both sides of the May 2 cleanup date. The May 2026 concurrency duplication, documented in the trading cache and Ledger path, did not enter the study population.
Symbol-validity limitation. A conservative local screen (non-canonical syntax, whitespace, foreign/futures formats, plus the directly observed non-equity symbols EUR, XAU, SPX, JGB) fails 87 of the 3,815 published rows; 86 of the 87 fall inside the 891 initially-missing-T0 signals, confirming that such symbols fail entry-bar resolution by construction. The screen is a lower bound — a dated exchange-master sweep was not performed — and published population counts are unchanged; the discrepancy is reported here as a limitation.
In the initial run, 891 equity signals (~23%) lacked a resolvable T0 bar (539 bullish, 352 bearish, concentrated in two public newswire sources). Final coverage: 2,976 T0-resolved equities with non-null adjusted observations at every horizon (11,904 benchmark-adjusted markouts total); 839 remain unresolved. The ≥0.95 gate population is 604 signals, 570 with non-null outcomes. Of the 38 gate signals initially missing T0: 4 recovered, 32 unrecoverable (no resolvable bar), 2 terminal-print. Ignorability of the missingness mechanism is not formally established; the symbol screen shows at least 86 of the missing rows are structurally unpriceable rather than informatively censored, but that covers roughly a tenth of the missing set.
This was a sequence of studies, not a single frozen protocol; each run's plan was written before that run's results were viewed. Run 1's plan pre-specified the gate-versus-zero endpoint; its results exposed the price-adjustment and threshold-selection problems. The full-population rank correlation was specified prospectively for Run 2 as a family across four horizons with no designated primary. Run 3's rescue hypotheses, time-ordered 70/30 holdout, and acceptance criteria were frozen in a written preregistration before evaluation. Run 4 tested the latency hypothesis at its most favorable configuration.
Artifact status, stated plainly: contemporaneous artifacts with original timestamps exist for Run 1 (PR #40, opened and merged 2026-06-05, alongside the repository PR-description file) and Run 3 (preregistration.md, 2026-06-09, pre-evaluation). The Run 2 plan survives as a PR-description file whose content is hashed in the manifest but whose file timestamp post-dates the run; the original prompt file was not retained. No file artifact of the Run 4 plan was retained; the Run 4 contemporaneous record is its report (hashed in the manifest, file timestamp 2026-06-09 06:27 UTC) and PR #44 (opened 2026-06-09 06:30 UTC, a third-party timestamp), whose findings match the report and this appendix. All hashes are in MANIFEST.md.
No reusable semantic-dedup key exists in the stored population (snapshot_id is unique wherever populated). The cluster rule is ticker × T0 session × direction with anchored 30-minute windows: 3,214 clusters over the resolved population, 88.9% singletons, mean size 1.19, maximum 12 (a DJT event on 2026-01-05). The largest clusters are dominated by DJT, NVDA, XLE, AMZN, and HMC events — multiple feeds and reposts relaying single underlying stories.
Bootstrap intervals are non-studentized percentile 95% intervals, 10,000 resamples per method; event-cluster resampling seed 20260822; circular session-block resampling (blocks of 10 consecutive ordered T0 sessions, wrapping) seed 20260823. Spearman is computed as Pearson correlation of average-method ranks. Library versions are frozen in the manifest (numpy 2.4.6, pandas 3.0.3, pyarrow 24.0.0, Python 3.12.8).
| Estimand | Horizon | Pooled | Cluster 95% CI | Block-10 95% CI |
|---|---|---|---|---|
| Spearman ρ | 1h | −0.0282 | −0.0750 to +0.0183 | −0.0767 to +0.0299 |
| Spearman ρ | session close | −0.0113 | −0.0583 to +0.0356 | −0.0510 to +0.0415 |
| Spearman ρ | next-session | +0.0265 | −0.0222 to +0.0744 | −0.0182 to +0.0701 |
| Spearman ρ | fifth-session | −0.0535 | −0.0987 to −0.0078 | −0.1243 to +0.0206 |
| Gate mean (bps) | 1h | −17.18 | −36.71 to +1.87 | −36.51 to +3.82 |
| Gate mean (bps) | session close | −2.20 | −27.21 to +22.04 | −20.63 to +17.18 |
| Gate mean (bps) | next-session | +7.02 | −33.53 to +45.89 | −21.26 to +44.73 |
| Gate mean (bps) | fifth-session | −61.13 | −119.51 to −3.32 | −173.11 to +34.05 |
The pre-registered decision rule required the fifth-session interval to exclude zero under both resampling schemes for the "significantly negative" label. The session-block interval crosses zero; the label is accordingly retired, and the fifth-session result stands as a negative point estimate. The same logic applies to the fifth-session gate mean.
Deduplicated point estimates (estimand sensitivity). With one canonical observation per cluster (earliest) or cluster means, the fifth-session ρ moves from −0.0535 (pooled) to −0.0316 / −0.0310 — a shift of 0.0225, exceeding the pre-registered 0.02 tripwire. Repeated events materially reweight the fifth-session estimate: both estimands are negative, but the pooled figure reflects production exposure (every emitted signal) while the deduplicated figure reflects event-level information content, and they are not interchangeable. No other horizon exceeds the tripwire (max 0.018 at 1h).
The marketed gate was applied across production versions, so the pooled result tests the deployed commercial claim as customers experienced it. Stored version identifiers cover most of the population (gemini-2.5-pro: 2,818 signals across two prompt-version states; unstored: 797; gemini-2.5-flash: 106; claude-sonnet-4-6: 52; claude-opus-4-6: 42); judge_prompt_version is unstored for the large majority, a documented limitation. No version cell shows robust positive discrimination: the two largest cells (n = 1,314 and n = 824) have fifth-session ρ of −0.009 and −0.081, and every cell with n < 30 is flagged as exploratory.
Hand-computed reference cases are supplied from outside the pipeline, and the pipeline must reproduce each within ±0.5 bps (returns), 1e-9 (T0 price), and exact UTC match (T0 timestamp), or the run is void. Chronology disclosure: this gate was specified in writing from the first study plan, and the harness enforced its no-fabrication rule throughout — but the operator-supplied case file was not produced until the final evidence freeze, at which point the harness halted the publication run until it existed. Runs 1–4 therefore rested on synthetic-bar unit tests and cross-run consistency, a weaker integrity guarantee, disclosed here.
Final result: the 20-row file (16 priced rows across four signals spanning an in-hours entry, a pre-market roll, a bearish sign test, and an ex-dividend-crossing window; 4 rows asserting unrecoverable behavior on a delisted ticker) passed 20/20 against the frozen build. The unrecoverable rows exercised the dedicated harness branch: the pipeline reported missing_t0_bar and unrecoverable true with null prices at all four horizons while still resolving the calendar T0 deterministically. Reconciling the priced rows required identifying three implementation conventions documented in the Definitions section (the 1h bar-close endpoint, the 16:00-bar session close, and the extended-session dividend prior close); every golden value was then independently recomputed from raw REST data under those conventions. Golden cases verify implementation integrity — that the pipeline computes what it claims — and in this study they additionally surfaced the precise statement of what it claims. They do not validate the statistic, benchmark, or hypothesis. Golden file hash: 649b0ea6… (manifest).
The production gate was re-derived as confidence ≥ 0.95 on equities. Runtime-state rules — open-position deduplication and opposite-direction suppression — could not be reconstructed from stored fields and are documented as approximations.
Run 1 — unadjusted bars. The ≥0.95 gate: −17.9 bps at 1h (47.0% win rate), −3.5 at session close (49.3%), +6.0 at next-session close (48.8%), −64.3 at fifth-session close (42.0%). Multi-day horizons on unadjusted prices are contaminated by splits and dividends, so only intraday cuts were treated as interpretable. A post-hoc 0.92–0.95 band showing large positive fifth-session returns was flagged as a selection artifact and quarantined.
Run 2 — adjusted, total-return basis. Adjustment moved the gate estimates by one to three basis points (next-session +6.0 → +7.0; fifth-session −64.3 → −61.1) — it repaired the methodology and left the conclusion alone. The pre-specified headline, the full-population Spearman correlation between confidence and SPY-adjusted outcome, appears with final coverage and dependence-robust intervals in the tables above.
Run 3 — pre-specified ember hunt. Survivorship: closed-only realized adjusted return +28.9 bps (CI −25.2 to +81.0); closed plus open marked to market +29.7 (−20.9 to +79.3). Exits versus fixed horizons: +12.2 bps (−41.2 to +58.9) — consistent with exit-policy contribution, imprecise. No pre-specified source, direction, catalyst, or regime cut passed holdout. Markout from model-decision time at 1h: −21.2 bps (−36.9 to −6.4); post-rule-update period −21.1 (−40.4 to −2.1). The execution-time leg was unavailable in the frozen join (0 of 632).
Run 4 — entry-time recovery and the latency floor. The raw ingestion timestamp is co-written with the scoring timestamp (median and 95th-percentile difference 0.00 s), so no earlier anchor than the post time exists. 200 of 232 equity Ledger positions matched to signals; the ordered cohort held 166 rows, 159 with non-null adjusted markouts. From actual notional fills: −10.8 bps at 1h (−39.2 to +16.7), −12.1 at session close (−44.6 to +21.7), +2.4 at next-session close (−68.3 to +76.8). Markouts anchored at the post time showed no earned alpha. The latency rescue failed at its most favorable tested configuration.
The full-population analyses rest on 2,976 non-null observations per horizon — well powered for the rank-correlation question, which is why the headline was recentered there. The gate signal population is 570 non-null observations per horizon; the realized book is thinner still, on the order of 160–260 closed or matched positions depending on the run, producing intervals of roughly ±30 to ±75 bps. Findings at the realized tier are absence-of-evidence within those bounds, not evidence of absence beyond them.
The trading domain offered unusually favorable evaluation conditions: outcomes existed whether or not the system acted; notional trades did not move prices; rejected signals could be graded counterfactually from historical bars; the outcome clock and benchmark were mechanically specifiable. The framework as implemented audits decision systems whose predictions and eventual outcomes can be joined reliably, including rejected cases where counterfactual labels remain observable. Domains where the decision changes the outcome, or where labels are selectively observed, require an additional causal layer that this implementation did not build.
Tag evidence-package-v1; branch tip e3c4ec0; evidence-package implementation commit 65c8dc25f04b; golden-harness fixes 8033d28 and 2663376 (merged via 8447e98 and 0aa1624); sanitized configuration SHA-256 ce7fb6b2…. Data snapshot hashes (bar cache over 642 files, frozen population extract, raw-window extract, population and markout snapshots, 20-row golden file, read-only Ledger copy), final build run identifiers for all three benchmark versions (canonical 15,260 rows / 3,815 signals each), study-plan document hashes, and exact library versions are recorded in MANIFEST.md at the same tag. A documentation-only manifest label correction (study-plan row labels, Run 4 report row, PR provenance) ships at tag evidence-package-v1.1 (merge commit 30a98997); no data or analysis changed.