How I built an agentic trading product, falsified its central claim, and kept the machinery that ruled against me
I spent eight months building an agentic trading system, took on paying users while marketing a "60%+ win rate," and then built a falsification study that showed the system's entry score had no measurable tradable edge. I shut the paid product down and made the study, methodology, and signal record public.
The useful artifact was not the signal engine. It was the machinery that proved the signal wrong.
Here is how I got there.
Donald Trump's social-media posts frequently appeared to move individual companies and sectors. I wanted to know whether any of that movement remained tradable by the time a public-feed system could detect it, interpret it, and act. The presidential-post hypothesis was the door in, not the building: the real object was any market-moving text hitting public feeds — newswires, short-seller research, crypto whale trackers.
I had worked versions of the signal-extraction problem for decades. My first neural network, built in 1991, was a Kohonen self-organizing map classifying wavelet-transformed signals buried in noise. Large language models changed the economics of the problem: they could process disparate, unstructured feeds quickly enough to resemble an event-driven research desk. So I built one. I called it attentive.io.
The engine behind it, Alpha Council, is a graph of specialized agents. Scout ingests the feeds and extracts entities and sentiment. Chartist reads candlestick images. Judge weighs both and issues BUY, SELL, or PASS, attaching a model-generated score between zero and one that I labeled confidence. Ledger, the system of record, paper-trades every qualifying signal to resolution under versioned exit rules. The gate was simple: a score at or above 0.95 fired a "Sniper" alert to subscribers and opened a notional position. Everything — every signal at every score, every position, every exit — was timestamped and stored. That habit turned out to matter more than the signals it recorded.
The first numbers were intoxicating. A tiny early sample showed an 85.7% win rate and +124 basis points average at score 1.0 — on fourteen trades. What I was calling the calibration table looked monotone: higher score, better outcome. By Q1 2026 the realized record on the notional book (equities, ≥0.95) was 80 closed positions, a 45.0% win rate, +34 bps average, +$2,648 in paper P&L. A partial outcome table in the cloud database — 38 of the 82 positions then on the book, synced through a pipeline I did not yet know was broken — showed 63.2% and +74 bps, and that flattering subsample became the marketing: "60%+ win rate, +75 bps average alpha."
Two people paid, more curious than committed. Two things about those numbers deserve italics in hindsight: the best ones came from the smallest samples, and the most-quoted one came from a biased subsample created by an infrastructure bug I hadn't found yet.
One user did his own analysis and found the crack. When a signal arrived after trading hours, the system marked its notional entry at the last known or closing price — a fill no one could actually have gotten. Fantasy fills, flattering the alpha claim. Around the same time, an external review of one of my analysis memos flagged a subtler version of the same disease: enough analytical degrees of freedom across horizons, buckets, and sources that some cell would always look good by chance.
The ordinary engineering failures kept arriving too. The vision agent misclassified a preferred stock as common equity and inverted a directional read on a large-cap — confident narratives, wrong instruments. An earnings surprise cost the notional book a four-figure day. A display bug overstated crypto returns roughly a thousandfold. Worst of all: in May 2026 I discovered that five to ten instances of the trading engine had been running concurrently for weeks — manual starts that bypassed the service manager — producing 962 duplicate records in the trading cache and the mysterious 64% synchronization gap that had created the biased subsample quoted above. Operational defects had been manufacturing scientific-looking evidence.
I decided to stop patching individual findings and build the thing that could prove the whole claim wrong.
The design commitments came first, in writing, labeled non-negotiable: one uniform entry reference for every signal — the open of the next 1-minute bar after the signal, with after-hours signals rolling to the next session open — recomputed from a single historical-bar source, never reusing the live captured prices that had produced the fantasy fills; outcomes reported as excess return versus SPY over the identical window (a β = 1 adjustment, not a fitted regression); a sample size and bootstrap confidence interval on every cut; a pre-specified primary endpoint and a time-ordered 70/30 holdout, with false-discovery correction across everything exploratory; and hand-computed golden cases from outside the pipeline that the pipeline must reproduce within half a basis point — a check of implementation integrity, not of the model, and a gate that would later freeze the final evidence run until I actually supplied the cases.
One clarification the word "pre-specified" deserves: this was a sequence of studies, not a single frozen protocol. Each run's plan was versioned and written before that run's results were viewed. Run 1 exposed the price-adjustment and threshold-selection problems; the full-population rank endpoint was then specified prospectively for Run 2 and reported as a family across the four horizons; Run 3's rescue hypotheses, holdout split, and acceptance criteria were frozen before evaluation. The study plans and their hashes ship with the appendix.
One structural question could have killed the study at the door: only ≥0.95 signals were ever traded, so was the sample censored, with no outcomes for the rejects? It wasn't. Every scored signal at every level had been stored in the enrichment table all along; only the trading cache was filtered. And because outcomes are reconstructed from historical bars, a signal doesn't need to have been traded for its result to be knowable. The population: roughly 6,000 directional signals, 3,815 of them equities (crypto counted and excluded), yielding 2,976 resolvable signals per horizon — 11,904 benchmark-adjusted markouts in all.
Four runs, June 2026. The headline is one table — the rank correlation between the confidence score and SPY-adjusted outcome across the full population:
| Horizon | n | Spearman ρ | 95% CI |
|---|---|---|---|
| 1 hour | 2,976 | −0.028 | −0.068 to +0.009 |
| Session close | 2,976 | −0.011 | −0.048 to +0.024 |
| Next-session close | 2,976 | +0.027 | −0.012 to +0.064 |
| Fifth-session close | 2,976 | −0.053 | −0.091 to −0.017 |
n = non-null benchmark-adjusted observations per horizon; intervals are signal-level bootstrap. Dependence-robust intervals are in the appendix.
The rank relationship was indistinguishable from zero at the first three horizons. At the fifth-session close it was small and negative under the signal-level bootstrap — a result that did not survive the pre-registered dependence-robust test (the session-block interval crosses zero; appendix), and so stands as a negative point estimate, not a significant one. What matters is the direction of the failure: the score's ordering carried no stable positive information about subsequent SPY-relative returns, and because rank correlation is unchanged by any monotone transform, no order-preserving recalibration — Platt scaling, isotonic regression, anything of the kind — could turn that ranking into the claimed probability of a profitable outcome.
The production gate told the same story from the other side. At the tested entry anchors and horizons, the ≥0.95 population showed no robust positive edge, and no pre-specified source, direction, catalyst, regime, or latency rescue survived the holdout. Split and dividend adjustment moved the gate estimates by one to three basis points. The missing-bar analysis revealed no high-confidence rescue population, though missingness was not proven ignorable and full-population results remain conditional on resolvable market data.
Three of those rescue hypotheses were mine, and each was tested at its most favorable configuration. Price adjustment — splits and dividends corrupting multi-day windows — moved the results by one to three basis points. The 891 signals initially missing entry bars were not a hidden cache of short-side winners; only 38 of the 604-signal gate population were affected. And latency — "the signal is real, my pipeline is just too slow" — failed at the most favorable tested configuration: no earlier ingestion timestamp existed (the raw write was simultaneous with the scoring write, to the second), so the post time itself was the floor, and markouts anchored there showed no earned alpha either. The machine was pointed at the claims I most wanted to be true, and it ruled against me each time.
The study therefore rejected the commercial claim the product had actually made: that a customer receiving a high-confidence public-feed signal could enter at a feasible time and expect the score to rank profitable outcomes.
A companion methodology appendix provides the population construction, exact conventions, run-level results, sample sizes, power limits, and the dependence sensitivities.
Established: at the tested entry anchors and horizons, this system had no measurable, product-relevant entry edge. Its confidence score provided no useful positive discrimination for the decision it was supposed to support — a semantic failure before a statistical one. The early record was contaminated by market beta, measurement artifacts, and selection. And the remaining positive return on the realized book — roughly +23 to +29 bps of SPY-relative return on closed positions — was more consistent with the exit policy than with entry discrimination, but the estimate was imprecise and cannot support a claim of exit alpha.
Not established: that all public-information strategies are impossible. The study shows that this implementation, on this population, over this period, at achievable post-publication latency, extracted no robust tradable edge — nothing more universal than that. It cannot rule out a sub-minute reaction available to specialized high-frequency participants; it does rule out the edge the product claimed to deliver at an entry a customer could achieve. It did not establish that exit management generates alpha. One mechanism fits the data without being uniquely identified by it: the score plausibly ranked how clear and salient an event looked, and the most salient news is the most immediately priced — so a high score bought entry after the reaction and exposure to the reversal. And the β = 1 subtraction does not neutralize heterogeneous factor exposure, so the figures should be read as SPY-relative returns, not estimated alpha; whether a beta-matched benchmark would look better or worse depends on the book's long–short mix, and is reported as a limitation rather than assumed.
When the study finished, I could not keep charging for the product, so I shut the paid tier down. The signal list and associated data are public at https://live.attentive.io/, the system overview at https://live.attentive.io/overview, and the methodology at https://live.attentive.io/methodology. The system itself returned to its original job — my own capital, my own screen — as a discipline cockpit, with the score demoted to decoration.
Honesty requires the ordering confession: the measurement was built second and the claim first. The whole arc played out on a notional book and two paying subscribers, so the claim was falsified before real capital or real customers scaled into it. The ordering I recommend is the one I eventually followed, not the one I started with.
What survived is the machinery. I had been calling it the dis-prover; the more precise name is a falsification harness. It is running code, not a framework diagram: a versioned outcome-audit system that makes you state the claim before seeing the result, preserves the full population including everything the system declined to act on, joins each decision to an external outcome, tests whether the score ranks outcomes and whether its levels mean anything, and reruns end to end without silently rewriting history. Its strongest credential is behavioral: it was aimed, four times, at claims its builder wanted to be true.
I will not present it as a general solution to agent evaluation, because trading gave it an unusually favorable environment. Outcomes existed whether or not the system acted. Notional trades moved no prices. Rejected signals could be graded counterfactually from historical bars. The clock and the benchmark could be specified mechanically. Many consequential agent domains have none of these properties: in lending, hiring, or medicine, the rejected case may never reveal its outcome, the action changes the result, and labels arrive late, disputed, or not at all. What generalizes is the operating pattern — claim specification, decision lineage, outcome joining, censoring audits, discrimination and calibration reports, immutable runs — for decision systems whose predictions and eventual outcomes can be joined reliably, including rejected cases where counterfactual labels remain observable. Domains where the decision itself changes the outcome require an additional causal layer. This implementation did not solve that problem, and saying so is part of the point.
The failure was semantic before it was statistical. Judge's score appears to have measured how clear and salient an event looked to a model. I labeled it confidence, and users — including me — reasonably heard it as the probability of a profitable outcome. Those are different quantities. The surrounding analysis was fluent and internally coherent; the score was numerically precise — and unrelated to the external result it appeared to predict. Nothing about that failure is unique to trading. Any system that presents a number as predictive of an external outcome is making a claim about the world, and the claim deserves the same treatment mine finally got: graded against reality, on the full population, before anyone is asked to rely on it.
I'm pointing the same machinery at my own discretionary trading decisions next. If I have an edge the model did not, it should find that too; if I do not, better to learn it from the evidence than from the drawdown.
I'm also testing whether the surviving machinery can become Assay: a fixed-scope, independent examination of consequential model claims against the outcomes they are supposed to predict. One early proving ground is a healthcare-claims workflow where the labels are delayed and imperfect but observable — paid, partially paid, denied, corrected, appealed, abandoned — and where acting on a recommendation can change the result, which demands an explicit causal design rather than a simple join of score to outcome. It began as a problem in my own family. The first claim filed through it has already been paid — reimbursed at exactly the submitted amount — which is one data point, not a validation. The pilot will be judged against predeclared operational metrics, and I will publish the results whichever way they come out.
The rule going forward is the one attentive.io taught me: no score earns the name confidence until it has been graded against reality. Mine had not. That sentence cost eight months, a product, and a claim I liked. It was worth the price.