Statistics · 6 min read

"This setup usually works" — the sample-size illusion

It's the most natural sentence in trading: “this setup usually works for me.” Usually, based on what? Almost always the answer is a small handful of vivid trades — nowhere near enough to distinguish a real edge from a lucky run. Sample size is the difference between a belief and a fact.

Here's the uncomfortable math. A fair coin — pure chance, zero edge — will come up heads seven times out of ten fairly often. If those ten flips were trades, you'd walk away convinced you had a 70% setup. Small samples are full of patterns that look like edges and are actually noise. Your brain has no built-in correction for this; it treats ten convincing trades as proof.

The illusion'It works most of the time'Based on ~8-15 memorable tradesWins remembered, misses forgottenConfidence far exceeds the evidenceThe reality of small NA coin can flip 7/10 heads easilyVariance dominates small samplesYou'd need hundreds to be sure'Usually' is doing no statistical work
A handful of trades cannot separate skill from luck — random sequences produce convincing 'edges' all the time. The feeling of 'usually works' is variance wearing a confident face.

Memory makes it worse

It's not even a clean small sample. Thanks to confirmation and recency bias, you remember the times the setup worked and quietly forget the times it didn't. So the “usually” isn't built on ten trades — it's built on the winning subset of a sample that was already too small. The confidence is doubly unearned.

How many trades you actually need

To distinguish a modest real edge from randomness with any confidence, you typically need hundreds of occurrences, not a dozen — the exact reason serious strategy testing runs over long samples and thousands of simulated paths. This is the statistical core under every discipline article here: the only way to know if a setup works is to test it at scale and then let the rules — not your highlight reel — execute it. “Usually works” is a hypothesis, not a strategy.

A coin flips 7/10 heads all the time. 'This setup usually works' on a dozen trades is that coin — and your memory only kept the heads.

Discipline you don't have to summon

The strategies are delivered as rules a machine executes the same way every time. Free 9-page Playbook.

Get the Playbook

All figures and examples are hypothetical and illustrative, based on backtested data and Monte Carlo simulation. Past and simulated performance does not guarantee future results. This is educational content, not financial advice. Diagrams are schematic, not specific trade recommendations. Prop firm rules and Terms of Service compliance are your responsibility.