Multiple testing
Why does the best-performing rule out of many tested rules usually look better than it really is?
In plain words
Multiple testing, or the multiple comparisons problem, describes what happens when many rules, settings or signals are tried against past data and only the best performer is reported. Even random data contains patterns, so the maximum result across many trials is systematically higher than any single rule's true skill.
Why it matters
A researcher who tests dozens of indicators can always find one that appears to work. Selecting the winner and discarding the rest inflates expected performance, so a live position sized on that overstated result can carry far more risk than the underlying rule actually supports.
In a real week: ETH, week of 4 August 2025
ETH returned 21.56% in the week of 4 August 2025, in the 95th percentile of its 390 weeks of history, with volume above its prior 12-week average and a close 90.3% up the week range. A rule set that requires a strong up week, high volume and an upper-third close may have produced this week's appearance of predictive power after searching many combinations.
A common mistake
Choosing the single best result from a large search and treating its backtest return as an unbiased estimate of future performance, without correcting for how many alternatives were discarded.