Data snooping
Why can a trading rule look brilliant on past data and still be mostly luck?
In plain words
Data snooping is fitting or choosing a rule using the same historical data on which it is then judged. Trying many rules, settings or time windows and keeping the best one lets ordinary chance variation look like skill.
Why it matters
Every extra test raises the odds that something will look good without any real edge. The chosen rule is fitted to noise, so its recorded results overstate what it can earn, and its behaviour can change sharply once fresh data arrives.
In a real week: BTC, week of 17 December 2018
In the week of 17 December 2018, BTC rose 21.69%, a move in the 98th percentile of its history, after three of the prior four weeks were down. A rule buying after three down weeks would have caught it. Yet the dataset spans only 44 weeks of history, and many such rules could be tested on it, so picking the one that caught this week selects on luck as much as on any edge.
A common mistake
Treating a backtest's best result as its typical result. The headline rule is usually the winner of a contest against many discarded variants, so it carries an unearned advantage.