Out-of-sample test
How do you check whether a trading rule works, rather than merely having been fitted to the past?
In plain words
An out-of-sample test evaluates a rule on data that played no part in its design. If a rule was tuned to a set of weeks, the honest check is how it behaves in later or separate weeks that the tuning never saw.
Why it matters
A rule that was adjusted until it looked good on its own data usually carries that history forward and fails when conditions shift. Out-of-sample testing separates a pattern that generalises from one that only describes the sample it was drawn from.
In a real week: BTC, week of 26 February 2018
In the week of 26 February 2018, BTC rose 20.07%, closing at 11515.0 within a 23.69% range, with realised volatility of 81.1% a year. The record held only 2 weeks of history. Any rule tuned on those 2 weeks had no untouched data left, so the week's outcome could not honestly count as an out-of-sample result.
A common mistake
Treating any forward check as out-of-sample. If the rule's settings were changed after seeing part of the test data, the test is no longer out-of-sample, even if it covers new dates.