Robustness check
How can you tell whether a market finding is real, or just an accident of the method used to measure it?
In plain words
A robustness check is repeating an analysis with small changes to the method, such as a different window length, a different baseline or a different price series, and seeing whether the conclusion still holds.
Why it matters
Many market statistics depend on arbitrary choices: where the window starts, which prices are used, and what history the result is compared against. If a conclusion flips when one of those choices changes, it was probably carried by the choice rather than by the market, and it should carry little weight.
In a real week: BTC, week of 20 September 2021
For the week of 2021-09-20, the data describes BTC's fall through several framings: from Monday's open of 47241.75 to the close of 43160.9 the weekly return was -8.64%, and the week's return ranked in the 16th percentile against 188 weeks of history. Daily returns ranged from -8.95% on 2021-09-20 to +6.9% on 2021-09-22, while volume was 1.2 times its prior 12-week average. A robustness check would re-run these measures with altered windows or baselines and compare the results.
A common mistake
Checking only the variants that agree with the original result, or treating a check that fails as proof the market is random rather than as a sign the method was fragile.