Probability of backtest overfitting (PBO) is the estimated chance that a trading strategy's best in-sample result will underperform the median out-of-sample result among all the configurations tested. In the method's own worked examples, an overfit strategy carrying an in-sample Sharpe ratio of 1.27 registered a PBO of 55%, while a strategy with genuine signal and a similar in-sample Sharpe ratio of 1.54 registered a PBO of 13%.
The diagnostic comes from a 2015 paper by David H. Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu, and it targets a specific failure mode in quantitative finance: a backtest can be optimized, directly or by trial and error, until it fits its own historical sample rather than any persistent market pattern. A high Sharpe ratio alone does not distinguish the two cases. PBO was built to.
What does overfitting mean in a backtested strategy?
Overfitting is what happens when a strategy configuration is tuned until it matches the noise in a specific historical sample rather than a pattern likely to recur. The authors define it precisely: overfitting occurs when the configuration with the best in-sample (IS) performance systematically underperforms the other tested configurations out-of-sample (OOS).
The distinction matters because in-sample performance is not itself evidence of a working strategy. Any sufficiently flexible search process, with enough parameters and enough variations tried, can find a configuration that performed well on a fixed historical window purely by chance. Out-of-sample underperformance is the signature that reveals it.
How is probability of backtest overfitting actually calculated?
The paper's proposed method, combinatorially symmetric cross-validation (CSCV), requires only the time series of backtested returns for each configuration under consideration, not knowledge of the trading rule itself. It splits the historical performance data into S equal submatrices, forms every possible training set from half of those submatrices, and tests each one against its complementary out-of-sample half.
For each split, CSCV ranks how the in-sample winner performed out-of-sample and converts that rank into a logit score. Repeating the process across every combination produces a distribution of those scores; PBO is the share of the distribution below zero, meaning the share of splits in which the in-sample winner ranked below the out-of-sample median. With S set to 16 partitions, the method generates 12,780 combinations, which the authors report is sufficient for acceptable statistical error.
| Measure | Overfit example | Signal-bearing example |
|---|---|---|
| In-sample Sharpe ratio | 1.27 | 1.54 |
| Probability of backtest overfitting | 55% | 13% |
| Share of out-of-sample returns negative | 53% | Not reported as elevated |
The comparison illustrates the paper's central point: two strategies with comparable, even favorable, in-sample Sharpe ratios can carry very different odds of having been overfit, and the Sharpe ratio alone does not separate them.
Does testing more strategy configurations raise the risk?
Yes, and the authors treat this as the core reason PBO is needed at all. They write that the probability of finding false positives increases with the number of tests conducted on the same data, and they single out a common shortcut, the simple hold-out method, for a specific flaw: it does not take into account the number of trials attempted before a particular configuration was selected. CSCV is built to incorporate that trial count directly, through the relative ranking of every tested configuration against every other, rather than validating one chosen configuration in isolation.
The paper also separates two related but distinct questions. PBO answers whether the in-sample winner is likely to underperform the group's out-of-sample median. A second statistic the authors report alongside it, the probability that the selected strategy's out-of-sample return is negative, can remain elevated even when PBO looks moderate. In one set of examples with in-sample Sharpe ratios between 1 and 3, all of them positive in-sample, the authors found that roughly 78% of the corresponding out-of-sample Sharpe ratios came back negative, illustrating that a strategy can clear one overfitting check and still lose money out-of-sample by a different measure.
Why do regulators also require disclosure around backtested performance?
Backtest overfitting is not only an academic concern. Federal bank regulators and securities regulators treat backtested and hypothetical model results as a distinct risk category requiring independent scrutiny and disclosure. Supervisory guidance jointly issued by the Federal Reserve and the Office of the Comptroller of the Currency on April 4, 2011, known as SR 11-7, requires that model outputs be validated through outcomes analysis, defined as comparing actual outcomes against model forecasts using a sample period not used in development, and that this validation work be subject to critical review by a party independent of the model's development.
On the securities side, a Securities and Exchange Commission risk alert dated September 19, 2022 on examinations of the investment adviser marketing rule addresses hypothetical performance separately from actual track records, requiring advisers who present it to adopt policies ensuring the performance shown is relevant to the likely financial situation and objectives of the audience receiving it, among other conditions. Both regimes start from the same premise CSCV formalizes: a backtest's headline number is not self-validating.
Where does this diagnostic break down?
PBO measures whether a selection process across a defined universe of configurations reproduced overfitting; it does not certify that a strategy will perform well going forward, and the authors are explicit about several limits on what the number can establish.
- A flawed backtest produces a flawed PBO. The authors state that if the underlying backtest used incorrect assumptions, such as wrong transaction costs, the method assesses overfitting relative to flawed information and inherits the flaw.
- The symmetric-split design assumes the return series does not have strong autocorrelation across the sample; the authors note it may not suit every strategy type.
- Structural breaks are only visible within the dataset actually used. CSCV cannot anticipate a regime change that has not yet occurred in the historical sample.
- Results depend on knowing every configuration actually tried. If some tested and discarded configurations are left out of the input, a pattern the authors call the file drawer problem, the reported PBO understates true overfitting.
- Using PBO itself to search for the best-scoring configuration reintroduces the same problem it is meant to detect; the authors warn that any counter-overfitting technique used to select a strategy will result in overfitting.
- A high PBO among several configurations that all perform similarly well can also reflect a group of genuinely skillful strategies clustered together, not proof that all of them failed.
Regulatory frameworks carry a parallel limitation. SR 11-7 acknowledges that institutions may face constrained data and limited ability to use key model validation tools, in which case the guidance calls for closer attention to a model's stated limitations rather than treating validation as complete. No diagnostic in this space, statistical or supervisory, substitutes for live, out-of-sample performance measured after the fact.
The authors' own recommendation for practitioners follows from these limits rather than around them: disclose every configuration tested, not only the one published, and treat PBO as a reporting requirement rather than a search target. They suggest that journal editors and investors could reasonably ask researchers to estimate and report the probability when a backtest is submitted, on the same logic that clinical trials disclose every arm tested, not only the one that worked.
Frequently asked questions
Does a low PBO mean a strategy will work in live trading?
No. A low PBO indicates the selection process among the tested configurations did not show the signature of overfitting; it says nothing about future market conditions, capacity constraints, or execution costs the backtest did not model.
What is the file drawer problem in this context?
It is the risk that configurations tested and discarded before the final version never enter the PBO calculation, which understates how much searching actually produced the reported result.
Does SR 11-7 apply only to banks?
SR 11-7 was issued to institutions supervised by the Federal Reserve and the OCC, but its outcomes-analysis and independent-review framework is widely cited across finance as a reference standard for model validation generally.
For a related business news perspective, read AI-washing is not a model failure: what the SEC's enforcement actually measured.
For more context, read Complete Guide to Building a Sustainable Wellness Routine.
For more context, read How does peer review actually work before publication?.
For more context, read What Dollar-Cost Averaging Means and How It Works.