CASE 16
The Optimization Lottery: Why the Best of 10,000 Backtests Is Usually Suspicious
You tested 10,000 parameter combinations. One produced a Sharpe-like ratio of 2.4. The dangerous question is not why that run looks so good. It is how many chances the search had to manufacture a winner.
Validation verdict for optimization selection bias
Optimization changes the statistical question. Once thousands of variants compete, the best result is expected to contain selection luck even when every individual calculation is correct. A winner must repay the search burden in untouched data—not merely win the leaderboard.
What the headline metric obscures about optimization selection bias
The optimizer presents a ranked table, so the top row feels like a discovery. But the ranking does not show how many failed hypotheses were consumed to create that top row. The more combinations, symbols, timeframes, filters and objectives you inspect, the more extreme the winner can become by chance.
A smooth parameter neighborhood helps, but it does not erase the lottery if the neighborhood itself was selected after repeated searches. The relevant evidence is the full research path: how many attempts were made, which decisions were data-driven and whether genuinely untouched observations remained.
How optimization selection bias enters the backtest
The maximum is biased upward
Selecting the best result from many noisy estimates guarantees that positive estimation error is overrepresented in the winner.
Hidden trials multiply the search
Symbols, chart intervals, entry filters, exit rules and objective functions are all additional tickets even when the optimizer reports only one parameter grid.
The objective teaches the strategy the sample
Optimizing net profit, drawdown or Profit Factor steers choices toward the quirks of the historical period used for scoring.
Human iteration is also optimization
Manually changing code after every disappointing chart creates a search process even without an automated optimizer.
Compact reconstruction of optimization selection bias
| Research path | Variants inspected | Best in-sample Sharpe ratio | Untouched-period Sharpe ratio | Verdict |
|---|---|---|---|---|
| Single preregistered rule | 1 | 1.10 | 1.02 | Modest evidence |
| Visible parameter grid | 250 | 1.72 | 0.84 | Selection decay |
| Grid + symbols + timeframes | 10,000 | 2.40 | 0.31 | Lottery winner |
| Locked rule, new period | 1 final | — | 0.76 | Re-estimated edge |
The 2.40 score is not “wrong.” It answers: which of 10,000 noisy candidates looked best in this sample? It does not answer: what performance should the selected rule deliver next. The untouched result is the evidence that addresses that second question.
The test that can overturn the optimization selection bias verdict
Treat every meaningful research branch as a trial. Reconstruct the search count, freeze one final specification and make it face data that could not have influenced the choice.
What trade-list analysis can and cannot identify about optimization selection bias
Export-level red flags for optimization selection bias
- The best run is shown without the number of runs attempted
- Many symbols and timeframes were screened before one was selected
- The holdout was checked repeatedly during development
- Parameters sit on a narrow isolated peak
- Code was rewritten after every disappointing out-of-sample result
What the export reveals about optimization selection bias
- Dispersion, rank stability and neighborhood smoothness across an uploaded parameter batch
- How sharply the selected candidate deteriorates outside its winning slice
- Whether many near-identical candidates produce materially different results
- Comparison of locked specifications across independent periods or datasets when supplied
What optimization selection bias still requires from settings, code, or market data
- A final trade CSV does not reveal discarded experiments. Search-history integrity requires logs, version control, parameter manifests or preregistration.
- No correction can create untouched data after it has been repeatedly consulted. A new market period or independently sourced dataset may be required.
Turn selection bias from large-scale optimization into a falsifiable backtest diagnosis.
Case file 16/20 · MULT-TEST · one failure mechanism, one falsifiable protocol
Research abstract: optimization selection bias
Case file 16/20 · MULT-TEST · one failure mechanism, one falsifiable protocol
This article tests one central proposition: selecting the maximum from 10,000 trials yields an impressive accidental winner even when true edge is zero, while ordinary p-values ignore the selection. The question is not merely whether the displayed net profit or win rate was arithmetically calculated. The deeper identification problem is whether we know what constitutes one observation, what information was available at the decision time, which assumptions are necessary for the profit to exist, and how much of the conclusion survives when those assumptions are perturbed. The research object is therefore not one performance table; it is the linked data-generation, fill-generation, estimation, selection, and capital-allocation process.
The primary estimand is the selected strategy’s true edge after correcting for the fact that many candidates were searched. The observation unit is defined as one preregistered hypothesis or parameter configuration plus its selection path. Without this definition, split fills, duplicated signals, common events, synthetic prices, or timestamp conversions can be double-counted as independent evidence. A larger row count does not necessarily contain more independent information. An academically defensible analysis fixes the relationship between the observation unit and the estimand before it reports sample size, standard error, or statistical confidence.
The principal sensitivity axes are trial count, candidate correlation, selection rule, and unreported trials. The hidden state is total trials, correlated candidates, interim rejections, and unreported failures. In particular, the extreme best candidate regresses toward the mean out of sample and collapses below expectations. Means and medians alone are incapable of describing that mechanism, so the analysis combines central estimates with lower quantiles, expected shortfall, sign stability, boundary-hitting frequency, and contribution concentration. The objective is not to find one pessimistic number, but to map the full region in which the original conclusion changes sign or ceases to be economically usable.
The conclusion does not attempt to prove that a backtest is good. It separates the component that remains after attempted falsification from the component that disappears when assumptions are reconstructed. The governing decision principle is to record the full search and require selection-adjusted evidence such as Reality Check, SPA, PBO, and the Deflated Sharpe Ratio. This is not trading advice; it is a research procedure for measuring how much evidentiary weight a TradingView trade export can carry. Liquidity not present in the file, broker-specific rules, future regimes, outages, and gaps require separate evidence, and statistical survival never guarantees future profit.
The numerical values illustrate the method for selection bias from large-scale optimization; they are not a real strategy, client record, or forecast.
Hypotheses and identification target for optimization selection bias
the selected strategy’s true edge after correcting for the fact that many candidates were searched
H₀ for optimization selection bias: The reported performance is not materially dependent on the suspected failure mechanism and survives reasonable perturbations.
H₁ for optimization selection bias: The reported performance depends materially on the suspected failure mechanism and deteriorates after reconstruction, perturbation, or dependence-aware resampling.
the selected strategy’s true edge after correcting for the fact that many candidates were searched
one preregistered hypothesis or parameter configuration plus its selection path
total trials, correlated candidates, interim rejections, and unreported failures
trial count, candidate correlation, selection rule, and unreported trials
Formal estimands for optimization selection bias
Definitions precede inference.
M=max_{1≤j≤K}T_jMaximum statistic selected from K candidates; candidate correlation can make the effective trial count smaller than K.H₀:max_jE[f_j]≤0Data-snooping null that no candidate has positive expected excess performance.DSR=Φ((SR̂−SR₀)√(T−1)/√(1−γ̂₃SR̂+((γ̂₄−1)/4)SR̂²))Adjusts for non-normality and the selection threshold SR₀ implied by multiple trials. Here γ̂₃ is skewness and γ̂₄ is ordinary kurtosis (3 under normality), not excess kurtosis. Do not replace T with an ad hoc effective sample size; address serial dependence separately and report block-based sensitivity. Application requires a positive denominator radicand, T>1, and an SR₀ calculation that reflects effective trial count and candidate correlation.The primary estimand is the selected strategy’s true edge after correcting for the fact that many candidates were searched. The observation unit is defined as one preregistered hypothesis or parameter configuration plus its selection path. Without this definition, split fills, duplicated signals, common events, synthetic prices, or timestamp conversions can be double-counted as independent evidence. A larger row count does not necessarily contain more independent information. An academically defensible analysis fixes the relationship between the observation unit and the estimand before it reports sample size, standard error, or statistical confidence.
The principal sensitivity axes are trial count, candidate correlation, selection rule, and unreported trials. The hidden state is total trials, correlated candidates, interim rejections, and unreported failures. In particular, the extreme best candidate regresses toward the mean out of sample and collapses below expectations. Means and medians alone are incapable of describing that mechanism, so the analysis combines central estimates with lower quantiles, expected shortfall, sign stability, boundary-hitting frequency, and contribution concentration. The objective is not to find one pessimistic number, but to map the full region in which the original conclusion changes sign or ceases to be economically usable.
Illustrative recomputation design for optimization selection bias
For the mass optimization reconstruction, table values are illustrative calculations used to expose a verdict reversal; they are not a user’s observed TradingView result.
| ID | Recomputation layer | Operation | Comparison | Diagnostic purpose |
|---|---|---|---|---|
| S0 | Reported result | Restate the Strategy Tester aggregate | Base | Apparent conclusion |
| S1 | Unit reconstruction | one preregistered hypothesis or parameter configuration plus its selection path | Reassess count and dependence | Information correction |
| S2 | Independent recomputation | Rebuild price, size, cost, and currency row by row | Separate reconciliation error | Measurement validity |
| S3 | Local stress | trial count, candidate correlation, selection rule, and unreported trials | Perturb one factor only | Causal sensitivity |
| S4 | Tail injection | the extreme best candidate regresses toward the mean out of sample and collapses below expectations | Recompute lower quantiles and boundary hits | Capital preservation |
| S5 | Dependence-aware resampling | Generate paths across several block lengths | Intervals and sign stability | Estimation uncertainty |
| S6 | Selection adjustment | Log search, OOS review, and exclusions | Correct maximum-selection bias | Generalization |
| S7 | Full gate | record the full search and require selection-adjusted evidence such as Reality Check, SPA, PBO, and the Deflated Sharpe Ratio | Compare with predeclared thresholds | Pass / hold / reject |
The illustrative recomputation for optimization selection bias changes one processing layer at a time, then combines only predeclared layers. S0 is never treated as ground truth; it is the statement to be audited. S1 and S2 ask whether the exported unit and arithmetic are coherent. S3 and S4 identify local sensitivity and tail failure. S5 changes the uncertainty model rather than the trade list. S6 adjusts for the search that preceded publication. S7 applies the same gate to every version. This order prevents an adverse result from being explained away by simultaneously changing several assumptions.
In the mass optimization figures, color and position encode diagnostic sensitivity only; they do not represent statistical significance or future P&L.
Diagnostic figures specific to optimization selection bias
Four separate visual tests; no decorative chart reuse.
Multi-layer audit questions for optimization selection bias
A result is only as strong as its weakest unresolved layer.
From an audit perspective, fix the estimand as “the selected strategy’s true edge after correcting for the fact that many candidates were searched.” Do not substitute net profit, win rate, or a visually smooth curve for that target. Declare the horizon, account currency, included frictions, and operating-stop boundary before calculation. Any post-result change creates a new hypothesis and version, preventing the question from being selected after the answer is known.
Reconstruct the observation unit as “one preregistered hypothesis or parameter configuration plus its selection path” before treating rows as independent evidence. Report raw rows, parent trades, decisions, event clusters, and the denominator used for each average or standard error. Recompute total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance under more than one defensible aggregation rule so that a larger export is not mistaken for a larger information set.
Preserve the hash of the TradingView export and the symbol, timeframe, session, timezone, order-processing settings, costs, account currency, and Pine version. For selection bias from large-scale optimization, total trials, correlated candidates, interim rejections, and unreported failures directly affects reproducibility. Keep immutable source, normalized, and analysis layers separate, with every join, deletion, imputation, and conversion recorded in a transformation ledger.
The export identifies only what can be rebuilt from recorded time, price, quantity, and P&L. unlogged manual edits, selection by alternative metrics, and deleted losing candidates requires additional evidence. Mark each causal link as observed, bounded by assumption, or externally unverified. This prevents total trials, correlated candidates, interim rejections, and unreported failures from being presented as a confirmed fact when the available data support only an interval or conditional conclusion.
Do not adopt the platform summary as ground truth. Independently restore every candidate, trial order, interim rejection, metric, and unreported result to the search population and evaluate the selected maximum as an order statistic. Reconcile total and row-level differences by sign, date, symbol, and order type. If discrepancies concentrate in the exact state associated with selection bias from large-scale optimization, treat that concentration as a primary finding rather than dismissing it as rounding.
Report total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance with intervals or resampling distributions, not point estimates alone. Match the uncertainty method to sample size, skewness, heavy tails, censoring, and selection history. If normal, quantile, and dependence-aware methods disagree on the sign, classify the edge as unidentified and show the minimum detectable effect and lower decision bound.
Do not narrow uncertainty with an IID shuffle alone. Resample candidate-by-time resampling that preserves candidate correlation separately from serial dependence using several fixed block lengths and stationary bootstrap. Preserve random seed, repetition count, wrap rule, and missing-data treatment. For each block specification, report the distribution of total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance, the rejection-side tail mass, and the rate at which the verdict changes sign.
Interrogate the mechanism “total trials, correlated candidates, interim rejections, and unreported failures” with lower quantiles, expected shortfall, influence, cluster length, and boundary-hitting measures. Historical maximum loss is not a loss cap. Define several absorbing or operating boundaries—capital, margin, mandate drawdown, and recovery time—and record which boundary fails first under each stress.
A flat commission deduction is not an execution model for selection bias from large-scale optimization. Allocate spread, slippage, financing, borrow, roll, conversion, rounding, and rejected orders to the relevant unit. Recompute total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance under base, upper-quantile, and crisis states while preserving the possibility that costs and losses worsen together.
Count the complete population of periods, symbols, timeframes, parameters, exits, filters, and metrics that were tried. Do not detach the attractive result for selection bias from large-scale optimization from rejected candidates, interim changes, or repeated validation reviews. Where appropriate, use PBO, SPA, and a Deflated Sharpe Ratio, and treat an unrecorded trial count as a material audit limitation.
Test whether selection bias from large-scale optimization is concentrated in one trend, volatility, liquidity, rate, or session state. Define regimes prospectively or on training data only. Report statewise total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance, occupancy, transition probabilities, and costs, then reweight the mixture to adverse but realistic future compositions.
For the optimization selection bias case, the same trade set can follow different capital paths under another inception date, order, initial balance, rounding rule, or stop condition. Separate fixed quantity, fixed R, and percentage sizing, then use circular shifts and block orderings to recompute drawdown, recovery, and boundary hits. Equal terminal P&L does not imply equal path risk.
Perturb “trial count, candidate correlation, selection rule, and unreported trials” one axis at a time before creating a joint sensitivity surface. Add the negative control “generate an equally large correlated candidate set with no true edge and compare the observed maximum with the null distribution of maxima.” Predefine the grid and crisis rule so that neither the most favorable nor the most damaging cell is selected after inspection. Save the slope, curvature, and exact point where the decision boundary is crossed.
Have a second implementation restore every candidate, trial order, interim rejection, metric, and unreported result to the search population and evaluate the selected maximum as an order statistic, then compare critical row-level outputs. Regression fixtures should include empty files, duplicate timestamps, extreme costs, reverse ordering, missing values, and boundary cases. Agreement between implementations is insufficient if they share the same bad input, so separate data construction and review roles where feasible.
Predeclare the decision rule. This case passes only if “PBO, SPA, and deflated performance clear predeclared limits after the complete search is counted, and the selected rank survives untouched data.” Near a boundary, disclose interval width and economic materiality rather than a binary badge. If only one favorable block length, cost state, or implementation passes, classify the result as assumption-sensitive rather than robust.
The evidence ledger must store the input hash, code version, settings, exclusions, “trial count, candidate correlation, selection rule, and unreported trials,” block lengths, random seed, repetition count, and every scenario output. Keep exploratory and confirmatory results in separate namespaces and retain failed trials. When new TradingView data arrive, create a new version and track cumulative trials, candidate correlation, selection rule, PBO, deflated Sharpe, and forward rank rather than overwriting the old result.
Translate statistical changes into capital consequences. A shift in expectancy, lower quantile, recovery time, or boundary risk caused by selection bias from large-scale optimization should be mapped to trade count, capital, margin, and continuation. A small per-trade difference can compound under high turnover, while a rare loss can be decisive near an absorbing boundary.
Separate hypothesis design, implementation, independent recalculation, and approval where practical. Stop automatically on material reconciliation error, unresolved missing data, non-reproducibility, or a predeclared threshold breach. Audit the chain “many trials → select the maximum → mistake luck for skill → inflated expectation → regression on unseen data,” and monitor cumulative trials, candidate correlation, selection rule, PBO, deflated Sharpe, and forward rank prospectively without turning a historical pass into a promise of future profit.
Falsification protocol for optimization selection bias
record the full search and require selection-adjusted evidence such as Reality Check, SPA, PBO, and the Deflated Sharpe Ratio
Freeze the TradingView source for the optimization selection bias audit
Store the export without alteration and record its hash, export time, strategy, symbol, timeframe, and settings. Preserve every column relevant to selection bias from large-scale optimization; deletions and imputations belong only in derived tables.
Reconstruct the observation unit for optimization selection bias
Aggregate rows into “one preregistered hypothesis or parameter configuration plus its selection path,” and report raw rows, parent trades, events, and independent clusters. Recompute the critical result under another defensible aggregation.
Independently recompute the displayed optimization selection bias result
Independently restore every candidate, trial order, interim rejection, metric, and unreported result to the search population and evaluate the selected maximum as an order statistic. Reconcile row-level and aggregate outputs with Strategy Tester and preserve where discrepancies concentrate.
Isolate the optimization selection bias mechanism
Treat selection bias from large-scale optimization as the principal mechanism and move “trial count, candidate correlation, selection rule, and unreported trials” one axis at a time while holding other settings fixed.
Map the operating boundary for optimization selection bias
Combine the primary and interacting axes on a predeclared grid and recompute total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance. Record the width and connectivity of the acceptable region and every boundary crossing.
Resample the dependence structure relevant to optimization selection bias
Use candidate-by-time resampling that preserves candidate correlation separately from serial dependence with several fixed block lengths and stationary bootstrap. Save every random seed, repetition count, and block specification.
Inspect influence points and operating boundaries for optimization selection bias
For the mass optimization influence test, remove the largest contributor, top-k contributors, selected periods, and relevant regimes in sequence; then recompute lower-tail measures and the operating boundary.
Apply negative controls and conservative bounds to optimization selection bias
Generate an equally large correlated candidate set with no true edge and compare the observed maximum with the null distribution of maxima. Bound unlogged manual edits, selection by alternative metrics, and deleted losing candidates as unobserved factors rather than elevating the optimistic value into the final answer.
Apply the predeclared gate to optimization selection bias
Do not move the threshold after seeing results. Compare with “PBO, SPA, and deflated performance clear predeclared limits after the complete search is counted, and the selected rank survives untouched data,” and distinguish pass, hold, and reject. Any unresolved material mismatch causes a hold.
Save a reproducible evidence package for optimization selection bias
Bundle the source, transformation ledger, formulas, figures, all scenarios, failure logs, and code version for rerun in another environment. Prospectively monitor cumulative trials, candidate correlation, selection rule, PBO, deflated Sharpe, and forward rank.
Decision gate for optimization selection bias
Reject the story before trusting the curve.
How to read the optimization selection bias figures and equations
The figures for optimization selection bias use illustrative recomputations constructed to expose this specific failure mode. Do not infer statistical significance from line position or color alone; first verify the estimand, units, denominator, censoring rule, and cost sign defined by the equations. A sensitivity surface is not a causal estimate. It shows how a conclusion changes only within the stated assumptions. Resampling should compare an IID shuffle with stationary and block bootstrap procedures across several block lengths so that loss clustering and regime persistence are not silently destroyed. Store the random seed, iteration count, block length, bandwidth, and missing-data treatment, and claim reproducibility only after an independent implementation reproduces the same aggregates.
This case passes only if “PBO, SPA, and deflated performance clear predeclared limits after the complete search is counted, and the selected rank survives untouched data” across reconstructed values, local perturbations, joint sensitivity, dependence-preserving resampling, and the negative control, with no material sign reversal or unresolved reconciliation error. A pass is limited evidence against the stated failure mode, not certification of future profit.
- The estimand and observation unit were fixed before outcomes were reviewed
- For mass optimization, any material disagreement between reported and independently recomputed values must be resolved or explicitly explained.
- The mass optimization claim passes this gate only when its acceptable stress region is broad and connected rather than one isolated favorable island.
- The sign of the mass optimization estimate must remain stable across defensible block lengths, saved seeds, and reasonable interval methods.
- For optimization selection bias, economic margin remains after deleting the largest and top-five contributors and key regimes
- For optimization selection bias, conservative cost, fill, and capital-boundary scenarios remain inside the stopping mandate
Limitations, external validity, and reproducibility of the optimization selection bias audit
Every inference has a boundary.
The first limitation is that a trade export does not contain the complete market state. If order-book depth, queue position, network latency, rejected orders, broker liquidity, or realized financing history is absent, the selected strategy’s true edge after correcting for the fact that many candidates were searched remains model-mediated. Model outputs should be displayed as scenario ranges and must not be formatted as though they were directly observed facts.
A second limitation specific to the optimization selection bias analysis is structural change. A long historical sample does not guarantee a common population when market rules, participants, volatility, rates, spreads, data construction, or Pine execution semantics change. Do not increase nominal sample size by indiscriminately pooling old periods. Estimate rolling and regime-conditioned behavior and test parameter stability around detected changes.
A third limitation specific to the optimization selection bias analysis is reuse of the diagnostic battery. Applying these tests repeatedly to the same data and editing the strategy until it passes turns the diagnostic process itself into another optimizer. Every post-test edit starts a new model version and requires untouched or prospective evidence. A test chosen after reading the outcome belongs to exploration and cannot be counted as independent confirmation.
A fourth limitation for the optimization selection bias analysis is the distinction between statistical survival and operational suitability. Behavioral tolerance, locked capital, tax, regulation, outages, account terms, order-size limits, market-order restrictions, and liquidity discontinuities cannot be resolved from a CSV alone. The lab is a diagnostic for discovering hidden failure risk earlier; it is not investment advice, a performance warranty, or a guarantee of bounded loss. User-specific constraints remain a separate decision layer.
The estimand “the selected strategy’s true edge after correcting for the fact that many candidates were searched” is identified only within the columns present in the TradingView export and the stated assumptions. If unlogged manual edits, selection by alternative metrics, and deleted losing candidates cannot be observed, report bounds rather than a false point estimate.
Past estimates of selection bias from large-scale optimization need not belong to the same population after changes in rules, participants, volatility, costs, or data specifications. Track cumulative trials, candidate correlation, selection rule, PBO, deflated Sharpe, and forward rank in rolling and regime-specific windows.
For mass optimization, repeatedly applying the same diagnostic battery and editing until it passes turns verification into another optimizer. Every post-audit change therefore creates a new model version and requires untouched evidence.
Even if PBO, SPA, and deflated performance clear predeclared limits after the complete search is counted, and the selected rank survives untouched data, the analysis does not establish tax, regulatory, behavioral, liquidity, order-size, or systems suitability. Separate statistical diagnosis from live-operating approval.
Deleting observations related to total trials, correlated candidates, interim rejections, and unreported failures may improve the result. Compare no deletion, conservative imputation, and worst-case imputation, and display how total candidates, effective trial count, PBO, SPA test, Deflated Sharpe Ratio, and untouched holdout performance changes.
Run the control “generate an equally large correlated candidate set with no true edge and compare the observed maximum with the null distribution of maxima.” If the control performs similarly, suspect processing rules or common market drift before attributing performance to the strategy.
After a provisional pass, log cumulative trials, candidate correlation, selection rule, PBO, deflated Sharpe, and forward rank sequentially and stop on persistent departures from the predeclared predictive range. Diagnose implementation drift before reoptimizing history.
Multiple methods can agree because they share the same bad input or the same mechanism “total trials, correlated candidates, interim rejections, and unreported failures.” Give lower-tail outcomes, failed scenarios, and unresolved mismatches the same visual prominence as favorable results; test count is not proof of correctness.
Independent and adversarial findings for optimization selection bias
The mass optimization case has a separate review line for formulas, chart encodings, data definitions, and falsifiability so agreement on one layer cannot mask failure on another.
The formula audit checks numerator, denominator, sign, unit, domain, and every conditioning assumption as one system. The material caution for this case is: The Deflated Sharpe Ratio incorporates a selection threshold SR₀ and non-normality through skewness and ordinary kurtosis γ̂₄ (3 under normality, not excess kurtosis). Serial dependence does not justify replacing T with an ad hoc effective sample size; report a separately dependence-adjusted return analysis, HAC, or block sensitivity. A correct symbolic expression can still calculate the wrong quantity when a column, currency, time unit, or fee sign is misdefined, so those mappings are part of the mathematical audit.
The figure audit assigns distinct jobs: Figure 1 diagnoses selection bias from large-scale optimization; Figure 2 maps joint sensitivity; Figure 3 shows the dependence-preserving distribution of selection-adjusted performance statistic; Figure 4 traces causal propagation. Color denotes distance to a predeclared gate, not probability or observed performance. Axis units, zero, quantiles, censoring, and bounds must agree with captions and tables. A smooth SVG line is explanatory geometry, not evidence of estimation precision.
The adversarial test does not cherry-pick one hostile scenario. It uses the negative control “generate an equally large correlated candidate set with no true edge and compare the observed maximum with the null distribution of maxima,” resamples candidate-by-time resampling that preserves candidate correlation separately from serial dependence at several block lengths, and bounds unlogged manual edits, selection by alternative metrics, and deleted losing candidates as unobserved factors. Repetitions, seeds, exclusions, block specifications, and plotting range are frozen before results so the implementer cannot tune the audit after seeing the answer.
The independent conclusion is restricted to whether “PBO, SPA, and deflated performance clear predeclared limits after the complete search is counted, and the selected rank survives untouched data.” It does not certify a good strategy or future profit. Any material reconciliation error, formula-domain violation, table-figure contradiction, sign reversal across defensible block lengths, or failure to outperform the negative control produces hold or reject. Prospectively, monitor cumulative trials, candidate correlation, selection rule, PBO, deflated Sharpe, and forward rank.
Methodological references for optimization selection bias
Primary methods and official platform documentation.
- White, H. (2000). A Reality Check for Data Snooping. Econometrica.
- Hansen, P. R. (2005). A Test for Superior Predictive Ability. JBES.
- Bailey, D. H. et al. The Probability of Backtest Overfitting.
- Bailey, D. H. & López de Prado, M. The Deflated Sharpe Ratio.
- Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. Annals of Statistics.
- Politis, D. N. & Romano, J. P. (1994). The Stationary Bootstrap. JASA.
- Newey, W. K. & West, K. D. (1987). A Simple, Positive Semi-definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica.
- Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal.
- TradingView Pine Script® documentation: Strategies.
References for the optimization selection bias case provide methodological context; they do not validate the synthetic numbers in this article or certify any backtest result. TradingView documentation is used for platform semantics, while statistical papers motivate uncertainty and selection controls.
Frequently asked questions about optimization selection bias
How many optimization runs are too many?
There is no universal cutoff. The burden rises with every effective trial and with correlation between trials. The key is transparent search accounting and truly independent validation.
Does a broad parameter plateau solve the problem?
It is better than an isolated spike, but a plateau discovered after a large adaptive search can still be selected by luck. Test the locked region on new data.
Is manual strategy development exempt?
No. Repeated human choices based on the same backtest are a form of adaptive optimization, even when no optimizer button was pressed.
Can a backtest exposed to optimization selection bias be trusted?
Do not judge the mass optimization case from a finished equity curve alone. Use the TradingView trade list to inspect the mechanism-specific concentration, path, cost, timing, and dependence evidence shown on this page.
Important limitations for the optimization selection bias analysis
This article provides educational, descriptive analysis of constructed backtest failure examples. It is not investment advice, a buy or sell signal, a forecast or a promise of performance. Backtest results depend on data, code, broker-emulator assumptions, costs, sizing and market structure. TradingView is a trademark of TradingView, Inc.; SG Group is independent and does not claim endorsement or sponsorship by TradingView.
Counterpart: 1万回の最適化で「最良」を選ぶほど疑うべき理由