When Out-of-Sample Data Becomes Training Data

Validation verdict for holdout reuse

Out-of-sample is a relationship between data and decisions, not a permanent label attached to dates. When results from the window influence the next version, the window is no longer untouched evidence for that version.

All figures in this article are educationally constructed examples created to explain the failure mode. They are not real strategy results or recommended thresholds.

What the headline metric obscures about holdout reuse

A neat train/OOS split looks methodologically clean in a report. The problem is often invisible in the file: version 2 was created after seeing where version 1 failed, version 3 after seeing version 2, and every version was judged on the same “OOS” years.

No direct parameter optimizer is required. Changing a session filter, removing a symbol, adding a volatility rule or abandoning an exit because of the test result transfers information from the test period into the strategy.

How holdout reuse enters the backtest

Human feedback is optimization

A discretionary code change made after reading OOS results is still a search over model variants.

The same difficult episodes become known

Developers learn the exact dates and failure modes, then unconsciously build exceptions that fit them.

Version count is omitted

The final report shows one OOS curve but not the ten rejected versions that were also evaluated on it.

Walk-forward folds are repeatedly revisited

A fold that influences later architecture choices is no longer a pristine assessment of the final architecture.

Compact reconstruction of holdout reuse

CASE 07 · out of sample data contaminationFocused analysis · educationally constructed educational figures

Version	Change after reviewing same OOS	IS PF	Reused OOS PF	Truly untouched PF
V1	Original	1.61	1.42	—
V3	Session filter added	1.68	1.66	—
V5	Volatility exception	1.72	1.93	—
V5 on new holdout	No further change	1.72	1.93	0.96

The rising OOS score is not independent confirmation because each improvement used knowledge from that same window. The first genuinely untouched holdout provides the first clean test of V5—and reverses the verdict.

The test that can overturn the holdout reuse verdict

Build a decision ledger, not only a date split. Every time a result is viewed, record which code, parameter, universe or rule changed afterward and which datasets influenced that change.

Assign immutable roles to datasets: development, validation and final holdout.

Record every version evaluated on each window, including rejected versions.

After material feedback from a holdout, retire it as final evidence and create a new untouched holdout.

Use nested or rolling processes where architecture selection and final evaluation are separated.

Report OOS results together with the number of development decisions made after observing them.

What trade-list analysis can and cannot identify about holdout reuse

Export-level red flags for holdout reuse

The same OOS period appears in every strategy presentation
No log exists for rejected versions
Filters were added to “fix” known OOS losses
The final code was chosen after comparing many OOS curves
A walk-forward report is rerun until folds look acceptable

What the export reveals about holdout reuse

Side-by-side performance of versioned exports on declared IS, validation and holdout windows
Whether degradation reappears when a genuinely later or separately held-out file is loaded
Version ranking stability across folds and rolling windows
The gap between in-sample, reused-OOS and untouched-holdout outcomes when the files are supplied

What holdout reuse still requires from settings, code, or market data

A trade export cannot reveal how many unseen code versions were discarded. Preserve the development log and version-control history.
No tool can restore independence to a period after it has influenced decisions. It can only label the contamination and require new evidence.

ACADEMIC VALIDATION DOSSIER

Turn reused out-of-sample data into a falsifiable backtest diagnosis.

Case file 07/20 · OOS-DECAY · one failure mechanism, one falsifiable protocol

Research abstract: holdout reuse

Case file 07/20 · OOS-DECAY · one failure mechanism, one falsifiable protocol

This article tests one central proposition: repeatedly viewing and improving against the OOS period converts it into training data and destroys its meaning as an independent test. The question is not merely whether the displayed net profit or win rate was arithmetically calculated. The deeper identification problem is whether we know what constitutes one observation, what information was available at the decision time, which assumptions are necessary for the profit to exist, and how much of the conclusion survives when those assumptions are perturbed. The research object is therefore not one performance table; it is the linked data-generation, fill-generation, estimation, selection, and capital-allocation process.

The primary estimand is generalization performance on data never inspected by the researcher. The observation unit is defined as a fully frozen model specification paired with a one-time evaluation window. Without this definition, split fills, duplicated signals, common events, synthetic prices, or timestamp conversions can be double-counted as independent evidence. A larger row count does not necessarily contain more independent information. An academically defensible analysis fixes the relationship between the observation unit and the estimand before it reports sample size, standard error, or statistical confidence.

The principal sensitivity axes are number of OOS inspections, specification changes, and a final untouched holdout. The hidden state is rule edits, feature additions, threshold tuning, and period selection after viewing OOS results. In particular, selection among many attempts chooses an accidentally favorable OOS result and manufactures apparent generalization. Means and medians alone are incapable of describing that mechanism, so the analysis combines central estimates with lower quantiles, expected shortfall, sign stability, boundary-hitting frequency, and contribution concentration. The objective is not to find one pessimistic number, but to map the full region in which the original conclusion changes sign or ceases to be economically usable.

The conclusion does not attempt to prove that a backtest is good. It separates the component that remains after attempted falsification from the component that disappears when assumptions are reconstructed. The governing decision principle is to log every inspection and edit, preserving a final untouched window or prospective paper test opened only once. This is not trading advice; it is a research procedure for measuring how much evidentiary weight a TradingView trade export can carry. Liquidity not present in the file, broker-specific rules, future regimes, outages, and gaps require separate evidence, and statistical survival never guarantees future profit.

The numerical values illustrate the method for reused out-of-sample data; they are not a real strategy, client record, or forecast.

Hypotheses and identification target for holdout reuse

generalization performance on data never inspected by the researcher

Null hypothesis / H₀

H₀ for holdout reuse: The reported performance is not materially dependent on the suspected failure mechanism and survives reasonable perturbations.

Alternative hypothesis / H₁

H₁ for holdout reuse: The reported performance depends materially on the suspected failure mechanism and deteriorates after reconstruction, perturbation, or dependence-aware resampling.

Estimand

generalization performance on data never inspected by the researcher

Observation unit

a fully frozen model specification paired with a one-time evaluation window

Latent mechanism

rule edits, feature additions, threshold tuning, and period selection after viewing OOS results

Stress axes

number of OOS inspections, specification changes, and a final untouched holdout

Formal estimands for holdout reuse

Definitions precede inference.

FWER_ind=1−(1−α)^MProbability of at least one false discovery under the simplifying assumption of M independent tests; do not use it as the final correction when candidates are dependent.

Bias_sel=E[max_j θ̂_j]−max_j θ_jExpected upward bias created by selecting the maximum estimated performance.

ω_s=r_s/(N+1), λ_s=log[ω_s/(1−ω_s)], PBÔ=S⁻¹Σ_s1{λ_s≤0}For each split, normalize the out-of-sample rank r_s of the in-sample winner (1 worst, N best) by N+1, logit-transform it, and count the fraction of splits whose logit is non-positive.

OOS reviews14

Rule edits27

Best OOS Sharpe1.62

Untouched Sharpe0.31

Estimated PBO72%

Illustrative recomputation design for holdout reuse

For the holdout reuse reconstruction, table values are illustrative calculations used to expose a verdict reversal; they are not a user’s observed TradingView result.

ID	Recomputation layer	Operation	Comparison	Diagnostic purpose
S0	Reported result	Restate the Strategy Tester aggregate	Base	Apparent conclusion
S1	Unit reconstruction	a fully frozen model specification paired with a one-time evaluation window	Reassess count and dependence	Information correction
S2	Independent recomputation	Rebuild price, size, cost, and currency row by row	Separate reconciliation error	Measurement validity
S3	Local stress	number of OOS inspections, specification changes, and a final untouched holdout	Perturb one factor only	Causal sensitivity
S4	Tail injection	selection among many attempts chooses an accidentally favorable OOS result and manufactures apparent generalization	Recompute lower quantiles and boundary hits	Capital preservation
S5	Dependence-aware resampling	Generate paths across several block lengths	Intervals and sign stability	Estimation uncertainty
S6	Selection adjustment	Log search, OOS review, and exclusions	Correct maximum-selection bias	Generalization
S7	Full gate	log every inspection and edit, preserving a final untouched window or prospective paper test opened only once	Compare with predeclared thresholds	Pass / hold / reject

The illustrative recomputation for holdout reuse changes one processing layer at a time, then combines only predeclared layers. S0 is never treated as ground truth; it is the statement to be audited. S1 and S2 ask whether the exported unit and arithmetic are coherent. S3 and S4 identify local sensitivity and tail failure. S5 changes the uncertainty model rather than the trade list. S6 adjusts for the search that preceded publication. S7 applies the same gate to every version. This order prevents an adverse result from being explained away by simultaneously changing several assumptions.

In the holdout reuse figures, color and position encode diagnostic sensitivity only; they do not represent statistical significance or future P&L.

Diagnostic figures specific to holdout reuse

Four separate visual tests; no decorative chart reuse.

Figure 1. Primary diagnostic for reused out-of-sample data. Values are methodological illustrations, not estimates of a real strategy or future return.

Figure 2. Contamination matrix across training, selection, and holdout folds. Once holdout feedback changes the model, that period no longer supplies independent evidence. Values are illustrative recomputations, not observed performance or forecasts.

Figure 3. Performance decay as the same holdout is reviewed repeatedly. Repeated review can improve the reused holdout while performance deteriorates on the next untouched period. Values are illustrative recomputations, not observed performance or forecasts.

Figure 4. Contamination loop from holdout feedback into the next model. The feedback cycle itself is the contamination mechanism, and each loop creates another selection trial. Values are illustrative recomputations, not observed performance or forecasts.

The primary diagnostic decomposes number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance along a causal axis. Read slope, curvature, and the first decision-boundary crossing as “number of OOS inspections, specification changes, and a final untouched holdout” changes, not merely the height of the favorable point.

The two-dimensional surface exposes interaction among “number of OOS inspections, specification changes, and a final untouched holdout.” Color is a normalized margin to a predeclared gate, not an empirical probability. A broad connected pass region is different evidence from a narrow isolated island.

The resampling statistic is performance on untouched validation data. Compare an IID benchmark with purged temporal splits that preserve chronology and the training-validation boundary across several block lengths, reporting the 2.5th, 50th, and 97.5th percentiles and verdict-reversal rate. Save seeds and repetitions.

The causal map traces “view validation → edit specification → reevaluate the same period → accumulate adaptation → fail on genuinely unseen data.” A displayed metric is an intermediate product, not the first cause; perturb the input or assumption, rebuild trades and capital boundaries, and return to the predeclared gate.

Multi-layer audit questions for holdout reuse

A result is only as strong as its weakest unresolved layer.

AUDIT LAYER 0101 · Fix the estimand

An independent verifier should, fix the estimand as “generalization performance on data never inspected by the researcher.” Do not substitute net profit, win rate, or a visually smooth curve for that target. Declare the horizon, account currency, included frictions, and operating-stop boundary before calculation. Any post-result change creates a new hypothesis and version, preventing the question from being selected after the answer is known.

AUDIT LAYER 0202 · Reconstruct the observation unit

Reconstruct the observation unit as “a fully frozen model specification paired with a one-time evaluation window” before treating rows as independent evidence. Report raw rows, parent trades, decisions, event clusters, and the denominator used for each average or standard error. Recompute number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance under more than one defensible aggregation rule so that a larger export is not mistaken for a larger information set.

AUDIT LAYER 0303 · Preserve provenance and settings

Preserve the hash of the TradingView export and the symbol, timeframe, session, timezone, order-processing settings, costs, account currency, and Pine version. For reused out-of-sample data, rule edits, feature additions, threshold tuning, and period selection after viewing OOS results directly affects reproducibility. Keep immutable source, normalized, and analysis layers separate, with every join, deletion, imputation, and conversion recorded in a transformation ledger.

AUDIT LAYER 0404 · Separate identification from assumption

The export identifies only what can be rebuilt from recorded time, price, quantity, and P&L. unlogged manual trials, shared screenshots, and undisclosed period selection requires additional evidence. Mark each causal link as observed, bounded by assumption, or externally unverified. This prevents rule edits, feature additions, threshold tuning, and period selection after viewing OOS results from being presented as a confirmed fact when the available data support only an interval or conditional conclusion.

AUDIT LAYER 0505 · Reconcile row-level arithmetic

Do not adopt the platform summary as ground truth. Independently reconstruct a chronological ledger of model versions, viewing dates, specification changes, and decisions, then isolate a truly untouched final holdout and forward period. Reconcile total and row-level differences by sign, date, symbol, and order type. If discrepancies concentrate in the exact state associated with reused out-of-sample data, treat that concentration as a primary finding rather than dismissing it as rounding.

AUDIT LAYER 0606 · Quantify finite-sample uncertainty

Report number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance with intervals or resampling distributions, not point estimates alone. Match the uncertainty method to sample size, skewness, heavy tails, censoring, and selection history. If normal, quantile, and dependence-aware methods disagree on the sign, classify the edge as unidentified and show the minimum detectable effect and lower decision bound.

AUDIT LAYER 0707 · Preserve serial and cluster dependence

Do not narrow uncertainty with an IID shuffle alone. Resample purged temporal splits that preserve chronology and the training-validation boundary using several fixed block lengths and stationary bootstrap. Preserve random seed, repetition count, wrap rule, and missing-data treatment. For each block specification, report the distribution of number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance, the rejection-side tail mass, and the rate at which the verdict changes sign.

AUDIT LAYER 0808 · Measure tails and operating boundaries

Interrogate the mechanism “rule edits, feature additions, threshold tuning, and period selection after viewing OOS results” with lower quantiles, expected shortfall, influence, cluster length, and boundary-hitting measures. Historical maximum loss is not a loss cap. Define several absorbing or operating boundaries—capital, margin, mandate drawdown, and recovery time—and record which boundary fails first under each stress.

AUDIT LAYER 0909 · Model execution and market frictions

A flat commission deduction is not an execution model for reused out-of-sample data. Allocate spread, slippage, financing, borrow, roll, conversion, rounding, and rejected orders to the relevant unit. Recompute number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance under base, upper-quantile, and crisis states while preserving the possibility that costs and losses worsen together.

AUDIT LAYER 1010 · Count the complete search path

Count the complete population of periods, symbols, timeframes, parameters, exits, filters, and metrics that were tried. Do not detach the attractive result for reused out-of-sample data from rejected candidates, interim changes, or repeated validation reviews. Where appropriate, use PBO, SPA, and a Deflated Sharpe Ratio, and treat an unrecorded trial count as a material audit limitation.

AUDIT LAYER 1111 · Condition on market regimes

Test whether reused out-of-sample data is concentrated in one trend, volatility, liquidity, rate, or session state. Define regimes prospectively or on training data only. Report statewise number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance, occupancy, transition probabilities, and costs, then reweight the mixture to adverse but realistic future compositions.

AUDIT LAYER 1212 · Separate path, inception, and sizing

For the holdout reuse case, the same trade set can follow different capital paths under another inception date, order, initial balance, rounding rule, or stop condition. Separate fixed quantity, fixed R, and percentage sizing, then use circular shifts and block orderings to recompute drawdown, recovery, and boundary hits. Equal terminal P&L does not imply equal path risk.

AUDIT LAYER 1313 · Design counterfactual stress tests

Perturb “number of OOS inspections, specification changes, and a final untouched holdout” one axis at a time before creating a joint sensitivity surface. Add the negative control “hide a frozen replica from the researcher and evaluate it once, comparing the result with the repeatedly reviewed version.” Predefine the grid and crisis rule so that neither the most favorable nor the most damaging cell is selected after inspection. Save the slope, curvature, and exact point where the decision boundary is crossed.

AUDIT LAYER 1414 · Verify through an independent implementation

Have a second implementation reconstruct a chronological ledger of model versions, viewing dates, specification changes, and decisions, then isolate a truly untouched final holdout and forward period, then compare critical row-level outputs. Regression fixtures should include empty files, duplicate timestamps, extreme costs, reverse ordering, missing values, and boundary cases. Agreement between implementations is insufficient if they share the same bad input, so separate data construction and review roles where feasible.

AUDIT LAYER 1515 · Use a predeclared decision gate

Predeclare the decision rule. This case passes only if “an advantage remains after accounting for the search path and the one-time untouched evaluation meets predeclared limits.” Near a boundary, disclose interval width and economic materiality rather than a binary badge. If only one favorable block length, cost state, or implementation passes, classify the result as assumption-sensitive rather than robust.

AUDIT LAYER 1616 · Maintain a reproducibility ledger

The evidence ledger must store the input hash, code version, settings, exclusions, “number of OOS inspections, specification changes, and a final untouched holdout,” block lengths, random seed, repetition count, and every scenario output. Keep exploratory and confirmatory results in separate namespaces and retain failed trials. When new TradingView data arrive, create a new version and track model-version count, OOS review count, reasons for edits, remaining untouched data, and forward error rather than overwriting the old result.

AUDIT LAYER 1717 · Translate statistics into capital impact

Translate statistical changes into capital consequences. A shift in expectancy, lower quantile, recovery time, or boundary risk caused by reused out-of-sample data should be mapped to trade count, capital, margin, and continuation. A small per-trade difference can compound under high turnover, while a rare loss can be decisive near an absorbing boundary.

AUDIT LAYER 1818 · Separate roles and enforce stop conditions

Separate hypothesis design, implementation, independent recalculation, and approval where practical. Stop automatically on material reconciliation error, unresolved missing data, non-reproducibility, or a predeclared threshold breach. Audit the chain “view validation → edit specification → reevaluate the same period → accumulate adaptation → fail on genuinely unseen data,” and monitor model-version count, OOS review count, reasons for edits, remaining untouched data, and forward error prospectively without turning a historical pass into a promise of future profit.

Falsification protocol for holdout reuse

log every inspection and edit, preserving a final untouched window or prospective paper test opened only once

Freeze the TradingView source for the holdout reuse audit

Store the export without alteration and record its hash, export time, strategy, symbol, timeframe, and settings. Preserve every column relevant to reused out-of-sample data; deletions and imputations belong only in derived tables.

Reconstruct the observation unit for holdout reuse

Aggregate rows into “a fully frozen model specification paired with a one-time evaluation window,” and report raw rows, parent trades, events, and independent clusters. Recompute the critical result under another defensible aggregation.

Independently recompute the displayed holdout reuse result

Independently reconstruct a chronological ledger of model versions, viewing dates, specification changes, and decisions, then isolate a truly untouched final holdout and forward period. Reconcile row-level and aggregate outputs with Strategy Tester and preserve where discrepancies concentrate.

Isolate the holdout reuse mechanism

Treat reused out-of-sample data as the principal mechanism and move “number of OOS inspections, specification changes, and a final untouched holdout” one axis at a time while holding other settings fixed.

Map the operating boundary for holdout reuse

Combine the primary and interacting axes on a predeclared grid and recompute number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance. Record the width and connectivity of the acceptable region and every boundary crossing.

Resample the dependence structure relevant to holdout reuse

Use purged temporal splits that preserve chronology and the training-validation boundary with several fixed block lengths and stationary bootstrap. Save every random seed, repetition count, and block specification.

Inspect influence points and operating boundaries for holdout reuse

For the holdout reuse influence test, remove the largest contributor, top-k contributors, selected periods, and relevant regimes in sequence; then recompute lower-tail measures and the operating boundary.

Apply negative controls and conservative bounds to holdout reuse

Hide a frozen replica from the researcher and evaluate it once, comparing the result with the repeatedly reviewed version. Bound unlogged manual trials, shared screenshots, and undisclosed period selection as unobserved factors rather than elevating the optimistic value into the final answer.

Apply the predeclared gate to holdout reuse

Do not move the threshold after seeing results. Compare with “an advantage remains after accounting for the search path and the one-time untouched evaluation meets predeclared limits,” and distinguish pass, hold, and reject. Any unresolved material mismatch causes a hold.

Save a reproducible evidence package for holdout reuse

Bundle the source, transformation ledger, formulas, figures, all scenarios, failure logs, and code version for rerun in another environment. Prospectively monitor model-version count, OOS review count, reasons for edits, remaining untouched data, and forward error.

Decision gate for holdout reuse

Reject the story before trusting the curve.

How to read the holdout reuse figures and equations

The figures for holdout reuse use illustrative recomputations constructed to expose this specific failure mode. Do not infer statistical significance from line position or color alone; first verify the estimand, units, denominator, censoring rule, and cost sign defined by the equations. A sensitivity surface is not a causal estimate. It shows how a conclusion changes only within the stated assumptions. Resampling should compare an IID shuffle with stationary and block bootstrap procedures across several block lengths so that loss clustering and regime persistence are not silently destroyed. Store the random seed, iteration count, block length, bandwidth, and missing-data treatment, and claim reproducibility only after an independent implementation reproduces the same aggregates.

This case passes only if “an advantage remains after accounting for the search path and the one-time untouched evaluation meets predeclared limits” across reconstructed values, local perturbations, joint sensitivity, dependence-preserving resampling, and the negative control, with no material sign reversal or unresolved reconciliation error. A pass is limited evidence against the stated failure mode, not certification of future profit.

The estimand and observation unit were fixed before outcomes were reviewed
For holdout reuse, any material disagreement between reported and independently recomputed values must be resolved or explicitly explained.
The holdout reuse claim passes this gate only when its acceptable stress region is broad and connected rather than one isolated favorable island.
The sign of the holdout reuse estimate must remain stable across defensible block lengths, saved seeds, and reasonable interval methods.
For holdout reuse, economic margin remains after deleting the largest and top-five contributors and key regimes
For holdout reuse, conservative cost, fill, and capital-boundary scenarios remain inside the stopping mandate

6/6required gates · not a performance forecast

Limitations, external validity, and reproducibility of the holdout reuse audit

Every inference has a boundary.

The first limitation is that a trade export does not contain the complete market state. If order-book depth, queue position, network latency, rejected orders, broker liquidity, or realized financing history is absent, generalization performance on data never inspected by the researcher remains model-mediated. Model outputs should be displayed as scenario ranges and must not be formatted as though they were directly observed facts.

A second limitation specific to the holdout reuse analysis is structural change. A long historical sample does not guarantee a common population when market rules, participants, volatility, rates, spreads, data construction, or Pine execution semantics change. Do not increase nominal sample size by indiscriminately pooling old periods. Estimate rolling and regime-conditioned behavior and test parameter stability around detected changes.

A third limitation specific to the holdout reuse analysis is reuse of the diagnostic battery. Applying these tests repeatedly to the same data and editing the strategy until it passes turns the diagnostic process itself into another optimizer. Every post-test edit starts a new model version and requires untouched or prospective evidence. A test chosen after reading the outcome belongs to exploration and cannot be counted as independent confirmation.

A fourth limitation for the holdout reuse analysis is the distinction between statistical survival and operational suitability. Behavioral tolerance, locked capital, tax, regulation, outages, account terms, order-size limits, market-order restrictions, and liquidity discontinuities cannot be resolved from a CSV alone. The lab is a diagnostic for discovering hidden failure risk earlier; it is not investment advice, a performance warranty, or a guarantee of bounded loss. User-specific constraints remain a separate decision layer.

LIMIT 01Identification boundary

The estimand “generalization performance on data never inspected by the researcher” is identified only within the columns present in the TradingView export and the stated assumptions. If unlogged manual trials, shared screenshots, and undisclosed period selection cannot be observed, report bounds rather than a false point estimate.

LIMIT 02Structural change

Past estimates of reused out-of-sample data need not belong to the same population after changes in rules, participants, volatility, costs, or data specifications. Track model-version count, OOS review count, reasons for edits, remaining untouched data, and forward error in rolling and regime-specific windows.

LIMIT 03Reuse of the diagnostic

For holdout reuse, repeatedly applying the same diagnostic battery and editing until it passes turns verification into another optimizer. Every post-audit change therefore creates a new model version and requires untouched evidence.

LIMIT 04Operational suitability

Even if an advantage remains after accounting for the search path and the one-time untouched evaluation meets predeclared limits, the analysis does not establish tax, regulatory, behavioral, liquidity, order-size, or systems suitability. Separate statistical diagnosis from live-operating approval.

LIMIT 05Missing data and anomalies

Deleting observations related to rule edits, feature additions, threshold tuning, and period selection after viewing OOS results may improve the result. Compare no deletion, conservative imputation, and worst-case imputation, and display how number of OOS reviews, post-review edits, total candidates, PBO, selection-adjusted performance, and final untouched performance changes.

LIMIT 06Negative controls

Run the control “hide a frozen replica from the researcher and evaluate it once, comparing the result with the repeatedly reviewed version.” If the control performs similarly, suspect processing rules or common market drift before attributing performance to the strategy.

LIMIT 07Prospective monitoring

After a provisional pass, log model-version count, OOS review count, reasons for edits, remaining untouched data, and forward error sequentially and stop on persistent departures from the predeclared predictive range. Diagnose implementation drift before reoptimizing history.

LIMIT 08Common-mode failure and reporting

Multiple methods can agree because they share the same bad input or the same mechanism “rule edits, feature additions, threshold tuning, and period selection after viewing OOS results.” Give lower-tail outcomes, failed scenarios, and unresolved mismatches the same visual prominence as favorable results; test count is not proof of correctness.

10A

Independent and adversarial findings for holdout reuse

The holdout reuse case has a separate review line for formulas, chart encodings, data definitions, and falsifiability so agreement on one layer cannot mask failure on another.

The formula audit checks numerator, denominator, sign, unit, domain, and every conditioning assumption as one system. The material caution for this case is: PBO uses the relative rank ω_s=r_s/(N+1), with 1 as worst and N as best; reversing that orientation reverses the logit sign. Dividing by N+1 also avoids the endpoints 0 and 1. The independent-test FWER formula is an illustration and benchmark, not the final correction for correlated candidates. Every specification change after viewing validation data belongs in the search count. A correct symbolic expression can still calculate the wrong quantity when a column, currency, time unit, or fee sign is misdefined, so those mappings are part of the mathematical audit.

The figure audit assigns distinct jobs: Figure 1 diagnoses reused out-of-sample data; Figure 2 maps joint sensitivity; Figure 3 shows the dependence-preserving distribution of performance on untouched validation data; Figure 4 traces causal propagation. Color denotes distance to a predeclared gate, not probability or observed performance. Axis units, zero, quantiles, censoring, and bounds must agree with captions and tables. A smooth SVG line is explanatory geometry, not evidence of estimation precision.

The adversarial test does not cherry-pick one hostile scenario. It uses the negative control “hide a frozen replica from the researcher and evaluate it once, comparing the result with the repeatedly reviewed version,” resamples purged temporal splits that preserve chronology and the training-validation boundary at several block lengths, and bounds unlogged manual trials, shared screenshots, and undisclosed period selection as unobserved factors. Repetitions, seeds, exclusions, block specifications, and plotting range are frozen before results so the implementer cannot tune the audit after seeing the answer.

The independent conclusion is restricted to whether “an advantage remains after accounting for the search path and the one-time untouched evaluation meets predeclared limits.” It does not certify a good strategy or future profit. Any material reconciliation error, formula-domain violation, table-figure contradiction, sign reversal across defensible block lengths, or failure to outperform the negative control produces hold or reject. Prospectively, monitor model-version count, OOS review count, reasons for edits, remaining untouched data, and forward error.

Methodological references for holdout reuse

Primary methods and official platform documentation.

References for the holdout reuse case provide methodological context; they do not validate the synthetic numbers in this article or certify any backtest result. TradingView documentation is used for platform semantics, while statistical papers motivate uncertainty and selection controls.

Previous case← Why Heikin-Ashi Strategy Results Can Be Misleading
Backtest Analysis Articles
Next caseProfit Factor Without Drawdown Duration Is Incomplete →

Frequently asked questions about holdout reuse

Does merely viewing OOS contaminate it?

Viewing alone creates knowledge; contamination becomes material when that knowledge influences model selection, code, parameters or whether the strategy is accepted.

Can I use the window again for monitoring?

Yes, but call it validation or known historical data, not a final untouched test of a version shaped by it.

How many holdouts do I need?

There is no universal count. The key is a governance process that preserves at least one genuinely decision-independent evaluation for the final version.

Backtest Analysis

Can a backtest exposed to holdout reuse be trusted?

Do not judge the holdout reuse case from a finished equity curve alone. Use the TradingView trade list to inspect the mechanism-specific concentration, path, cost, timing, and dependence evidence shown on this page.

Analyze My Backtest Verification Status View Plans

Important limitations for the holdout reuse analysis

This article provides educational, descriptive analysis of constructed backtest failure examples. It is not investment advice, a buy or sell signal, a forecast or a promise of performance. Backtest results depend on data, code, broker-emulator assumptions, costs, sizing and market structure. TradingView is a trademark of TradingView, Inc.; SG Group is independent and does not claim endorsement or sponsorship by TradingView.

Counterpart: そのOOSはもう学習データになっている｜使い回した検証期間の罠