The validation verdict
When one trade contributes more than the strategy’s entire net profit, the backtest has not demonstrated a broad edge. It has demonstrated that one historical event was captured exceptionally well.
What the headline metric makes you believe
Cumulative profit compresses every trade into one endpoint. A smooth-looking total of +10R does not tell you that gross profit was 32R, gross loss was 22R and one winner supplied 14R of the gross profit. The curve’s final height hides who built it.
The exceptional trade may be genuine and may even reflect the strategy’s intended convexity. The diagnostic question is different: does the strategy remain economically viable when that event is absent, delayed, partially filled or reduced by an exit rule change?
How this failure enters the backtest
Net profit is a residual
A single gain can exceed the final net result because it offsets losses elsewhere. Reading only the endpoint conceals this dependency.
A rare regime supplies the winner
A crisis breakout, overnight gap or one-directional trend may appear only once in the sample and may not recur at the same scale.
Exit logic harvests one extreme
A trailing exit can hold one winner for months while most trades churn. That may be intentional, but the repeatability must be tested.
Data or fill anomalies inflate the observation
Bad contract multipliers, synthetic prices, duplicated rows or an unrealistic fill can turn one trade into a false hero.
A compact diagnostic reconstruction
| Version | Trades | Net profit | Profit Factor | Largest winner / gross profit |
|---|---|---|---|---|
| Original report | 120 | +10R | 1.45 | 43% |
| Largest winner removed | 119 | −4R | 0.82 | 27% |
| Largest winner capped at 95th percentile | 120 | +0.8R | 1.03 | 22% |
| Top three winners removed | 117 | −11R | 0.61 | 13% |
The largest winner is not “14% of profit”; it is 140% of net profit because the remaining trades lose 4R. That distinction changes the question from “How profitable was the strategy?” to “How often can the strategy reasonably capture another event of this magnitude?”
The test that changes the verdict
Use leave-one-out and contribution concentration. The goal is not to delete every big winner, but to measure whether the conclusion survives plausible impairment of the most favorable observation.
What the SG Group Lab can expose — and what it cannot prove
Red flags in your export
- Largest trade contributes more than 50% of net profit
- Net profit becomes negative after removing one trade
- The dominant trade comes from a unique symbol or regime
- The trade’s size, duration or point value differs sharply from the rest
- The backtest was optimized after seeing the exceptional event
What the export can reveal
- Trade contribution ranking and concentration of gross and net profit
- Leave-one-out, top-N removal and capped-outlier counterfactuals
- Changes in Profit Factor, expectancy and drawdown after the dominant trade is impaired
- Whether the equity curve has one vertical step followed by flat or negative performance
What still requires source settings, code or market data
- A trade list cannot determine whether the extreme event will recur. That requires a market-structure thesis and additional unseen data.
- Removing a winner is a stress test, not proof that the winner was invalid. Convex strategies legitimately depend on rare large gains; they simply need enough evidence and capital tolerance.
Convert apparent backtest success into a falsifiable research object.
Case file 02/20 · LOO-INFL · one failure mechanism, one falsifiable protocol
Research abstract
Case file 02/20 · LOO-INFL · one failure mechanism, one falsifiable protocol
This article tests one central proposition: when aggregate P&L is dominated by one outlier, the remaining trade set can have negative expectancy while the whole backtest still appears profitable. The question is not merely whether the displayed net profit or win rate was arithmetically calculated. The deeper identification problem is whether we know what constitutes one observation, what information was available at the decision time, which assumptions are necessary for the profit to exist, and how much of the conclusion survives when those assumptions are perturbed. The research object is therefore not one performance table; it is the linked data-generation, fill-generation, estimation, selection, and capital-allocation process.
The primary estimand is a diversified strategy edge that survives deletion of any single trade. The observation unit is defined as an independent position idea, with pyramids and partial exits re-aggregated to the parent trade. Without this definition, split fills, duplicated signals, common events, synthetic prices, or timestamp conversions can be double-counted as independent evidence. A larger row count does not necessarily contain more independent information. An academically defensible analysis fixes the relationship between the observation unit and the estimand before it reports sample size, standard error, or statistical confidence.
The principal sensitivity axes are leave-one-out deletion, top-k deletion, and contribution-share caps. The hidden state is the largest-contribution trade, top-k profit concentration, and duplicated exposure to one event. In particular, a one-off news gap or data anomaly allocates an unreproducible gain across the entire sample. Means and medians alone are incapable of describing that mechanism, so the analysis combines central estimates with lower quantiles, expected shortfall, sign stability, boundary-hitting frequency, and contribution concentration. The objective is not to find one pessimistic number, but to map the full region in which the original conclusion changes sign or ceases to be economically usable.
The conclusion does not attempt to prove that a backtest is good. It separates the component that remains after attempted falsification from the component that disappears when assumptions are reconstructed. The governing decision principle is to require positive expectancy and profit factor after deleting the largest and top-five trades, with contribution HHI below a stated threshold. This is not trading advice; it is a research procedure for measuring how much evidentiary weight a TradingView trade export can carry. Liquidity not present in the file, broker-specific rules, future regimes, outages, and gaps require separate evidence, and statistical survival never guarantees future profit.
All values below are synthetic and used only to explain the method. They do not describe a real strategy, instrument, client result, or future performance.
Research hypotheses and identification target
a diversified strategy edge that survives deletion of any single trade
H₀: The reported performance is not materially dependent on the suspected failure mechanism and survives reasonable perturbations.
H₁: The reported performance depends materially on the suspected failure mechanism and deteriorates after reconstruction, perturbation, or dependence-aware resampling.
a diversified strategy edge that survives deletion of any single trade
an independent position idea, with pyramids and partial exits re-aggregated to the parent trade
the largest-contribution trade, top-k profit concentration, and duplicated exposure to one event
leave-one-out deletion, top-k deletion, and contribution-share caps
Formalization and estimands
Definitions precede inference.
θ̂(−i) = T(D \ {i})The estimate after deleting trade i.Iᵢ = θ̂ − θ̂(−i)The influence of one trade on the reported estimate.Gᵢ=max(Rᵢ,0), sᵢ=Gᵢ/ΣⱼGⱼ, HHI=Σᵢsᵢ², ΣⱼGⱼ>0Concentration of positive profit contributions, defined only when total positive profit is greater than zero; loss concentration is reported separately.The primary estimand is a diversified strategy edge that survives deletion of any single trade. The observation unit is defined as an independent position idea, with pyramids and partial exits re-aggregated to the parent trade. Without this definition, split fills, duplicated signals, common events, synthetic prices, or timestamp conversions can be double-counted as independent evidence. A larger row count does not necessarily contain more independent information. An academically defensible analysis fixes the relationship between the observation unit and the estimand before it reports sample size, standard error, or statistical confidence.
The principal sensitivity axes are leave-one-out deletion, top-k deletion, and contribution-share caps. The hidden state is the largest-contribution trade, top-k profit concentration, and duplicated exposure to one event. In particular, a one-off news gap or data anomaly allocates an unreproducible gain across the entire sample. Means and medians alone are incapable of describing that mechanism, so the analysis combines central estimates with lower quantiles, expected shortfall, sign stability, boundary-hitting frequency, and contribution concentration. The objective is not to find one pessimistic number, but to map the full region in which the original conclusion changes sign or ceases to be economically usable.
Synthetic experiment and recomputation specification
All values below are synthetic and used only to explain the method. They do not describe a real strategy, instrument, client result, or future performance.
| ID | Recomputation layer | Operation | Comparison | Diagnostic purpose |
|---|---|---|---|---|
| S0 | Reported result | Restate the Strategy Tester aggregate | Base | Apparent conclusion |
| S1 | Unit reconstruction | an independent position idea, with pyramids and partial exits re-aggregated to the parent trade | Reassess count and dependence | Information correction |
| S2 | Independent recomputation | Rebuild price, size, cost, and currency row by row | Separate reconciliation error | Measurement validity |
| S3 | Local stress | leave-one-out deletion, top-k deletion, and contribution-share caps | Perturb one factor only | Causal sensitivity |
| S4 | Tail injection | a one-off news gap or data anomaly allocates an unreproducible gain across the entire sample | Recompute lower quantiles and boundary hits | Capital preservation |
| S5 | Dependence-aware resampling | Generate paths across several block lengths | Intervals and sign stability | Estimation uncertainty |
| S6 | Selection adjustment | Log search, OOS review, and exclusions | Correct maximum-selection bias | Generalization |
| S7 | Full gate | require positive expectancy and profit factor after deleting the largest and top-five trades, with contribution HHI below a stated threshold | Compare with predeclared thresholds | Pass / hold / reject |
The illustrative recomputation experiment changes one processing layer at a time, then combines only predeclared layers. S0 is never treated as ground truth; it is the statement to be audited. S1 and S2 ask whether the exported unit and arithmetic are coherent. S3 and S4 identify local sensitivity and tail failure. S5 changes the uncertainty model rather than the trade list. S6 adjusts for the search that preceded publication. S7 applies the same gate to every version. This order prevents an adverse result from being explained away by simultaneously changing several assumptions.
All values below are synthetic and used only to explain the method. They do not describe a real strategy, instrument, client result, or future performance.
Diagnostic figures
Four separate visual tests; no decorative chart reuse.
Figure 1 decomposes the attractive aggregate result along a causal diagnostic axis. The important object is not the height of one line but the order in which sign, lower quantiles, and capital boundaries deteriorate as leave-one-out deletion, top-k deletion, and contribution-share caps changes. A steep slope near failure converts small input-estimation errors into large decision errors. Preserve the slope, curvature, and break-even point together with the preferred estimate rather than exporting only the most favorable value.
Figure 2 exposes interactions hidden by one-factor stress tests. Cost and slippage, sample size and dependence, or inception date and sizing can combine nonlinearly and move the strategy into a loss region even when each perturbation looks tolerable alone. A broad connected acceptable region suggests relative invariance; an isolated island suggests parameter selection or luck. The white decision contour must be derived from a predeclared rule, not drawn after observing the surface.
Figure 3 replaces one recomputed value with a distribution obtained from dependence-aware resampling. A positive center can coexist with substantial mass on the rejection side, meaning the same research design can plausibly produce a negative conclusion. Vary block length, repetition count, and random seed; report the median, 5th and 95th percentiles, and sign-flip rate. Do not select IID shuffling merely because it creates a narrower and more marketable distribution.
Figure 4 traces a causal chain from the largest-contribution trade, top-k profit concentration, and duplicated exposure to one event through the displayed metric and selection decision to live-capital consequences. A backtest statistic is an intermediate product, not the root cause. Substituting another headline metric without reconstructing the data leaves the original error intact. The falsification loop perturbs a cause, rebuilds rows, recomputes statistics and boundary hits, and returns to the predeclared gate. Reproducibility of this loop distinguishes a research diagnostic from decorative visualization.
Multi-layer audit questions
A result is only as strong as its weakest unresolved layer.
The first object to freeze is not a headline metric but the estimand. Here it is a diversified strategy edge that survives deletion of any single trade, not merely net profit, win rate, or profit factor. An ambiguous estimand lets the same CSV answer different questions opportunistically and encourages selection of whichever statistic looks strongest. Before analysis, declare the target population, horizon, account currency, included costs, and shutdown boundary. Any post-result change becomes a new hypothesis and a new version, preserving a traceable boundary between exploration and confirmation.
The observation unit is an independent position idea, with pyramids and partial exits re-aggregated to the parent trade. Treating every exported row as an independent trade inflates evidence through split fills, partial exits, pyramids, or simultaneous signals. Re-aggregate with parent-trade, signal, position, and event-cluster identifiers. Report row count, closed-trade count, decision count, and approximately independent cluster count separately. State which denominator enters each standard error, then perturb the aggregation rule. A conclusion that exists only under one convenient row definition is not robust evidence.
Store export time, TradingView settings, symbol, timeframe, session, timezone, account currency, commission, and order-processing options together with a content hash. If the largest-contribution trade, top-k profit concentration, and duplicated exposure to one event disappears from the record, the result cannot later be regenerated. Log correction, imputation, sorting, conversion, and deduplication operations in a transformation ledger, separating immutable source, normalized data, and analysis tables. Reproducibility is a property of the entire input-to-output chain, not of the analysis code alone.
Separate claims supported by the export from claims requiring assumptions. A trade list can reconstruct distributions, concentration, drawdown, sequence, and modeled cost sensitivity, but it does not directly observe order-book depth, queue priority, rejected orders, or future liquidity. The central failure chain is that when aggregate P&L is dominated by one outlier, the remaining trade set can have negative expectancy while the whole backtest still appears profitable. For each arrow in that chain, label what is identified by data, what is imposed by a model, and what requires an additional source. Estimated fills and future tail losses must not be written as known facts.
Do not take platform totals as ground truth. Rebuild quantity, price change, fees, currency conversion, and R-normalization at row level. Rounding and definition differences may look small yet compound over long, high-frequency, or percentage-sized histories. Reconcile original and rebuilt outputs both in aggregate and by row, then locate differences by sign, date, symbol, and order type. Concentrated reconciliation error is not harmless noise; it can identify the exact market state that manufactured the apparent edge.
A positive point estimate is not equivalent to evidence that true expectancy is positive. With heavy-tailed and asymmetric trade outcomes, compare ordinary intervals with bootstrap, block-bootstrap, HAC, and quantile-based uncertainty. Report effective sample size, minimum detectable effect, and the lower confidence bound, not only the center. When reasonable methods disagree on the sign, the scientific label is “not identified with the present sample,” not “profitable.” This discipline prevents precision from being inferred from a visually smooth curve.
Independent and identically distributed trades are often an unrealistic convenience. Trades inside one trend, news event, overlapping position, or signal burst share a common shock. Measure return autocorrelation, within-cluster dependence, losing runs, and holding-period overlap, then vary block length. Blocks that are too short destroy dependence; blocks that are too long destroy resampling diversity. If the conclusion survives only one block length, that instability is itself a primary result rather than a nuisance to hide.
The mean dilutes the mechanism that a one-off news gap or data anomaly allocates an unreproducible gain across the entire sample. Report lower quantiles, expected shortfall, largest-loss influence, losing-cluster length, and gap multipliers without treating the historical maximum as a physical limit. Define several absorbing boundaries: zero capital, margin failure, mandate drawdown, operational shutdown, or investor redemption. Boundary-hitting probability and time-to-hit are distinct estimands from historical maximum drawdown and should be reported separately.
A flat commission deduction is not an execution model. Decompose spread, slippage, financing, borrow, roll, currency conversion, order legs, size, and session. Costs may be dependent on losses rather than independent noise. Compare base, 2×, 3×, upper-quantile, and crisis scenarios and solve for break-even cost. A strategy whose sign flips after a modest realism adjustment has little execution margin no matter how attractive its gross backtest appears.
The rarity of the reported winner cannot be judged without knowing how many periods, symbols, timeframes, parameters, filters, exits, and metrics were tried. Treat the entire search path—including rejected candidates, interim edits, and repeated OOS reviews—as the relevant experiment family. Reality Check, SPA, PBO, and the Deflated Sharpe Ratio are not universal cures, but they expose selection effects that an unadjusted maximum ignores. Failure to log trial count is itself a high-risk finding.
A positive full-sample mean can hide concentration in one trend, volatility, session, liquidity, or rate regime. Define states prospectively or only on a training sample rather than from the profitable outcome. Report statewise expectancy, cost, drawdown, occupancy, transitions, and profit contribution, then reweight plausible future mixtures. If a small occupancy change makes the total negative, the honest description is conditional performance, not a universal strategy edge.
The same set of trades produces different wealth under different orderings, inception dates, initial capital, and sizing rules. Separate fixed quantity, fixed R, and percentage compounding; compare reversal, circular shifts, block permutations, and inception-date ensembles. If early gains expanded later risk, final profit combines signal quality with sizing feedback. Decompose those components and judge investability from lower path quantiles, not from the one realized representative curve.
A serious analysis asks not only what happened but what would have happened under another defensible processing rule. Perturb leave-one-out deletion, top-k deletion, and contribution-share caps, separating local counterfactuals that move one assumption from joint stress tests that move several. One-at-a-time tests support causal understanding; joint tests support capital preservation. Do not cherry-pick a dramatic worst case. Predeclare the rules generating base, mild, moderate, and crisis scenarios and retain all outputs.
The same written logic can change under different calculation order, missing-value handling, rounding, timezone conversion, bar aggregation, or fill rules. Where practical, recompute critical metrics through two independent implementations and compare row-level differences. Regression fixtures should include boundaries, anomalies, reversed order, empty data, and extreme costs. When calculation-engine verification is used commercially, disclose not only a test count but the functions covered, tolerances, and unresolved mismatches.
Move no acceptance threshold after seeing the answer. Predeclare lower expectancy, maximum contribution share, cost multiplier, ruin probability, sign stability, and effective-sample requirements. Near a boundary, show uncertainty and economic importance rather than only pass/fail. A statistically detectable effect can be commercially irrelevant if the cost margin is tiny; a large economic effect can remain unverified if uncertainty is broad. Use a two-axis decision table separating statistical evidence from operational materiality.
Save the input hash, analysis time, code version, parameters, random seed, block length, repetition count, exclusions, and outputs in one ledger. Reanalysis of the same file must reproduce the same result before versions can be compared. Keep exploratory and confirmatory outputs in separate namespaces; any edit after inspecting OOS begins a new research record. Preserving failed trials as well as successful ones creates the audit trail needed to measure selection bias.
Map statistical changes into capital consequences. A 0.03R decline in expectancy, five additional drawdown points, 120 more recovery days, or a rise in ruin probability from 3% to 12% affects trade frequency, capital, margin, and continuation differently. Small per-trade differences accumulate at high frequency, while rare changes become critical near an absorbing boundary. Replace the binary language of significant/not significant with operational actions: continue, resize, redesign, or reject.
Do not leave final approval solely to the researcher who discovered the attractive curve. Separate hypothesis, implementation, verification, and approval roles where practical. Automatically stop on material reconciliation error, missing data, non-reproducibility, or threshold failure. The governing principle is to require positive expectancy and profit factor after deleting the largest and top-five trades, with contribution HHI below a stated threshold. Passing means only that current evidence survived a specified falsification set; it is not certification of future profit. Re-run the same gate when new data arrive rather than granting permanent status to an old pass.
Falsification protocol
require positive expectancy and profit factor after deleting the largest and top-five trades, with contribution HHI below a stated threshold
Freeze the source export
Store the TradingView export without modification and record its hash, export time, strategy, symbol, timeframe, account currency, and settings. Never delete or overwrite source columns; every transformation belongs in a derived table.
Reconstruct the observation unit
Aggregate rows into parent trades, decisions, and event clusters according to the definition of an independent position idea, with pyramids and partial exits re-aggregated to the parent trade. Display the difference from raw row count and recompute critical results under more than one defensible aggregation rule.
Independently recompute displayed metrics
Rebuild P&L, wins and losses, means, profit factor, drawdown, duration, costs, and currency conversion from explicit formulas. Reconcile row-level and aggregate differences against Strategy Tester instead of labeling all disagreement as rounding.
Isolate one failure mechanism
Restrict the principal perturbation to leave-one-out deletion, top-k deletion, and contribution-share caps while holding other settings fixed. Local isolation makes it possible to identify which assumption moved the conclusion and by how much before interactions are introduced.
Map a two-dimensional sensitivity surface
Vary the principal factor and one interacting factor on a grid, recomputing net expectancy, drawdown, lower quantiles, and boundary-hitting risk. Determine whether the acceptable region is broad or a narrow isolated island.
Resample while preserving dependence
Use stationary or block bootstrap across several block lengths in addition to any IID benchmark, preserving losing clusters and regime persistence. Save the random seed and repetition count.
Delete influential observations and segments
Remove the largest trade, top five contributors, selected periods, and selected regimes in sequence. Measure sign stability and contribution concentration, interpreting deterioration as evidence about diversification of the edge.
Conservatize execution and costs
Apply base through crisis cost regimes and include pessimistic fill bounds where intrabar ambiguity or non-executable prices exist. Do not elevate the most optimistic fill convention into the single final result.
Apply the predeclared gate
Do not move thresholds after seeing results. Compare against the predeclared lower expectancy bound, sign stability, maximum contribution, drawdown, ruin probability, and effective sample size. Distinguish pass, hold, and reject.
Save a reproducible evidence package
Bundle the input hash, settings, code version, figures, formulas, all scenarios, failure logs, and timestamps so the analysis can be rerun. New data create a new version rather than overwriting prior evidence.
Decision gate
Reject the story before trusting the curve.
How to Read the Figures and Equations
The values in the figures are educational illustrations constructed to expose a failure mode. Do not infer statistical significance from line position or color alone; first verify the estimand, units, denominator, censoring rule, and cost sign defined by the equations. A sensitivity surface is not a causal estimate. It shows how a conclusion changes only within the stated assumptions. Resampling should compare an IID shuffle with stationary and block bootstrap procedures across several block lengths so that loss clustering and regime persistence are not silently destroyed. Store the random seed, iteration count, block length, bandwidth, and missing-data treatment, and claim reproducibility only after an independent implementation reproduces the same aggregates.
This case passes only if the requirement to require positive expectancy and profit factor after deleting the largest and top-five trades, with contribution HHI below a stated threshold is satisfied across reconstructed values, local stresses, the two-dimensional surface, dependence-aware resampling, and deletion of dominant contributors. Any material sign reversal, non-reproducibility, unresolved reconciliation error, or boundary breach produces hold or reject. The same thresholds continue in prospective monitoring; a historical pass never becomes a promise about future performance.
- The estimand and observation unit were fixed before outcomes were reviewed
- Material disagreement between reported and recomputed values is resolved or explained
- The acceptable stress region is broad and connected rather than an isolated island
- Sign remains stable across block lengths, seeds, and reasonable interval methods
- Economic margin remains after deleting the largest and top-five contributors and key regimes
- Conservative cost, fill, and capital-boundary scenarios remain inside the stopping mandate
Limitations, external validity, and reproducibility
Every inference has a boundary.
The first limitation is that a trade export does not contain the complete market state. If order-book depth, queue position, network latency, rejected orders, broker liquidity, or realized financing history is absent, a diversified strategy edge that survives deletion of any single trade remains model-mediated. Model outputs should be displayed as scenario ranges and must not be formatted as though they were directly observed facts.
The second limitation is structural change. A long historical sample does not guarantee a common population when market rules, participants, volatility, rates, spreads, data construction, or Pine execution semantics change. Do not increase nominal sample size by indiscriminately pooling old periods. Estimate rolling and regime-conditioned behavior and test parameter stability around detected changes.
The third limitation is reuse of the diagnostic battery. Applying these tests repeatedly to the same data and editing the strategy until it passes turns the diagnostic process itself into another optimizer. Every post-test edit starts a new model version and requires untouched or prospective evidence. A test chosen after reading the outcome belongs to exploration and cannot be counted as independent confirmation.
The fourth limitation is the distinction between statistical survival and operational suitability. Behavioral tolerance, locked capital, tax, regulation, outages, account terms, order-size limits, market-order restrictions, and liquidity discontinuities cannot be resolved from a CSV alone. The lab is a diagnostic for discovering hidden failure risk earlier; it is not investment advice, a performance warranty, or a guarantee of bounded loss. User-specific constraints remain a separate decision layer.
When several metrics deteriorate together, the strategy may hit margin, mandate drawdown, recovery-duration, or minimum-sample boundaries before net profit reaches zero. At every grid point over leave-one-out deletion, top-k deletion, and contribution-share caps, record the first violated boundary. This prevents the inference that a still-positive backtest is operationally safe and creates a risk map aligned with actual stopping rules.
Test not only whether the mean remains positive but whether candidate rankings survive resampling. If a small perturbation moves the selected model from first to twentieth, precise-looking optimization is unstable. Preserve Kendall rank correlation, top-k retention, and selection frequency rather than presenting one realized ranking as an intrinsic ordering.
Whenever missing rows, anomalies, zero quantities, or duplicate timestamps are removed, publish pre- and post-exclusion results and the reason. If missingness is not random, complete-case analysis can delete the loss state of interest. Compare conservative imputation, worst-case imputation, and no exclusion to test whether data cleaning manufactured profit.
Currency P&L, percentage return, R-multiples, and log return answer different questions. Currency values overweight later observations under changing size, whereas R can hide proximity to capital boundaries. Display several scales and ask whether the central conclusion survives, or whether the apparent edge is a capital-allocation artifact.
Use time shifts, randomized signals, inverted labels, and irrelevant features as negative controls that should have no genuine edge. If controls perform similarly, the result may reflect processing, selection, or common market drift rather than the strategy. Report the percentile of the real strategy within the placebo distribution.
After backtest approval, log prospective predictions and realized outcomes sequentially. Monitor how often expectancy, win rate, cost, drawdown, and duration leave their forecast intervals. Persistent deviation should trigger a stop and a diagnosis of structural change versus implementation mismatch before historical re-optimization begins.
Agreement among statistical methods is useful but does not protect against a common bad input. Examine independence of data construction, implementation language, reviewer, and assumptions in addition to method diversity. Treat unanimous output as evidence, not truth, and enumerate common-mode failure channels.
Give lower-tail outcomes, failed scenarios, unresolved mismatches, and missing data the same visual prominence as favorable values. Enlarging the number of passing tests while minimizing the definition of failure turns a research report into advertising. Commercial communication requires especially explicit labeling of synthetic values, limitations, non-guarantees, and the source of dynamic verification status.
Methodological references
Primary methods and official platform documentation.
- Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. Annals of Statistics.
- White, H. (2000). A Reality Check for Data Snooping. Econometrica.
- Hansen, P. R. (2005). A Test for Superior Predictive Ability. JBES.
- TradingView Pine Script® documentation: Strategies.
- Politis, D. N. & Romano, J. P. (1994). The Stationary Bootstrap. JASA.
- Newey, W. K. & West, K. D. (1987). A Simple, Positive Semi-definite, Heteroskedasticity and Autocorrelation Consistent Covariance Matrix. Econometrica.
- Lo, A. W. (2002). The Statistics of Sharpe Ratios. Financial Analysts Journal.
References provide methodological context; they do not validate the synthetic numbers in this article or certify any SG Group result. TradingView documentation is used for platform semantics, while statistical papers motivate uncertainty and selection controls.
Frequently asked questions
Should I always remove the biggest winner?
No. Keep it in the historical record, but also show the result without it. The comparison measures dependency; it does not rewrite history.
What concentration level is acceptable?
There is no universal cutoff. Compare the largest-trade share with trade count, strategy design, regime frequency and how much the verdict changes under removal.
Is this the same as overfitting?
Not necessarily. Outlier dependency can exist without parameter optimization. But optimizing around the event makes the dependency more suspicious.
Can You Trust This Backtest?
Do not judge the strategy from a finished equity curve alone. Use the TradingView trade list to inspect profit concentration, drawdown paths, trade ordering, cost sensitivity, and serial dependence together.
Important limitations
This article provides educational, descriptive analysis of hypothetical backtest failure modes. It is not investment advice, a buy or sell signal, a forecast or a promise of performance. Backtest results depend on data, code, broker-emulator assumptions, costs, sizing and market structure. TradingView is a trademark of TradingView, Inc.; SG Group is independent and does not claim endorsement or sponsorship by TradingView.
Counterpart: たった1回の大勝がバックテストを黒字に見せていた