Backtesting: The Complete Field Guide

How to design, run, validate and deploy historical simulations for automated stock-trading strategies — the complete map of what works, what silently lies to you, and how to build a system that tells the truth.

• Focus: daily & swing equity systems
• Level: knows trading, new to backtesting
• Code: Python
• Compiled: August 2026

How to use this guide

Read §02 and §03 first — they change how you see everything else. §04–§09 are the engineering and statistics core. §10–§13 deepen specific areas. §15 is the condensed checklist you'll return to before every research cycle, and §17 is a ready-to-paste specification for having an AI (or a developer) build your backtesting system correctly.

The one-paragraph version A backtest cannot prove a strategy works; it can only fail to prove that it doesn't. Almost every impressive backtest is impressive because of an error — look-ahead, survivorship, unpaid costs, fantasy fills, or simple luck manufactured by testing many variants. Your defense is process: point-in-time data, a strictly causal event loop, pessimistic execution assumptions, out-of-sample discipline that treats every peek as spent capital, statistics that account for how many things you tried, and a paper-trading incubation period before real money. Build the process once, and every strategy you test afterwards inherits its honesty.

The ten commandments

Everything in this guide compresses into ten rules. Each links to the section that justifies it.

1. Decide at time t, act at t+1. No signal may touch the price it executes at. (§03, §06)
2. Test on the market that existed, not the one that survived. Delisted stocks and point-in-time universes are non-negotiable. (§03, §04)
3. Count every experiment you run. Your best backtest must be judged against how many attempts produced it. (§03, §09)
4. Pay more costs than you expect to pay. If the edge dies at 2× realistic costs, it was never an edge. (§03, §06)
5. Prefer plateaus to peaks. A parameter set surrounded by cliffs is a coincidence with good marketing. (§08)
6. Respect the arrow of time. Never shuffle time-series data; purge and embargo every train/test boundary. (§05, §09)
7. Assume the market changes. A backtest is a claim about regimes, so test across them and monitor for drift after deployment. (§05, §16)
8. Test the tester. An unvalidated backtesting engine is the most dangerous bug you will ever write. (§06)
9. Pre-commit your gates. Acceptance thresholds, kill-switches and holdout data are defined before you look, or they mean nothing. (§08)
10. Incubate before you allocate. Months of paper trading against live data is the cheapest insurance in finance. (§16)

The priority ladder for daily/swing systems

This guide covers institutional-grade machinery, but not everything matters equally at every scale. For end-of-day and swing systems trading liquid US equities with personal-scale capital, the value concentrates as follows — get the top of the ladder right before polishing the bottom:

A note on expectations A daily/swing equity system that survives everything in this guide with a net annualized Sharpe ratio around 0.8–1.5 is a good system. Backtests showing triple-digit annual returns or Sharpe ratios above 3 on daily bars are, until proven otherwise, evidence of a bug or a bias — not of genius. Calibrating your expectations is itself a defense: it tells you when to be suspicious of your own results.

02 What a Backtest Really Is (and Isn't)

Before any code, get the epistemology right. Most backtesting failures are not software bugs — they are category errors about what a historical simulation can tell you.

A backtest is an observational study, not an experiment

History ran once. You cannot re-run 2020 with a different Fed, and you cannot generate fresh market data from the same distribution — because there is no fixed distribution (see §05). A backtest is therefore not an experiment with controls and repetitions; it is a single realized path, observed after the fact, from a process that was changing while it ran. Every statistical claim you make sits on top of that limitation.

This has a practical consequence: a backtest cannot verify that a strategy works. What it can do — and does superbly when built honestly — is falsify. It can show that a strategy would have lost money, or made money only in one regime, or only before costs, or only with information you couldn't have had. The correct mental model is a rejection filter: research produces candidate strategies, and the backtest's job is to kill the bad ones as cheaply as possible before they meet real capital. A strategy that survives is not "proven" — it has merely earned the right to the next, more expensive test (out-of-sample, then paper trading, then small live size).

Rule — the final exam, not the study guide Marcos López de Prado's first law of backtesting: never backtest until the model is fully specified. If you tweak the strategy, re-run the backtest, look at the result, and tweak again, the backtest has become part of the training loop — you are fitting the strategy to one historical path, and the reported performance is no longer an estimate of anything. Research and calibrate on training data; run the meaningful backtest once the design is frozen. Every "just one more look" spends statistical validity you cannot get back (§09 shows how to account for the looks you do take).

The asymmetry of evidence

Backtest outcomes are asymmetric in what they license you to believe:

• A bad backtest is strong evidence. If an idea loses money on honest data with honest costs, the burden shifts heavily against it. Falsification works.
• A good backtest is weak evidence. It is consistent with a real edge — and equally consistent with look-ahead bias, survivorship bias, unpaid costs, one favorable regime, or plain selection luck across the many variants you tried. A good backtest only becomes meaningful after you have eliminated each of those explanations, which is what §03–§09 of this guide are for.

Practitioners internalize this as a discount factor: expect live performance to be substantially worse than the backtest — halving the backtested Sharpe ratio is a common planning heuristic, and the degradation is worst for the strategies that looked best (they were the most heavily selected). If your strategy only makes sense at 100% of backtested performance, it doesn't make sense.

You are the overfitting machine

Overfitting is usually described as a property of models. In trading research it is more accurate to describe it as a property of you plus your loop. Every time you look at a result and make a choice — keep this rule, drop that filter, try 20 days instead of 50 — information flows from the test set into the strategy through your decisions. No individual step feels like cheating. The cumulative effect is a strategy molded to the accidents of one historical sample.

Three design responses, which recur throughout this guide:

• Hypothesis first. Write down the economic reason the edge should exist — who is on the other side of the trade, and why they will keep paying you (risk transfer, behavioral bias, structural constraint, forced flows) — before touching data. Ideas with a mechanism survive out-of-sample far more often than patterns found by search.
• A trial registry. Log every variant you test — automatically, in the engine, not by memory and honor. The count of trials is an input to the statistics that decide whether your best result is skill or selection (§09).
• Data you are not allowed to touch. A final holdout period, used once, at the end, as a verdict — never as feedback.

What a good backtest actually estimates

Run honestly, a backtest gives you a useful, bounded set of things: the shape of a strategy (turnover, holding period, win rate, tail behavior, drawdown profile), its cost sensitivity, its capacity, how it behaved across regimes, and a defensible — though still optimistic — estimate of risk-adjusted return. It also gives you operating expectations for live trading: what a normal losing streak looks like, so you can distinguish "working as designed" from "broken" after deployment (§16). What it does not give you is a promise about the future. The market that generated your data is not the market you will trade in — the honest goal is a strategy whose mechanism is likely to persist, demonstrated on data that never got a vote in its design.

Mindset summary Treat backtesting as a discipline for rejecting your own ideas, not a machine for confirming them. The quality of your research process is measured by how efficiently it kills bad strategies — the good ones are whatever is left standing.

03 The Bias Catalog — Why Most Backtests Lie

Twelve ways a simulation flatters you. Each entry: what it is, how it sneaks in, the smell test that detects it, and the fix. These are ordered roughly by how much fake performance they manufacture in daily/swing equity backtests.

1. Look-ahead bias

What it is: the strategy uses information that was not available at the moment the decision was made. This is the king of biases — it produces the most spectacular fake results and hides in the most innocent-looking code.

How it sneaks in:

• Same-bar execution: computing a signal from today's close and "buying" at today's close. In reality the close didn't exist until the market closed. The honest default: decide after the close of day t, execute at the open of day t+1.
• Full-sample statistics: normalizing a feature with the mean/σ of the entire dataset (a z-score computed in 2015 that quietly knows the volatility of 2022). Any scaler, percentile rank, regression fit, or "top decile" cutoff computed over the full sample leaks the future into every bar.
• Intrabar knowledge: rules that use today's high or low ("buy if it dips to the day's low") — you couldn't have known it was the low until the day ended. Similarly, assuming a stop and a target that were both touched inside one bar resolved in your favor.
• Restated data: fundamentals as they appear today (revised) rather than as first reported; earnings dates backfilled to the wrong session; index membership known only after announcement.
• Adjusted prices used for decisions about levels: split/dividend adjustment is computed with knowledge of future corporate actions. Fine for computing returns; wrong for rules like "price above $5" or share-quantity math (see bias 8 and §04).
• ML leakage: shuffled train/test splits, features built with future windows, labels that overlap the test period — the whole family is cataloged in §13.

look_ahead.py — the two classic leaks, and their fixes

import pandas as pd

df = pd.read_parquet("prices.parquet")   # columns: open, high, low, close

# ---- LEAK 1: full-sample normalization -------------------------------
# BAD: the mean/std know the whole future
z_bad = (df.close - df.close.mean()) / df.close.std()

# GOOD: rolling stats use only the trailing window available at each bar
mu = df.close.rolling(252).mean()
sd = df.close.rolling(252).std()
z_ok = (df.close - mu) / sd

# ---- LEAK 2: same-bar execution --------------------------------------
signal = (z_ok < -2.0).astype(int)       # decided using close of day t

# BAD: earn day t's return with a signal that needed day t's close
ret_bad = signal * df.close.pct_change()

# GOOD: signal known after close t -> fill at open t+1, exit open t+2.
# Return actually captured spans open(t+1) -> open(t+2):
open_to_open = df.open.pct_change().shift(-1)      # ret open t+1 -> t+2
ret_ok = signal * open_to_open.shift(-0)           # align: decision at t
ret_ok = ret_ok.shift(1)                           # book it on day t+1

print(f"annualized, leaked : {ret_bad.mean()*252: .2%}")
print(f"annualized, honest : {ret_ok.mean()*252: .2%}")
# The gap between these two numbers is pure fiction.

Smell test: shift every signal one extra bar later. Performance should degrade modestly (edges decay). If returns collapse to zero — your edge lived entirely inside the look-ahead. Conversely, shift signals one bar earlier (deliberate peeking): if that barely improves things, timing isn't where your risk is; if it explodes upward, your pipeline is exquisitely sensitive to leakage and deserves an audit.

Fix: make look-ahead structurally impossible rather than relying on discipline — an event-driven engine whose strategy API can only see data up to the current event (§06), rolling/expanding windows for every statistic, point-in-time snapshots for fundamentals and membership, and a mandatory one-bar gap between decision and execution.

2. Survivorship bias

What it is: testing on the stocks that exist today, thereby excluding every company that went bankrupt, was delisted, or was acquired along the way. History rewritten by the winners.

Why it's lethal for you specifically: daily/swing strategies love filters like "stocks in the S&P 500" or "current NASDAQ constituents," and mean-reversion systems are the worst hit: buying large dips looks miraculous on a survivors-only universe, because by construction every stock in the sample eventually recovered. The ones that dipped and died aren't in the file. Estimates of the inflation vary by universe and strategy — commonly 1–4% per year, and far more for small-cap dip-buying systems, easily enough to turn a losing strategy into a "great" one.

How it sneaks in: free data sources (Yahoo Finance and most bulk-download APIs) return only active tickers; screening "the current index members" and backtesting them ten years back; dropping tickers with missing recent data (which is precisely the delisted set); and ignoring delisting returns — what you actually received when a holding was delisted (often a merger payout, sometimes near-zero in bankruptcy).

Fix: use a survivorship-bias-free dataset that includes dead tickers and delisting events (§04 lists vendors); define the universe point-in-time ("stocks that were in the index on that date", "stocks that met the liquidity filter on that date"); and model delisting outcomes explicitly — apply the vendor's delisting return, and where unknown, assume conservatively (a merger closes at the last price; a bankruptcy loses most or all of the position). Track securities by a permanent identifier, not by ticker symbol — tickers get recycled (bias 11).

Example — the classic self-deception "Buy any current S&P 500 stock that falls 30% below its 52-week high; hold until recovery." On today's constituents this backtests beautifully across 20 years. The strategy as stated was untradeable: in 2006 you could not have known which stocks would be in the index in 2026, and the 2008 cohort of fallers included Lehman, Washington Mutual and Bear Stearns — none of which are in today's file to hurt the backtest.

3. Data snooping & multiple testing

What it is: running many variants and reporting the best. The winner's performance is inflated by selection, even if every individual test was perfectly honest. This is the bias that survives even a technically flawless engine.

The brutal arithmetic: the expected maximum Sharpe ratio across N zero-skill trials grows like √(2·ln N) in units of the Sharpe ratio's own sampling noise. Concretely: test ~1,000 random strategies on ten years of daily data and the best one is expected to show an annualized Sharpe near 1 — with zero true edge (derivation and simulation in §09). Every parameter combination in a grid search, every "let me just try exits at 2× ATR instead," every discarded idea from last month — they all count toward N. So do the trials embedded in your tools: an optimizer that evaluated 500 combinations ran 500 trials on your behalf.

Smell test: ask "how many things did I try before this one?" If the honest answer is "I don't know," your Sharpe ratio is unfalsifiable marketing. A Sharpe reported without a trial count is a p-value reported without sample size.

Fix: maintain an automatic trial registry (§06); evaluate the final candidate with statistics that penalize the search — the Deflated Sharpe Ratio, minimum backtest length, and PBO (§09); prefer few-knob strategies searched over coarse grids; and let hypotheses, not search, generate candidates.

4. Overfitting (curve fitting)

What it is: the strategy has molded itself to the noise of one historical sample rather than a repeatable mechanism. Multiple testing (above) is how it happens across many runs; curve fitting is how it happens within one strategy's design.

The two flavors:

• Parameter overfitting: the 23-day lookback works and the 19- and 27-day don't. A real effect is almost never that precise. You want plateaus — broad parameter regions that all work — and you should pick the center of the plateau, not the peak (§08 formalizes this as a stability region with a "cliff veto").
• Structural overfitting: each added rule ("…but not in December", "…unless RSI is above 80", "…except Fridays") is a degree of freedom paid for with sample. A useful budget: dozens of trades per parameter at minimum — a common hard ceiling is one tunable parameter per 30–50 trades — and hundreds per parameter are better. A strategy with 6 tunable inputs and 90 trades is a memorized answer key.

Smell test: perturb every parameter ±20% and re-run. A robust strategy degrades gracefully; an overfit one falls off a cliff. Also plot yearly returns: an edge concentrated in one or two years is a regime bet or an accident, not a system.

Fix: fewer parameters with economically motivated ranges; sensitivity heatmaps as a standard report; walk-forward validation (§08); and a bias toward simple rules — if the 2-parameter version doesn't work at all, the 6-parameter version that "works" is a mirage.

5. Cost neglect

What it is: backtesting gross returns in a business where the edge is often smaller than the frictions. Costs are not a haircut applied at the end — they compound per trade and can invert the sign of a strategy.

The full bill for a US-equity swing trade:

• Commission & fees: often ~$0 retail for US stocks, but regulatory fees on sells and per-share ECN fees at some brokers still exist; outside the US, commissions and stamp duties bite (UK note below).
• Spread: you cross half the bid–ask spread each way as a taker. Liquid large caps: ~1–3 bps. Small caps: 10–50+ bps. This alone kills many high-turnover ideas.
• Slippage: the price moves between your decision and your fill, systematically against you (your signals correlate with everyone else's).
• Market impact: your own order moves the price — grows with the square root of size (§10); negligible at small size in liquid names, dominant at scale.
• Short-side costs: borrow fees (0.25–1% annualized for easy names, occasionally triple digits for hard-to-borrow), plus paying any dividends on borrowed shares (bias 10).
• Financing & cash drag: margin interest if levered; conversely, idle cash earns the T-bill rate — over a 20-year simulation this materially changes results in both directions.

Smell test: compute your average gross edge per trade in basis points and compare it to your all-in cost per trade. A strategy earning 12 bps per round trip against 8 bps of cost is a coin flip wearing a suit; you want edge-to-cost of 2–3× or more. Then re-run the backtest at 0×, 1×, 2×, 3× your cost model: a real strategy degrades linearly; survival at 2× is a reasonable bar to demand before continuing.

Fix: model costs per fill inside the engine (never as a flat annual subtraction), calibrate the model during paper trading against realized fills, and prefer strategy designs whose turnover matches their edge — slower signals tolerate friction that faster ones cannot.

UK note (you're trading from London) If you ever trade UK-listed shares: UK stamp duty (SDRT) is 0.5% on purchases of most LSE main-market equities — 50 bps per entry is instantly fatal to high-turnover systems, which is one reason UK-based systematic retail traders overwhelmingly trade US equities (no stamp duty; a W-8BEN gets treaty withholding on dividends). Model FX conversion costs if your account is GBP-based, and remember tax treatment (CGT, ISA eligibility, spread-betting wrappers) changes net results — verify current rules; this is not tax advice.

6. Liquidity & capacity fantasy

What it is: the backtest takes positions the real market would not have let you take — trading illiquid names at printed prices, or sizes far beyond what the volume could absorb.

How it sneaks in: the "amazing" backtest is very often a micro-cap backtest in disguise: tiny stocks have the wildest mispricings and spreads, impact and borrow constraints that make those mispricings unharvestable. Also common: sizing positions as a fixed % of equity while equity compounds — by year 12 the simulation is silently trading $2M into a stock that traded $800k a day.

Fix: enforce point-in-time eligibility filters (minimum unadjusted price, e.g. $5; minimum median dollar volume, e.g. $10–20M/day); cap any order at a participation rate of average daily volume (1–5% for swing is conservative; even 10% assumes patient execution) and let unfilled remainder carry or cancel; and report the strategy's capacity — the AUM at which impact erases the edge (§10) — as a standard output.

7. Fill fantasy

What it is: optimistic assumptions about which orders execute and at what price. The subtlest engine-level bias, and the classic killer of limit-order and stop-based systems.

• The limit-touch fallacy: your limit order does not fill because price touched your level — there was a queue of real orders ahead of you, and the touch may have consumed only part of it. Pessimistic rule for bar data: a limit fills only if price trades strictly through it (beyond by an epsilon), not merely touches. The bias is vicious because it's adverse-selected: the touches that "fill" you and then reverse were exactly the profitable ones you wouldn't have gotten.
• Stops don't bound losses: a stop at $48 on a stock that closes at $52 and opens at $41 fills near $41, not $48. Model stop fills at min(stop, next open) for longs. Gap risk is a fact of swing trading; a backtest that fills stops at the stop price is quietly selling you free insurance.
• The intrabar ambiguity: if one daily bar touches both your target and your stop, OHLC data cannot tell you which came first. Resolve it pessimistically (assume the stop hit first), or use intraday data to adjudicate. Never resolve it optimistically — that's a systematic thumb on the scale, one bar at a time.
• Auction slippage: market-on-open and market-on-close fills are realistic for liquid names but not free — model a few bps of auction slippage, more for less liquid names.

Evidence note — this is measured, not theoretical A 2023 Aalto University study (Jäkärä, From Candles to Ticks) ran a live limit-order market-making algorithm for nine days, then backtested the identical period with the same engine: candle-based backtests executed roughly 2–3.5× as many trades as actually occurred live (1-minute bars were worse than 10-second bars) and accumulated materially larger P&L errors, because OHLC bars destroy the ordering and spacing of prices inside the bar — trades fired that never happened, at prices never available. Tick-based backtests tracked the live run closely. The error shrank as the strategy was slowed down but remained measurable even at hourly decision cadence. The design law that follows: your data's granularity must be finer than your decision cadence. Daily bars honestly support end-of-day decisions with next-bar fills — and nothing faster; the moment orders live and die inside the bar, you need intrabar (ultimately tick) data or your fills are fiction.

Fix: encode pessimism as the engine default (trade-through fills, stop-gaps at opens, worst-case intrabar ordering); if passive execution is core to the strategy's economics, either simulate queue position with intraday data or accept that bar-level backtests cannot validate it honestly.

8. Corporate actions & adjustment errors

What it is: splits, dividends, mergers, spinoffs and ticker changes handled wrongly — a data-plumbing problem that produces phantom gains and losses.

• Unhandled splits look like −50% overnight crashes (or +100% spikes for reverse splits); a mean-reversion system will gleefully "buy" every 2:1 split ever recorded.
• Dividends: if you simulate on price-only series, a diversified equity portfolio loses ~2% a year of real return that never appears — and your buy-and-hold benchmark is understated too, flattering the strategy. Use total-return (dividend-adjusted) series for performance, and credit dividends explicitly in the cash accounting.
• Adjusted vs unadjusted: back-adjusted prices are rewritten history (a $30 stock in 2010 might show as $2.87 adjusted). Compute returns and signals on adjusted series; apply price-level rules, share quantities and cost models to unadjusted prices. Keep both columns (§04 shows the dual-track pattern).
• Mergers/spinoffs/ticker recycling: positions must convert to cash or new shares at the right terms and dates; symbols get reused by unrelated companies — key everything by a permanent security ID.

Smell test: scan your dataset for single-day returns beyond ±40% and reconcile each against a corporate-actions file. Unexplained ones are data bugs waiting to become "alpha."

9. Regime dependence

What it is: an edge that is real within one market regime, silently presented as if it were unconditional. Not a data error — a framing error, and the deepest of the twelve (all of §05 is devoted to it).

How it sneaks in: the sample is dominated by one environment. A long-biased dip-buyer tested on 2012–2021 learned a decade of QE-backed V-recoveries; 2022 taught it otherwise. Trend systems tested on 2004–2008 look immortal; 2011–2019's chop humbles them. The backtest window is a hidden parameter.

Fix: test across structurally different periods as a matter of course — include at minimum the dot-com unwind (2000–02), the GFC (2008–09), the 2010s low-vol grind, the 2018 Q4 and 2020 crashes, and the 2022 rate shock, data permitting; report per-year and per-regime tables (bull/bear × high/low volatility) with parameters fixed; and state the strategy's regime thesis explicitly — "this earns X in trending tapes, loses Y in chop, and its long-run profitability is a claim about the mix." §05 covers drift detection and adaptive validation; §16 covers monitoring the regime assumption live.

10. Short-side fantasy

What it is: assuming every stock can be shorted, in any size, for free, forever. None of those four things is true.

Reality: you need a locate; borrow fees range from ~0.25% annualized (easy large caps) to triple digits (crowded small-cap shorts) and change daily; you pay dividends on borrowed shares; positions can be bought in at the worst moment; short squeezes create unbounded-loss tails that daily bar data understates; and regulatory constraints (e.g. the alternative uptick rule after a 10% decline) restrict execution exactly when shorts are most active. Historically hard-to-borrow names are — not coincidentally — where short backtests find their fattest "alpha."

Fix: for long/short systems, restrict the short book to liquid, institutionally held names; subtract a borrow-fee estimate per position per day (with a punitive default where data is missing); include dividend payments on shorts; and stress the book against squeeze scenarios. If borrow data is unavailable, treat small-cap short alpha as unverifiable rather than free.

11. Timestamp & alignment errors

What it is: mechanical misalignments between when data claims to exist and when it was knowable — small plumbing errors with look-ahead consequences.

• Session boundaries: mixing UTC-stamped data with exchange-local logic; including pre/post-market prints in "daily" bars from some vendors but not others; half-days and holidays creating phantom bars.
• Knowability lag: the official close is finalized in the closing auction; fundamentals are published weeks after the period they describe; "yesterday's" short interest is published on a delay. Every field in your dataset needs an effective timestamp — when you could actually have known it.
• Resampling traps: weekly bars built Monday-to-Friday versus Friday-to-Friday give different signals; a "monthly rebalance" that trades on the 1st using month-end data is fine — one that uses the 1st's close to trade the 1st's open is a leak.
• Cross-sectional misalignment: ranking stocks where some have stale prices (halts, illiquidity) makes the fresh ones look falsely extreme.

Fix: one canonical exchange calendar; every record carries both an event time and a knowable-from time; joins are as-of joins on the knowable time (code in §04).

12. The implementation gap

What it is: everything true about the simulation that will not be true about you and your infrastructure. The backtest assumes a flawless robot with perfect uptime and no feelings.

• Behavioral tolerance: the equity curve contains a 34% drawdown lasting 14 months. Automation places the orders, but you own the off-switch — and most system traders override or abandon systems in drawdowns, usually at the trough. If you would not survive the backtest's worst stretch (in money and in morale), the strategy is not viable for you, whatever its Sharpe.
• Operational reality: missed signals from outages, data feed failures on volatile opens, partial fills, capital that arrives late, rounding to whole shares on a small account (a $10k account cannot hold 25 equal-weight positions in $400 stocks).
• Taxes: short-term trading is taxed at income-like rates in most jurisdictions; a strategy that beats buy-and-hold pre-tax can lose to it after-tax. Model your own situation.

Fix: stress the equity curve with Monte Carlo to see drawdowns worse than the single historical path (§09); size the system so its worst plausible stretch is survivable; build the boring operational layer (monitoring, retries, reconciliation — §16); and write down, in advance, the conditions under which you will turn the system off, so the decision is made by past-you, calmly, instead of present-you, bleeding.

The bias smell-test table

04 Data: The Foundation

Your backtest can never be more honest than its data. Budget accordingly: for a serious daily/swing equity system, data quality is worth more than any modeling idea you will have this year.

The minimum viable data stack (US equities, daily/swing)

Adjusted vs unadjusted: the dual-track pattern

Back-adjusted prices splice corporate actions out of the series so that returns are correct: a 2:1 split doesn't look like a −50% day, and dividend adjustment folds payouts into the price path. But adjustment rewrites historical price levels using knowledge of everything that came after — a form of look-ahead if you use those levels for decisions. The professional pattern is to carry both series and route each use to the right one:

dual_track.py — one table, two views, each used where it's honest

import pandas as pd

px = pd.read_parquet("prices.parquet")
# columns: sid (permanent id), date, open_u, close_u (unadjusted),
#          adj_factor (cumulative), div_cash, volume

# Adjusted series for signal math (returns are correct across splits/divs)
px["close_a"] = px.close_u * px.adj_factor
px["ret"]     = px.groupby("sid").close_a.pct_change()

# Signal example: 20-day return, computed on ADJUSTED closes
px["mom20"] = px.groupby("sid").close_a.pct_change(20)

# Eligibility uses UNADJUSTED price and real traded dollars
px["dollar_vol"] = px.close_u * px.volume
med_dv = (px.groupby("sid").dollar_vol
            .transform(lambda s: s.rolling(63).median()))
px["eligible"] = (px.close_u >= 5.0) & (med_dv >= 10_000_000)

# Order sizing later uses close_u / open_u; P&L credits div_cash
# to the cash account on pay dates -- never double count via close_a.

Pitfall — the shape-shifting file Dividend-adjusted prices change retroactively: every new dividend rescales the entire history. A dataset downloaded today will not match the one from last quarter, breaking reproducibility and any cached signals. Store unadjusted prices plus adjustment factors as your immutable raw layer, and derive adjusted series on demand — never store only the adjusted output.

Point-in-time everything

The unifying rule for every non-price dataset: each record needs two timestamps — when the fact occurred and when it became knowable — and every join in your pipeline is an as-of join on the knowable time. Fundamentals join on publication date, not period end. Index membership joins on the date the change took effect (announcement date if you trade the announcement). Earnings dates must distinguish before-open from after-close, or your "trade the day after earnings" system is sometimes trading the day of.

asof_join.py — the only safe way to attach slow data to prices

import pandas as pd

# fundamentals: one row per (sid, period_end), published weeks later
f = pd.read_parquet("fundamentals.parquet")
f = f.sort_values("published_at")          # when it became knowable

px = px.sort_values("date")

merged = pd.merge_asof(
    px, f,
    left_on="date", right_on="published_at",   # <= knowable-by date
    by="sid",
    direction="backward",       # latest fact PUBLISHED on/before this bar
    allow_exact_matches=False,  # published today => usable tomorrow
)

Where to get data (2026 landscape)

Verify pricing and coverage before committing — these change. The consistent finding across the systematic-trading community: free data is fine for learning mechanics, and disqualifying for validating strategies, because it is survivorship-biased, adjustment-only, and quietly revised.

The validation pipeline: distrust, then verify

Vendor data is wrong somewhere. Every serious shop runs automated checks on ingest; so should you. The canonical checklist, cheap to implement and priceless in practice:

data_checks.py — run on every ingest; fail loudly

import pandas as pd
import numpy as np

def validate(px: pd.DataFrame, calendar: pd.DatetimeIndex,
             corp_actions: pd.DataFrame) -> list[str]:
    problems = []
    g = px.groupby("sid")

    # 1. OHLC internal consistency
    bad = px[(px.low_u > px[["open_u","close_u"]].min(axis=1)) |
             (px.high_u < px[["open_u","close_u"]].max(axis=1)) |
             (px.low_u <= 0)]
    if len(bad): problems.append(f"OHLC violations: {len(bad)} rows")

    # 2. Calendar gaps: missing bars on days the exchange was open
    for sid, d in g.date:
        expected = calendar[(calendar >= d.min()) & (calendar <= d.max())]
        missing = expected.difference(pd.DatetimeIndex(d))
        if len(missing) > 2:
            problems.append(f"{sid}: {len(missing)} missing sessions")

    # 3. Extreme moves not explained by a corporate action
    px["ret_u"] = g.close_u.pct_change()
    wild = px[px.ret_u.abs() > 0.40]
    unexplained = wild.merge(corp_actions, on=["sid","date"],
                             how="left", indicator=True)
    n = (unexplained._merge == "left_only").sum()
    if n: problems.append(f"{n} moves >40% with no corporate action")

    # 4. Stale series: many identical consecutive closes
    stale = g.close_u.apply(lambda s: (s.diff() == 0).rolling(10).sum().max())
    if (stale >= 9).any(): problems.append("stale price series detected")

    # 5. Duplicates and zero-volume anomalies
    if px.duplicated(["sid","date"]).any(): problems.append("duplicate rows")
    return problems

Two further habits: cross-vendor spot checks (sample 20 symbol-days per month and reconcile against a second source — disagreements cluster exactly where your alpha will) and quarantine, don't patch: exclude broken series from the universe rather than hand-editing prices, and record the exclusion so results are reproducible.

Storage, snapshots and versioning

Backtests are only comparable if they ran on the same data. Treat data like code:

• Immutable raw layer: vendor files as delivered, partitioned Parquet (by date or symbol), never edited in place. New downloads are new snapshots with a date, not overwrites.
• Derived layer: adjusted series, features, universes — all rebuilt deterministically from raw + code, so they can be deleted and regenerated.
• A dataset version hash recorded with every backtest run (§06's run manifest). "Results changed" must be attributable to code, data, or config — never a mystery among the three.

For scale context: 25 years of daily bars for the entire US market is a few GB in Parquet — a laptop problem. Engineering heroics are not required; discipline is.

05 Non-Stationarity & Regime Change

Markets are not a fixed distribution you sample from — they are a river, not a pond. This section is the conceptual backbone for every validation choice in the guide: why standard machine-learning testing fails on financial data, and how to evaluate strategies in a world whose rules shift.

The i.i.d. illusion

Classical statistics and most off-the-shelf ML tooling assume observations are independent and identically distributed: each sample tells you nothing about the next, and all samples come from one stable underlying distribution. Financial time series violate both clauses, categorically:

The consequences are concrete. Dependence means your "1,000 observations" carry far less independent information than 1,000 coin flips, so naïve significance tests overstate confidence. Non-stationarity means a model can be perfectly fit and perfectly obsolete: the relationship it learned may simply have ended.

Pitfall — the shuffle trap Apply ordinary k-fold cross-validation (random splits) to time-series data and you will, by construction, train on observations that occur chronologically after your test observations. The model "learns the future" — not the specific answers, but the regime: the volatility level, the trend, the correlation structure surrounding each test point. The result is a systematic optimistic bias with a signature you should memorize: brilliant validation metrics, immediate collapse in live deployment. If a pipeline ever calls train_test_split(shuffle=True) or plain KFold on market data, its results are inadmissible. Chronological splits (train on years 1–4, validate on year 5) are the minimum; purging and embargoing (§09) complete the job.

Two ways the world changes: covariate shift vs concept drift

"The market changed" hides a distinction that determines the right response. Write the data as inputs X (your features) and target Y (what you predict — next-period return, hit-the-target-first, etc.):

• Covariate shift — P(X) moves, P(Y|X) holds. The inputs visit new territory but the relationship survives. Example: average daily volume doubles and volatility regime shifts from 12% to 28% annualized, but the way a volatility-normalized pullback predicts a bounce is unchanged. Models fail here because they extrapolate outside the feature ranges they trained on — often fixable with normalization by rolling scale, retraining on recent windows, or features designed to be regime-relative rather than absolute.
• Concept drift — P(Y|X) itself changes. The relationship breaks: the same pattern no longer implies the same outcome. Example: a post-earnings drift signal decays as it becomes widely harvested; dip-buying that worked under QE stops working under QT. No amount of renormalization repairs this — the edge's mechanism has weakened or reversed, and the honest responses are re-estimation, de-weighting, or retirement.

A useful diagnostic habit: monitor the distribution of your features (detects covariate shift) separately from the distribution of your prediction errors or trade outcomes (detects concept drift). §16 turns this into a live monitoring design.

Regimes, and why your backtest window is a parameter

Markets move through persistent environments — trending vs mean-reverting tapes, high vs low volatility, tightening vs easing liquidity — and most daily/swing edges are conditional on them. Two disciplines follow:

• Evaluation must span regimes. A test window inside one environment measures the strategy-in-that-environment, however long the window is. More data helps only if it adds diverse data; ten years of one regime is one observation of the thing that matters.
• Evaluation windows shouldn't straddle regimes blindly. The mirror error: averaging performance across a boundary (e.g., a fold spanning 2021 exuberance and the 2022 bear) blends two different behaviors into one misleading number — "evaluation lag" that masks how badly the model does in the new regime. Report per-regime results, not just the blend.

Detecting change: drift detectors

You can locate regime boundaries statistically instead of by eye. Three standard change-point tools, in increasing order of "online-ness":

ADWIN (ADaptive WINdowing) maintains a growing window W over a monitored stream — typically prediction errors or returns — and repeatedly asks: can W be split into an "old" half W₀ and "recent" half W₁ whose means differ by more than sampling noise allows? The threshold comes from a Hoeffding-style bound,

ε = sqrt( (1 / 2m) · ln(4n / δ) )

where m combines the two sub-window lengths harmonically (1/m = 1/n₀ + 1/n₁), n is the window length, and δ your tolerated false-alarm probability. When the difference exceeds ε, ADWIN declares drift, drops the stale half, and continues from the new regime — with logarithmic memory and amortized cost per observation, cheap enough to run live. Two tuning levers matter: δ — conservative values (~0.001) react only to major structural breaks (right for slow strategic models); looser values (~0.05–0.1) catch subtle shifts at the price of more false alarms (right for tactical signals); calibrate to how fast your strategy can afford to react — and s_min, a minimum segment/fold size acting as a warm-up floor: it guarantees enough data for estimation (and gradient convergence in stateful ML models, avoiding cold-start artifacts) and leaves room for the purge gap at each boundary.

Adaptive, regime-aligned validation

The frontier idea — worth knowing even if you adopt it late — is to let detected change-points define your validation folds instead of the calendar. Fixed splits ("retrain every 6 months") are arbitrary: a fold boundary lands mid-regime or a test window straddles two regimes purely by chronology. Adaptive time-series cross-validation instead segments history at statistically detected breaks (ADWIN online, or PELT offline), so each train/test fold is approximately regime-consistent, then applies the usual hygiene at every boundary:

• Purge gaps: a dead zone between train and test so lookback windows and overlapping labels can't bleed information across the boundary (mechanics in §09).
• State normalization: path-dependent strategies (open positions, inventory, trailing stops, grid levels) are reset to a canonical flat state at the start of each test fold — otherwise a "lucky" position carried across the boundary contaminates the fold's result. Your engine needs a reset_state() that genuinely clears everything (§06).
• Parameter locking: parameters are frozen before the fold's test segment runs — selection happens on train data only, with the timestamp of the lock recorded (§08's evidence pack).

Recent research on drift-aligned evaluation reports substantially more accurate out-of-sample error estimates than fixed rolling windows on several large-cap equities — with the improvement largest for simple models (linear models can't absorb drift internally, so honest folds help them most). Treat the specific magnitudes as study-dependent; the design logic stands on its own.

Advanced — the method-by-asset lesson The same research line found assets where adaptive validation made error estimates worse — names undergoing regime change so frequent and violent that no stable training history exists at all (high-drama single stocks are the canonical case). That is itself a diagnostic, and a valuable one: if performance depends on regime luck rather than a transferable mechanism, no validation scheme can rescue it — the asset (or strategy) is a candidate for a veto, not a cleverer split. Some things cannot be validated; the mature response is to not trade them.

Living with drift: passive vs active adaptation

Once deployed, a strategy handles non-stationarity in one of two modes — decide which yours uses, explicitly:

• Passive adaptation: continuous, scheduled re-estimation on a rolling window (e.g., refit monthly on trailing 3 years). Simple, predictable, and — crucially — it must be simulated identically inside the backtest (the walk-forward loop in §08 is exactly this). A backtest that fits once on all history and a live system that refits monthly are two different strategies.
• Active adaptation: a drift detector (ADWIN/CUSUM on prediction errors or strategy returns) triggers retraining, de-risking, or shutdown when change is detected. More responsive, more machinery, more false alarms to govern. Frequent detector triggers in production are also a message: your retraining cadence is too slow, or the edge is dying.

What this means at daily/swing scale — the practical distillation (1) Never shuffle; split chronologically with purge gaps. (2) Report per-year and per-regime performance tables with fixed parameters — a strategy you'd still deploy after seeing its 2008 and 2022 rows is a different animal from one you wouldn't. (3) Decide and simulate your re-estimation policy (usually: walk-forward with a fixed cadence). (4) Prefer regime-relative features (vol-normalized, rank-based) over absolute levels — they convert much concept-drift exposure into covariate shift, which is survivable. (5) Add a simple CUSUM/ADWIN monitor on live vs expected performance from day one of deployment (§16). Full adaptive fold construction is optional at your scale; the chronology, purging, and regime reporting are not.

06 Engine Architecture — Designing a Backtester That Can't Cheat

The engine's job is not speed or features. It is to make dishonesty structurally difficult: no path for future data to reach a decision, no fill the market wouldn't have given you, no dollar unaccounted for.

Vectorized vs event-driven

Use both, in that order — and require that the event-driven pass confirms the vectorized result before you trust it. A large gap between the two is diagnostic: it's usually costs, fills, or a leak in the vectorized prototype.

The timing contract

For a daily-bar system, one loop, engraved in stone:

**Close of day t — bar t finalized; universe & signals computed from data ≤ t → Overnight — portfolio constructor sizes target positions; orders generated → Open of day t+1 — fills simulated at open prices + slippage, participation-capped → During t+1 — stops/limits evaluated against bar t+1 with pessimistic intrabar rules → Close of t+1** — mark-to-market, accrue costs/borrow/interest, log equity

Variants exist (market-on-close execution is legitimate if the signal is computable from pre-close data — say, 3:50pm — and you model auction slippage), but every variant must answer the same question explicitly: at the moment this order is placed, what exactly was knowable? Write the answer into the engine, not into your habits.

Components and the "no peeking" boundary

Separate concerns so that correctness lives in one place each:

• DataFeed — owns all market data; exposes only as-of views: feed.history(sid, field, n_bars, asof=t). The strategy never touches a raw DataFrame. This is the single most valuable design decision in the whole system: look-ahead becomes an API violation instead of a subtle bug.
• Universe — answers "what was eligible on date t?" from point-in-time membership and liquidity filters (§04).
• Strategy — pure decision logic: consumes the as-of view, emits target positions or orders. No knowledge of fills, costs, or accounting. Must implement reset_state() that clears all internal memory (positions, trailing levels, counters) — required for honest fold boundaries (§05) and for restarts.
• PortfolioConstructor / Risk — turns desired positions into sized orders under constraints: max position %, sector caps, gross/net exposure, cash buffer (§10).
• ExecutionSimulator — the pessimist. Applies the fill rules and cost model (below). The only component allowed to know bar t+1's prices — and only when filling orders dated t.
• Accountant — cash, positions, dividends on pay dates, splits, borrow fees, margin interest, interest on idle cash. Maintains the invariant equity == cash + Σ(shares × price) at every bar, asserted, not assumed.
• Recorder / TrialRegistry — writes the run manifest, per-bar equity, and every fill to disk; appends every run to the registry that §09's statistics consume.

engine_core.py — a minimal honest daily engine (pedagogical skeleton)

from dataclasses import dataclass, field

@dataclass
class Fill:
    sid: str; date: object; shares: float; price: float; cost: float

@dataclass
class Portfolio:
    cash: float
    positions: dict = field(default_factory=dict)   # sid -> shares
    def equity(self, prices):                        # marked at close
        return self.cash + sum(sh * prices[sid]
                               for sid, sh in self.positions.items())

def run_backtest(feed, universe, strategy, sizer, costs, start, end):
    pf, equity_curve, fills = Portfolio(cash=100_000.0), [], []
    strategy.reset_state()
    days = feed.calendar(start, end)

    for t, t_next in zip(days[:-1], days[1:]):
        # 1) DECIDE using only data <= close of t
        view     = feed.asof(t)                      # as-of accessor
        eligible = universe.members(t)
        targets = strategy.target_weights(view, eligible)  # {sid: w}
        orders  = sizer.to_orders(targets, pf, view)       # share deltas

        # 2) FILL at open of t+1, pessimistically
        for sid, shares in orders.items():
            o   = feed.open_price(sid, t_next)             # unadjusted
            if o is None:            # halted/delisted -> handle explicitly
                continue
            cap = 0.05 * feed.adv_shares(sid, t)           # 5% ADV cap
            shares = max(-cap, min(cap, shares))
            px  = o * (1 + costs.slippage_bps(sid, shares, view) / 1e4
                         * (1 if shares > 0 else -1))
            fee = costs.commission(shares, px)
            pf.cash -= shares * px + fee
            pf.positions[sid] = pf.positions.get(sid, 0) + shares
            fills.append(Fill(sid, t_next, shares, px, fee))

        # 3) SETTLE close of t+1: corporate actions, carry, mark
        pf = feed.apply_corporate_actions(pf, t_next)      # divs, splits
        pf.cash *= (1 + feed.cash_rate(t_next) / 252)      # T-bill on cash
        closes = feed.close_prices(pf.positions, t_next)
        eq = pf.equity(closes)
        assert abs(eq - (pf.cash + sum(s * closes[k] for k, s
                   in pf.positions.items()))) < 1e-6       # identity
        equity_curve.append((t_next, eq))

    return equity_curve, fills

Everything real grows from this shape: stops become checks inside step 3 using bar t+1's range with pessimistic ordering; shorts add borrow accrual in step 3; partial fills carry an order book of residuals. What must never change is the phase separation — decisions in one phase, fills in the next, settlement after.

Fill rules: encode the pessimism

• Market orders: next open ± slippage (sign always against you).
• Limit orders: fill only if the bar trades through the limit by an epsilon — never on a touch (§03.7). If passive fills are the strategy's economics, bar data cannot validate it; you need intraday data and queue modeling, or downgrade the claim.
• Stops: for a long, fill at min(stop, open) when gapped; inside the bar, assume adverse ordering when both stop and target were touched.
• Latency/epsilon: even EOD systems have a decision→arrival delay; an extra slippage epsilon is the honest stand-in.
• Halts & delistings: no fill on halted days; delisted positions convert via the delisting event (proceeds or write-off), never silently disappear.

Cost models: pick a fidelity, state it

costs.py — a volatility- and size-aware cost model

import numpy as np

class CostModel:
    def __init__(self, half_spread_bps=2.0, vol_mult=0.10,
                 impact_Y=0.7, fee_per_share=0.0):
        self.hs, self.vm, self.Y, self.fps = (
            half_spread_bps, vol_mult, impact_Y, fee_per_share)

    def slippage_bps(self, sid, shares, view):
        sigma = view.daily_vol(sid)          # e.g. 0.02 = 2% daily vol
        adv   = view.adv_shares(sid)
        part  = abs(shares) / max(adv, 1)
        spread = self.hs                       # taker: half spread
        timing = self.vm * sigma * 1e4         # vol-scaled slippage
        impact = self.Y * sigma * np.sqrt(part) * 1e4   # square-root law
        return spread + timing + impact

    def commission(self, shares, price):
        return abs(shares) * self.fps          # + regulatory fees if sell

Whatever model you choose, expose its parameters in config and make the cost-multiplier sweep (0×/1×/2×/3×) a one-flag report. During incubation you will calibrate these numbers against your real fills (§16) — the model is a hypothesis too.

Reproducibility: the run manifest and the trial registry

Every run must be reconstructible and every run must be counted. Two artifacts, both automatic:

registry.py — no run goes unrecorded

import hashlib, json, subprocess, time

def run_manifest(config: dict, data_version: str) -> dict:
    cfg = json.dumps(config, sort_keys=True)
    return {
        "run_id":      hashlib.sha256(cfg.encode()).hexdigest()[:12],
        "utc_time":    time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "config":      config,
        "config_hash": hashlib.sha256(cfg.encode()).hexdigest(),
        "data_version": data_version,            # snapshot hash from §04
        "git_commit":  subprocess.check_output(
            ["git", "rev-parse", "HEAD"]).decode().strip(),
        "seed":        config.get("seed", 0),
    }

def log_trial(manifest, metrics, path="trials.jsonl"):
    with open(path, "a") as f:                   # append-only, forever
        f.write(json.dumps({**manifest, "metrics": metrics}) + "\n")

The registry is append-only and includes failures and abandoned ideas — its whole value is that it cannot flatter you. Its row count becomes N in §09's Deflated Sharpe Ratio; its timestamps become the "parameter locked before test ran" audit trail in §08's evidence pack. Same config + same data + same code must produce bit-identical results; anything nondeterministic (ML seeds, tie-breaking in rankings) is seeded from config.

Testing the tester

Your backtester is software that can be wrong, and its bugs are uniquely dangerous because they masquerade as alpha. Before believing any strategy result, make the engine pass a permanent suite:

• ☐ Buy-and-hold reconciliation: one liquid stock, buy day one, hold, zero costs → engine equity must match the total-return index to numerical precision. Catches dividend/split/accounting errors in one shot.
• ☐ Zero-edge null: random entry/exit signals with costs on → mean return ≈ −(modeled costs), Sharpe ≈ 0 across many seeds. If random signals make money, the engine is donating it.
• ☐ Known-answer test: a synthetic deterministic price path (e.g., a sawtooth) and a trivial rule, P&L computed by hand in a spreadsheet once → the engine must reproduce it exactly, forever (a regression test).
• ☐ Split/dividend fixtures: synthetic 2:1 split and a $1 dividend → equity continuous across the event, cash credited on pay date, share counts doubled.
• ☐ Look-ahead canary: feed the engine a "cheat" strategy that reads tomorrow's close via any available path. If it can, the API has a hole. This test should fail to compile/run.
• ☐ Accounting identity: cash + Σ(shares×price) == equity asserted every bar of every test (already in the skeleton above).
• ☐ Determinism: two runs, same config/data/seed → identical fills, byte-for-byte.

Research–live parity The gold standard (the design philosophy behind production frameworks like NautilusTrader and LEAN): the strategy code that ran in the backtest is the code that trades live, with only the data feed and execution adapter swapped. Every reimplementation between research and production is a chance for semantic drift — different rounding, different timing, different edge-case handling — which surfaces as unexplained live underperformance. If you build your own engine (§17's spec does), design the Strategy interface so a broker adapter can replace the simulator without touching strategy logic.

07 Metrics That Matter (and How They Deceive)

Metrics are compression, and compression hides things. Know what each number legitimately claims, what it quietly assumes, and which combinations expose the lies that any single number permits.

Return and risk: the core set

• CAGR — geometric annualized growth. The number that compounds. Always lower than the arithmetic mean of annual returns (by roughly half the variance) — quoting arithmetic averages of volatile returns overstates what you'll actually accumulate.
• Volatility — stdev of daily returns × √252. Assumes independent daily returns: strategies with autocorrelated P&L (trend followers, anything marked on stale prices) can show artificially low daily vol that balloons at monthly resolution. Check vol at two horizons; a large mismatch is information.
• Maximum drawdown & duration — worst peak-to-trough loss, and the longest time underwater. Duration is the underrated half: investors and traders abandon systems during long flat stretches more often than during sharp ones.
• Sharpe ratio — mean excess return over volatility, annualized. Three honesty requirements: subtract the risk-free rate (with cash yielding 4–5% in recent years, a "10% return" strategy is a 5% edge, not a 10% one); annualize from the native frequency (√252 × daily Sharpe — computing on monthly returns can flatter or punish depending on autocorrelation); include cost and cash drag.
• Sortino — penalizes only downside deviation; kinder to positively skewed strategies (trend) than Sharpe is. Calmar/MAR — CAGR ÷ max drawdown; the practitioner's "pain-adjusted" return.

Pitfall — Sharpe's blind spot is the tail Sharpe treats volatility as the only risk, so it rewards strategies that sell tail risk: steady small gains, rare catastrophic losses (short-vol profiles, aggressive mean reversion, naked premium selling). Such a strategy can print Sharpe 2+ for years and then delete itself in a week — the backtest window simply hadn't included its loss yet. Always read Sharpe together with skewness (negative = crash-prone profile), kurtosis (fat tails), and worst single day/week. §09's Probabilistic Sharpe Ratio bakes these higher moments into the confidence you assign.

Statistical honesty about the Sharpe you measured

A backtested Sharpe is an estimate with sampling error, roughly SE ≈ 1/√years (a touch larger once skew/kurtosis are accounted for). Handy consequence: t-stat ≈ Sharpe × √years. A Sharpe of 1.0 over 4 years is a t-stat of 2 — suggestive, not conclusive, and that's before any multiple-testing penalty (§09). A Sharpe of 0.7 over 25 years (t ≈ 3.5) is far stronger evidence than a Sharpe of 2.0 over one year (t = 2). Length beats height.

Drawdown has the mirror property: expected max drawdown grows with the window even when the strategy is unchanged — a 10-year backtest's max DD systematically understates the worst you'll see over a 30-year trading life. Never size positions to "the" historical max DD; size to the Monte Carlo distribution of drawdowns (§09), and assume the future draws from its bad tail.

Trade-level metrics: where diagnosis happens

• Number of trades — your sample size. Statistics on 60 trades are weather, not climate; hundreds are the entry ticket for confident inference, and complexity must be budgeted against it (§03.4).
• Expectancy per trade, in bps — the strategy's actual product, and the input to the break-even cost test: average P&L per trade must comfortably exceed the sum of expected spread, commission and slippage, or the strategy is execution-fragile and rejected regardless of its equity curve. Aim for edge ≥ 2–3× cost as the margin of safety against cost-model error (§03.5).
• Win rate × payoff ratio — read jointly (either alone is meaningless): 40% winners at 2.5:1 and 65% winners at 0.7:1 are both fine businesses. Which pair you have also predicts psychology: low win-rate systems demand tolerance for long losing streaks.
• Profit factor — gross wins ÷ gross losses. Below ~1.2 after costs is fragile; above ~2 on many trades deserves suspicion (revisit §03).
• Holding period & turnover — annual turnover × cost per trade = the annual friction bill; confirm the strategy you designed is the strategy you built (a "swing" system averaging 1.3 days of holding isn't one).
• Exposure — % of time in the market. A strategy that's invested 30% of the time and matches buy-and-hold's return is vastly better than it looks; compare return per unit of exposure.
• Trade P&L skew and concentration — remove the top 5 trades and recompute everything. Trend systems legitimately live on outliers (so this test is about knowing it, and the Monte Carlo skip-trade test in §09 quantifies it); mean-reversion systems that depend on a few home runs are broken.

Stability views

One full-period number hides regime failure. Standard views that should be in every report: per-calendar-year table (return, Sharpe, max DD, trades — the fastest regime check that exists), rolling 12-month Sharpe (is the edge persistent or one lucky cluster?), and underwater plot (drawdown over time — shows character: frequent shallow dips vs rare abysses).

Benchmarks and null models

Every strategy needs two comparisons, and most retail backtests run neither honestly:

• The passive alternative: total-return (dividends included!) buy-and-hold of the relevant index, plus a vol-matched version (index levered/delevered to the strategy's volatility). Beating the index while running half its vol is excellent; beating it by running 1.8× its vol is leverage, not skill. A regression of strategy returns on index returns splits the result into beta (rentable for ~0) and alpha (the part you built).
• The random twin: the same number of trades, same holding-period distribution, same exposure and sizing — but entries at random dates. Run it 1,000 times; your strategy's return should sit in the far tail of that distribution. This is the cleanest test that timing, specifically, is what you're being paid for — many "edges" are exposure to the market in disguise, and the random twin has the same exposure.

metrics.py — the core report, computed honestly

import numpy as np
import pandas as pd

def perf_report(daily_ret: pd.Series, rf_daily: pd.Series,
                trades: pd.DataFrame) -> dict:
    ex = daily_ret - rf_daily                     # EXCESS returns
    yrs = len(daily_ret) / 252

    equity = (1 + daily_ret).cumprod()
    cagr   = equity.iloc[-1] ** (1 / yrs) - 1
    vol    = daily_ret.std() * np.sqrt(252)
    sharpe = ex.mean() / ex.std() * np.sqrt(252)

    peak   = equity.cummax()
    dd     = equity / peak - 1
    max_dd = dd.min()
    dur    = (dd < 0).astype(int)                 # longest underwater run
    max_dur = (dur.groupby((dur == 0).cumsum()).cumsum()).max()

    pnl = trades["pnl"]
    pf  = pnl[pnl > 0].sum() / max(1e-9, -pnl[pnl < 0].sum())

    return {
        "years": round(yrs, 1),  "CAGR": cagr,   "vol": vol,
        "sharpe": sharpe,        "t_stat": sharpe * np.sqrt(yrs),
        "max_dd": max_dd,        "underwater_days": int(max_dur),
        "calmar": cagr / abs(max_dd),
        "skew": daily_ret.skew(), "kurtosis": daily_ret.kurt() + 3,
        "n_trades": len(trades), "profit_factor": pf,
        "expectancy_bps": pnl.divide(trades["notional"]).mean() * 1e4,
        "exposure": (daily_ret != 0).mean(),
        "worst_day": daily_ret.min(),
    }

The minimum honest report

Adopt a fixed report card — the same table for every strategy, so nothing can be quietly omitted: full-period core metrics (with t-stat), per-year table, rolling Sharpe, underwater plot, trade-level stats with expectancy-vs-cost, cost-multiplier sweep (0×/1×/2×/3×), benchmark and random-twin comparisons, and — after §09 — the Deflated Sharpe Ratio with the trial count that produced it. If a strategy's pitch omits any row, ask why.

08 The Research Workflow — From Idea to Verdict

Strategies don't fail at the coding step; they fail at the process level — evidence contaminated by iteration, gates invented after seeing results. This section is the pipeline that keeps evidence clean: In-Sample development → Walk-Forward validation → a one-shot Out-of-Sample verdict, with pre-committed gates at each boundary.

**0. Hypothesis — mechanism written down first → 1. Split design — IS / validation / lockbox frozen → 2–4. Develop (IS) — prototype → event-driven → sensitivity → 5–7. Validate — walk-forward, Monte Carlo, regimes → 8. Statistics — DSR vs trial registry (§09) → 9. Lockbox — one shot, pre-committed gates → 10–11. Incubate → Live** — paper trade, then small (§16)

Step 0 — Write the hypothesis before touching data

One page, written in advance, containing: the mechanism (who is on the other side and why they'll keep paying — a risk premium being earned, a behavioral bias, a structural constraint like forced index flows or month-end rebalancing); the predicted shape (expected Sharpe range, turnover, where it should work and — critically — where it shouldn't); and the falsifier (what result would make you abandon it). A hypothesis that can't fail isn't one. This page later becomes the header of your evidence pack, and its "shouldn't work" predictions become free robustness tests: a signal that also "works" where its mechanism can't exist is measuring a bias, not an edge.

Step 1 — Design the data splits before looking

• In-sample (IS) / development set — e.g., 2003–2017. You may do anything here: explore, fit, iterate, mine. Its results are for steering, never for reporting.
• Validation set — e.g., 2018–2022. Visited sparingly, at declared checkpoints, to test frozen candidates. Every visit is logged; each one spends information (after enough visits it is the training set — budget perhaps a handful of visits per project, and mean it).
• Lockbox / final OOS — e.g., 2023–2026. Touched exactly once, by the finished system, with pass/fail gates written down beforehand. Whatever happens, you do not "fix and re-try" against it — a failed lockbox sends the idea back, and the lockbox is only trustworthy again once enough new data has accrued or the redesign is fundamental.

Ensure the splits jointly span regimes (§05); a lockbox that's one long bull market can't test what you need tested.

Steps 2–3 — Prototype fast, then rebuild honest

Prototype vectorized on IS data for speed (§06); demand only crude viability — plausible gross edge, sane turnover. Then port to the event-driven engine with full costs and pessimistic fills, and reconcile the two: they should agree within costs; unexplained gaps are bugs or leaks and must be resolved before proceeding. Run the cost-multiplier sweep now — a strategy that dies at 2× costs dies here, cheaply.

Step 4 — Parameter sensitivity: plateaus, not peaks

Grid the 1–3 parameters that matter over economically sensible ranges and map performance. You are not searching for the best cell; you are asking whether a stable region exists. A useful formalization: define the plateau as all parameter sets achieving at least ~90% of the best cell's Sharpe, and require that it be contiguous and wide — then deploy from the center of that region, not the peak. Add an explicit cliff veto: examine each candidate's one-step neighbors in the grid; if any neighbor's Sharpe collapses (or its drawdown explodes) relative to the candidate, reject the candidate even if its own numbers are excellent. A parameter set surrounded by cliffs is a coincidence; a slight future shift in market behavior is equivalent to nudging the parameter — and you've already measured what happens then.

Step 5 — Walk-forward analysis: simulate the re-estimation you'll actually do

A single backtest with one parameter fit over all history answers a question you'll never face live. What you'll actually do is periodically refit on trailing data and trade the result forward — so simulate exactly that:

walk_forward.py — rolling refit, honest splice

import numpy as np
import pandas as pd

def walk_forward(data, fit_fn, run_fn, train_years=5, test_years=1,
                 purge_days=10, anchored=False):
    """fit_fn(train_slice) -> params ; run_fn(test_slice, params) -> daily returns"""
    results, folds = [], []
    start = data.index.min()
    step  = pd.DateOffset(years=test_years)
    t0    = start + pd.DateOffset(years=train_years)

    while t0 + step <= data.index.max():
        tr_start = start if anchored else t0 - pd.DateOffset(years=train_years)
        train = data.loc[tr_start : t0 - pd.Timedelta(days=purge_days)]
        test  = data.loc[t0 : t0 + step]        # params frozen before this
        params = fit_fn(train)                  # LOG the lock timestamp
        oos    = run_fn(test, params)           # state reset inside run_fn
        folds.append({
            "test_start": t0, "params": params,
            "is_sharpe":  sharpe(run_fn(train, params)),
            "oos_sharpe": sharpe(oos),
        })
        results.append(oos)
        t0 += step

    stitched = pd.concat(results)               # the only curve you report
    return stitched, pd.DataFrame(folds)

Mechanics that matter: rolling vs anchored windows (rolling adapts and matches how most people trade; anchored uses all history and suits slow-moving edges — pick by your drift beliefs from §05, don't try both and keep the winner, that's a trial); a purge gap between train and test (§09); state reset at each fold start (§05); and reporting only the stitched OOS curve — the IS numbers are scaffolding.

Read the fold table through two lenses:

• Resilience ratio per fold: η = OOS Sharpe / IS Sharpe (Pardo's classic walk-forward efficiency). A genuine edge degrades but survives: ≥ 0.6 average retention is the healthy zone; below ~0.5 is the classic red flag that the fitting step memorized noise rather than discovering structure. Also watch η's trend — steadily decaying resilience is an edge dying in real time.
• Consistency: the fraction of profitable folds, and whether failures cluster in identifiable regimes (fine, if the mechanism predicts it) or scatter randomly (worse).

Pitfall — meta-overfitting the walk-forward Walk-forward results are out-of-sample only for the strategy as configured. If you inspect the stitched curve, tweak the rule, and re-run the walk-forward — repeatedly — you are now fitting to the OOS data with extra steps. Every walk-forward run is a trial; it goes in the registry, and it's another reason the lockbox exists.

Step 6 — Monte Carlo stress (§09 for code)

The historical path is one draw. Before gating, generate the distribution around it: trade-order reshuffles (drawdown distribution), block bootstrap of returns (confidence intervals on Sharpe and DD), skip-trade tests (dependence on a few lucky fills), and entry jitter — shift every entry ±1 day at random; a daily/swing edge that dies under one day of jitter is timing luck, not signal.

Step 7 — Regime & universe robustness

With parameters frozen: per-year and per-regime tables (§05), the 2008/2020/2022 windows explicitly, adjacent universes (mid-caps if developed on large; a related market), and the strategy's own "shouldn't work here" predictions from Step 0. You are mapping the operating envelope — the goal is not that everything is green, but that the pattern of green and red matches the mechanism you claimed.

Step 8 — Statistical accounting

Pull the trial registry: N variants tried (across the whole project, including abandoned branches), then compute the Deflated Sharpe Ratio and PBO (§09). This is the step that converts "my best backtest looks great" into "my best backtest is unlikely to be the best of N coin flips." Numbers below threshold → back to research with a genuinely new hypothesis, not a re-roll of the same one.

Step 9 — The lockbox verdict: pre-committed gates

Before running the lockbox, write the gates into the repo (a dated, committed file — your future self will be tempted to negotiate). A robust gate design uses majority-pass plus a catastrophic veto:

• Majority-pass: across the lockbox folds/sub-periods, at least ⅔ must clear the pre-set benchmark vector — e.g., Sharpe above threshold, max DD below limit, Calmar above floor, and a minimum trade count (so a fold can't "pass" on three lucky trades). Demanding every sub-period pass rejects nearly everything (markets have bad quarters); demanding only the average lets one great year carry two awful ones.
• Catastrophic veto: regardless of averages, any single fold that trips a kill condition — drawdown beyond your stated tolerance, a per-trade loss implying broken risk control — fails the whole candidate. Some failures are not offset by other successes.

Typical gates for a daily/swing candidate: stitched OOS Sharpe ≥ ~0.7 net; OOS max DD ≤ 1.5× the IS max DD; resilience ratio ≥ 0.5; DSR ≥ 0.95; random-twin percentile ≥ 95th; survives 2× costs. Calibrate to your own risk tolerance — the point is not these exact numbers, it's that yours are written down before you look.

Steps 10–11 — Incubation, then small

Paper-trade the frozen system on live data for a meaningful period (3–6 months for a swing system — enough trades to compare distributions, not enough to prove much; what it really tests is operations: data arrival, order generation, fills vs cost model, your own adherence). Then deploy at a fraction of target size with pre-written kill criteria. §16 covers both in depth.

The evidence pack

The deliverable of the whole pipeline is not an equity curve — it's an auditable case file. For every strategy that reaches the lockbox, the pack contains: the Step-0 hypothesis page; the split design and every logged validation-set visit; the run manifests (config hash, data version, git commit — §06) for each stage; the sensitivity heatmap with the plateau/cliff analysis; the walk-forward fold table with parameter-lock timestamps; the Monte Carlo distributions; regime tables; the trial count and DSR; the pre-committed gate file and the lockbox result against it. If a claim in the pack can't be traced to a manifest, it doesn't exist. This sounds heavyweight; with the registry and report automation from §06 it is mostly a by-product — and it is exactly what §17's spec tells your AI-built system to produce automatically.

The one-line summary of this section Iterate freely where iteration is cheap (IS), spend looks like money where they're scarce (validation), and let data you never touched deliver a verdict you wrote down in advance (lockbox). Everything else is bookkeeping to make that honest.

09 Statistical Validation — Separating Skill from Selection

You searched, therefore your best result is inflated. This section is the toolkit that quantifies by how much: what the luckiest of N tries looks like, how to deflate a Sharpe ratio for the search that found it, and how to build distributions where history gave you only one path.

The expected maximum Sharpe of pure noise

Suppose you test N strategies that are all, in truth, worthless (expected return zero). Each backtest still produces a Sharpe estimate scattered around zero with standard deviation ≈ 1/√years. The maximum of N such draws is not near zero — extreme-value theory gives its expectation as approximately

E[max SR] ≈ σ_SR · [ (1−γ)·Z⁻¹(1−1/N) + γ·Z⁻¹(1−1/(N·e)) ] ≈ σ_SR·√(2·ln N)

where γ ≈ 0.5772 (Euler–Mascheroni) and σ_SR is the sampling noise of a single trial's Sharpe. The bracket evaluates to ≈ 2.5 standard deviations for N = 100 and ≈ 3.26 for N = 1,000. Consequences, in annualized terms:

• 1,000 worthless variants on 10 years of data (σ_SR ≈ 1/√10): best expected Sharpe ≈ 1.0 — a "deployable" strategy, made of nothing.
• The same 1,000 on 2 years of data: best expected Sharpe ≈ 2.3. Short samples + big searches manufacture monsters.
• Inverting the formula gives the minimum backtest length: to keep noise's best below a target Sharpe s after N trials you need roughly 2·ln(N)/s² years — after ~100 trials, hunting a Sharpe-1 strategy needs ≳ 7–10 years of data; after ~1,000 trials, ≳ 11–14. Every extra order of magnitude of search demands more history than you probably have.

noisemaxsharpe.py — reproduce the scary number yourself

import numpy as np

rng = np.random.default_rng(7)
N_TRIALS, YEARS, D = 1000, 10, 252
T = YEARS * D

# N strategies of pure noise: daily returns, zero mean, 1% vol
rets = rng.normal(0.0, 0.01, size=(N_TRIALS, T))
sr_annual = rets.mean(1) / rets.std(1) * np.sqrt(D)

print(f"best 'strategy' Sharpe : {sr_annual.max():.2f}")
print(f"top decile threshold   : {np.quantile(sr_annual, 0.9):.2f}")
# Typical output: best ~1.0-1.2 -- with ZERO true edge.
# Now recall §08: every grid cell and abandoned idea was one of these.

PSR and the Deflated Sharpe Ratio

Two refinements turn a raw Sharpe into a probability statement (Bailey & López de Prado):

• Probabilistic Sharpe Ratio (PSR): the probability that the true Sharpe exceeds a benchmark SR*, given the estimate's sampling error — which widens with short samples, negative skew, and fat tails. This is where the §07 warning about crash-shaped strategies becomes arithmetic: negative skew directly shrinks your confidence.
• Deflated Sharpe Ratio (DSR): the PSR evaluated against a benchmark that is not zero but the expected maximum Sharpe of your search — the noise ceiling from above, computed from the number and variance of your trials. DSR answers the only question that matters after a search: is my best result better than the best of N coin flips?

deflated_sharpe.py — PSR & DSR (per-period inputs, e.g. daily)

import numpy as np
from scipy import stats

EULER_GAMMA = 0.5772156649015329

def psr(sr_hat, sr_star, T, skew, kurt):
    """P(true SR > sr_star). Non-annualized (per-period) SR; kurt: normal=3."""
    num = (sr_hat - sr_star) * np.sqrt(T - 1)
    den = np.sqrt(1 - skew * sr_hat + (kurt - 1) / 4 * sr_hat**2)
    return stats.norm.cdf(num / den)

def expected_max_sr(n_trials, var_trial_sr):
    """Noise ceiling: E[max SR] across n_trials zero-skill strategies."""
    z = stats.norm.ppf
    e = np.e
    return np.sqrt(var_trial_sr) * (
        (1 - EULER_GAMMA) * z(1 - 1 / n_trials)
        + EULER_GAMMA      * z(1 - 1 / (n_trials * e)))

def dsr(returns, all_trial_srs):
    """returns: daily series of the BEST strategy.
       all_trial_srs: per-period Sharpes of EVERY trial in the registry."""
    r  = np.asarray(returns)
    sr = r.mean() / r.std()
    sr0 = expected_max_sr(len(all_trial_srs), np.var(all_trial_srs))
    return psr(sr, sr0, len(r),
               stats.skew(r), stats.kurtosis(r, fisher=False))

# Verdict: dsr >= 0.95 -> best trial is unlikely to be pure selection.
# dsr < 0.95   -> indistinguishable from the luckiest of your N tries.

Counting N honestly — correlated trials Your 500 grid cells are not 500 independent tries (a 20-day and a 21-day lookback are nearly the same strategy). The effective N lies between the number of genuinely distinct ideas and the raw registry count; it can be estimated by clustering the trial-return correlation matrix and counting clusters. Practical policy: compute DSR with both the raw count (conservative) and the cluster count (realistic) — if the verdict differs, you're in the gray zone and the lockbox decides. What's never acceptable is N = 1 after a month of searching.

Haircuts, FWER and FDR — the other school

Harvey & Liu attack the same problem with classical multiple-testing corrections, "haircutting" a Sharpe ratio by the volume of tests behind it. Two error philosophies:

Their famous headline translates the whole apparatus into one heuristic: given how heavily the field has mined the same data, a newly "discovered" strategy should clear roughly t ≈ 3 (recall t ≈ Sharpe × √years, §07) — not the t = 2 of single-hypothesis textbooks. A Sharpe of 1.0 needs ~9–10 years of honest history to reach that bar. For a solo researcher deploying one or two systems, DSR ≥ 0.95 and t ≥ 3 point the same direction: either bring long data, or bring low N.

Purged CV, CPCV, and the Probability of Backtest Overfitting

Purging fixes a leak specific to finance: labels span time. If Monday's training example is labeled with "return over the next 5 days," its label contains Wednesday's price — and if Wednesday is in your test set, train and test share information despite being 'different rows.' Purging deletes training samples whose label windows overlap the test window. Embargoing extends the cut: because markets carry memory (volatility clustering, §05), an additional buffer (~1% of the sample) after the test block is dropped from training. Any train/test boundary in this guide — walk-forward folds, CV splits, the lockbox edge — gets a purge gap; ML pipelines with overlapping labels need it doubly (§13).

Combinatorial Purged CV (CPCV) generalizes walk-forward's single history: split the sample into S sequential blocks; for every combination of blocks-as-test (all C(S, k) ways), train on the rest with purging at each boundary. Instead of one out-of-sample path you get many reconstructed paths — a distribution of Sharpe ratios rather than a point. From it comes the Probability of Backtest Overfitting (PBO): across the combinatorial splits, select the best in-sample variant, then look at its rank out-of-sample among all variants. PBO = the fraction of splits where the in-sample winner lands in the bottom half out-of-sample. A strategy selection process with PBO near 0.5 is a ranking machine with no memory of skill — its "best" means nothing; practitioners want PBO well below ~0.2, and treat higher values as a rejection of the selection process, not just one strategy.

Scope note: for a simple daily rule with non-overlapping outcomes, disciplined walk-forward plus DSR covers most of the risk, and CPCV can be overkill. The moment you have ML models, overlapping labels, or a large candidate pool being ranked, CPCV/PBO stop being optional refinements and become the honest default.

Bootstrap & Monte Carlo: distributions from one history

Naïve bootstrapping (resampling single days) destroys the serial structure — volatility clustering, momentum — that your strategy may live on. The stationary bootstrap (Politis–Romano) resamples blocks of consecutive returns with random, geometrically distributed lengths, preserving short-range dependence while generating thousands of plausible alternate histories:

stationary_bootstrap.py — CIs for Sharpe, distribution for drawdown

import numpy as np

def stationary_bootstrap(returns, n_sims=2000, avg_block=20, seed=0):
    r, T = np.asarray(returns), len(returns)
    rng = np.random.default_rng(seed)
    p = 1.0 / avg_block                       # geometric block lengths
    sims = np.empty((n_sims, T))
    for s in range(n_sims):
        idx, t = rng.integers(T), 0
        while t < T:
            L = min(rng.geometric(p), T - t)
            take = (np.arange(idx, idx + L)) % T      # wrap around
            sims[s, t:t+L] = r[take]
            t += L
            idx = rng.integers(T)
        # note: block start resampled after each block
    return sims

def summarize(sims, D=252):
    sr = sims.mean(1) / sims.std(1) * np.sqrt(D)
    eq = np.cumprod(1 + sims, axis=1)
    dd = (eq / np.maximum.accumulate(eq, axis=1) - 1).min(1)
    return {
        "sharpe_ci_5_95":  (np.quantile(sr, .05),  np.quantile(sr, .95)),
        "maxdd_median":     np.quantile(dd, .50),
        "maxdd_5pct_worst": np.quantile(dd, .05),   # plan for THIS one
    }

# Gate: if the 5th-percentile Sharpe is <= 0, the strategy is
# statistically indistinguishable from noise. Size risk to the
# 5th-percentile drawdown, not the single historical path's.

Complete the Monte Carlo suite with tests that target specific fragilities:

• Trade-order reshuffle: permute the sequence of trade P&Ls many times and recompute drawdowns. Same trades, different order → the drawdown distribution, exposing how much your historical max DD owed to lucky sequencing.
• Skip-trade: randomly drop 10–20% of trades per simulation. A robust system's profits shrink proportionally; a fragile one — dependent on a handful of home runs it might have missed — collapses. (Read jointly with §07's concentration check; for trend systems some concentration is the design.)
• Entry jitter: shift each entry ±1 day at random. The single most informative robustness test for daily/swing systems: real edges have temporal width; artifacts are one day wide.
• Noise injection: perturb prices by a small fraction of daily ATR and re-run the full backtest repeatedly. Rules balanced on exact thresholds die here; keep strategies whose results degrade smoothly.
• Random-twin cohort: the exposure-matched random strategy from §07, run 1,000× — your result should sit at or beyond its 95th percentile, or your "edge" is dressed-up market exposure.

The statistical gauntlet, assembled A candidate earns belief when, simultaneously: DSR ≥ 0.95 against the honest trial count; t ≈ SR·√years ≥ 3 (or the FDR-adjusted equivalent); PBO low where a selection process was involved; bootstrap 5th-percentile Sharpe > 0; and it survives reshuffle, skip-trade, jitter and noise with graceful degradation. This reads as harsh. It is calibrated to a field where almost everything that glitters is selection bias — and it is precisely the gauntlet §17's spec automates so that running it costs you one command.

10 Position Sizing, Portfolio Effects & Capacity

Two systems with identical signals and different sizing are different strategies — with different Sharpes, different drawdowns, and different verdicts. Sizing belongs inside the backtest, not bolted on after.

Sizing methods, in order of sophistication

• Equal weight, fixed N positions. Honest baseline; never embarrassing. Its hidden flaw: a 20%-vol tech name and an 8%-vol utility get equal capital, so your risk is secretly concentrated in the volatile names.
• Fixed-fractional risk ("R" sizing). Risk a constant fraction of equity per trade — shares = (equity × r%) ÷ stop distance, with r typically 0.25–1% for swing systems. Normalizes every trade to the same loss-if-wrong; makes results readable in R-multiples. Requires stops that are honest about gaps (§03.7).
• Volatility-scaled positions. Size inversely to the name's recent volatility (ATR or σ), so each position contributes similar risk. The workhorse of systematic equity trading.
• Portfolio volatility targeting. Scale gross exposure so the portfolio's trailing vol tracks a target (e.g., 15% annualized). Smooths the ride across regimes and usually improves risk-adjusted metrics — but it's a feedback loop on estimated vol: it de-levers after vol spikes, and it adds parameters (estimation window, cap/floor) that must be frozen like all others.
• Kelly, handled with tongs. The growth-optimal fraction (≈ edge/variance) assumes you know the edge. You have an estimate, biased upward by everything in §03 and §09, of a quantity that drifts (§05). Full Kelly on an overestimated edge over-bets catastrophically, and the penalty is asymmetric: betting half of optimal costs a little growth; betting double is ruin-bound. Practitioners who use it at all use quarter-to-half Kelly. Treat Kelly as an upper bound to stay far below, not a target.

sizing.py — vol-scaled positions under a portfolio vol target

import numpy as np

def target_shares(equity, signals, view, n_max=20,
                  pos_vol_target=0.02, port_vol_target=0.15,
                  max_weight=0.10, adv_cap=0.05):
    """signals: {sid: strength in [-1, 1]} decided at close t."""
    picks = sorted(signals, key=lambda s: -abs(signals[s]))[:n_max]
    weights = {}
    for sid in picks:
        sigma = view.daily_vol(sid)              # e.g. 0.025
        w = pos_vol_target / max(sigma, 1e-4)    # equal-risk weight
        weights[sid] = np.sign(signals[sid]) * min(w, max_weight)

    # scale whole book toward the portfolio vol target (crude but honest:
    # uses trailing realized portfolio vol, capped at 1x gross)
    port_vol = view.portfolio_vol(weights)       # annualized estimate
    scale = min(1.0, port_vol_target / max(port_vol, 1e-4))

    shares = {}
    for sid, w in weights.items():
        px  = view.close_unadj(sid)
        qty = (equity * w * scale) / px
        cap = adv_cap * view.adv_shares(sid)     # liquidity cap (§03.6)
        shares[sid] = float(np.clip(qty, -cap, cap))
    return shares

Portfolio-level realities

• Correlation is the portfolio killer. Twenty positions from one momentum screen are not twenty bets — in a selloff they are one bet, twenty times. Measure average pairwise correlation of positions and effective number of independent bets; cap sector/theme concentration; and stress the book against "all my signals are the same signal" days (2020-03 and 2022 supply real rehearsals).
• Constraint realism. Whole shares on small accounts (or confirm fractional support), a cash buffer for gaps and slippage, margin rules if levered (US Reg-T: 2× overnight buying power; maintenance calls modeled, not assumed away; pattern-day-trader rules if you day trade a small US margin account), and borrow availability for every short (§03.10).
• Rebalancing cadence. Calendar rebalancing (weekly/monthly) vs threshold rebalancing (act when drift exceeds a band). Thresholds usually cut turnover for the same tracking; either way, the cadence is a parameter — freeze it with the others.
• Compounding vs fixed-notional. Run reports both ways: fixed-notional isolates the per-trade edge from equity-curve luck (early wins inflate compounded results); compounding is what your account experiences and what capacity constraints bite on (§03.6's silently-growing-orders trap). Disagreement between the two views is itself diagnostic.

Market impact and the square-root law

Impact is why results don't scale with capital. The empirical regularity, robust across decades and venues (Almgren, Bouchaud and others): executing a total quantity Q in a stock with daily volatility σ and average daily volume V moves the price against you by approximately

impact ≈ Y · σ_daily · √(Q / V), with Y of order 0.5–1

Concave: the first shares are the cheap ones, and doubling size raises impact cost per share by ~41%. Two components matter for modeling: a temporary part (pressure that decays after you stop executing — the Obizhaeva–Wang picture of shock and exponential decay) and a permanent part (information your flow revealed; the price doesn't fully come back). A second regularity (Almgren): executing faster costs more — unit cost scales roughly with 1/√T of execution time — so impact can be traded against timing risk by spreading orders, which a daily-bar backtest approximates with participation caps and patient-execution assumptions (§06). At personal scale in liquid large caps (say, orders under ~1% of ADV), impact is a rounding error and the spread/slippage model carries the weight; in small caps or at fund scale it becomes the dominant cost — and the reason the next subsection exists.

Capacity: how much money the edge can carry

As AUM grows, impact costs grow concavely-but-relentlessly while gross alpha doesn't. Institutional practice names the milestones (Vangelisti's hierarchy):

The driver is turnover: costs scale with turnover × AUM, so high-turnover strategies hit their ceilings at far smaller size — which is why successful fast strategies are self-limiting, and why firms drift toward slower signals as they grow. A quick top-down bound for a cross-sectional equity strategy: capacity ≈ participation limit × Σ(tradable dollar volume of the universe), refined by how much of the book turns over per day. For a personal-scale swing account this rarely binds in the S&P 1500 — but it binds immediately if your backtest's alpha lives in $3M-a-day micro-caps, which is exactly §03.6's warning wearing a suit.

capacity.py — back-of-envelope terminal capacity

def capacity_estimate(gross_alpha_annual, turnover_annual,
                      Y, sigma_daily, adv_dollars, participation):
    """Terminal capacity: AUM where impact cost eats gross alpha.
    Impact per traded dollar ~ Y * sigma * sqrt(participation);
    annual cost ~ turnover * impact. Solve for the participation
    at which cost == alpha, then convert to AUM."""
    max_impact_per_trade = gross_alpha_annual / max(turnover_annual, 1e-9)
    part_at_zero = (max_impact_per_trade / (Y * sigma_daily)) ** 2
    part = min(part_at_zero, participation)      # respect your own cap
    return part * adv_dollars                    # deployable per name

# Example: 4% gross alpha, 12x turnover, Y=0.7, 2% daily vol,
# $20M ADV name, 5% participation cap:
# max impact 33bps/trade -> part (0.0033/0.014)^2 ~ 5.6% -> capped 5%
# -> ~$1M per name. 20 names -> ~$20M strategy capacity. Small-cap
# universes shrink this 10-100x. Run it before falling in love.

11 Strategy Archetypes and Their Specific Traps

Every strategy family has a characteristic shape — and a characteristic way its backtests lie. Knowing the archetype tells you which section of the bias catalog to point at it first.

Momentum / trend following

Mechanism: persistent underreaction and herding; winners keep winning over 3–12 month horizons. One of the most replicated effects in the literature — which also means it's crowded and its raw form decays.

Shape: low win rate (35–45%), strong positive skew, profits concentrated in a few big runners, low-to-moderate turnover, brutal whipsaw stretches in choppy regimes, sharp drawdowns at trend reversals ("momentum crashes" — 2009's snap-back is the canonical scar).

Backtest traps: profit concentration means the skip-trade Monte Carlo (§09) is the decisive test — if missing five trades kills a decade, you must be certain you'd have caught those five. Regime dependence is structural (§03.9): report chop years honestly. Gap entries on breakout days incur above-average slippage — cost the entries at stressed levels, not averages.

Validate hardest: per-regime tables, skip-trade, entry jitter, cost stress at entries.

Mean reversion (pullback/oversold systems)

Mechanism: liquidity provision — getting paid for absorbing short-term panic and overshoot in individual names.

Shape: the mirror of trend: high win rate (55–70%), negative skew, many small wins punctuated by occasional large losses when a "dip" is actually new information (the falling knife with a reason). Short holding periods, high turnover — so costs decide everything.

Backtest traps — the most bias-prone archetype in this guide: survivorship bias flatters it enormously (§03.2: every dip in a survivors-only file recovered, by construction); limit-touch fill fantasy is endemic (§03.7: passive fills you'd never have gotten, adversely selected); high turnover magnifies any cost-model optimism (§03.5); and its negative skew inflates Sharpe while hiding the tail (§07). Earnings dips are a separate population from flow-driven dips — a system that can't tell them apart is averaging two different games (§12).

Validate hardest: delisted-inclusive data, trade-through fills only, 2×–3× cost sweep, worst-day/week analysis, PSR with the skew it actually has.

Breakout systems

Mechanism: range expansion after compression; participation in the early phase of new trends.

Shape: trend-like (low win rate, positive skew) with even more sensitivity to entry mechanics.

Backtest traps: the intrabar ambiguity (§03.7) is chronic — the breakout bar often contains both the trigger and the pullback that would have stopped you; pessimistic ordering is mandatory. Slippage at the trigger moment is systematically worse than average (everyone's stop-entry fires together — momentum ignition); use elevated entry costs. Threshold rules ("20-day high") are exactly the kind of knife-edge that §09's noise-injection test exposes.

Validate hardest: intrabar pessimism, noise injection, elevated entry slippage.

Pairs & statistical arbitrage

Mechanism: relative-value convergence between economically linked names; hedged against market direction.

Shape: market-neutral, steady small P&L, negative skew via divergence blowups (a pair "breaking" structurally — merger, fraud, business-model shift — trends against you indefinitely).

Backtest traps: pair selection is a multiple-testing bomb — testing 5,000 pairs for cointegration is 5,000 trials (§03.3, §09), and in-sample cointegration is cheap: demand it persist out-of-sample. The short leg carries every §03.10 reality (borrow, buy-ins, squeezes). Costs double (two legs). Divergence risk needs explicit stops-on-spread, and the "spread" must be built from prices you could trade at, not mid-quotes.

Validate hardest: DSR with the true pair-trial count, OOS persistence of the relationship, borrow-cost stress, structural-break vetoes (§05).

Seasonality & calendar effects

Mechanism (when real): structural flows — month-end rebalancing, tax-date behavior, index reconstitution, options expiry mechanics.

Backtest traps: the purest data-mining minefield in trading. The calendar offers thousands of testable patterns (every weekday × month × holiday-adjacency combination), so something always "works" historically — §09's expected-max-of-noise arithmetic applies with full force. Demand: a named mechanism, survival across decades and markets, and an effect size that clears costs with margin (many calendar effects are real but only a few bps — priced away or untradable).

Validate hardest: mechanism-first, cross-market replication, DSR against the honest number of calendar combinations examined.

Event-driven (earnings and news)

Mechanism: systematic under/overreaction to scheduled information — post-earnings-announcement drift being the classic.

Backtest traps: everything hinges on event timestamps (§04): announcement date and session (before open vs after close) must be point-in-time correct, and historical earnings-calendar data is notoriously dirty — a one-session error flips "trading the reaction" into "trading before the news," a look-ahead that prints money on paper. Overnight gap risk is unhedgeable at daily granularity; halts happen; spreads around events are wide. Verify a sample of event dates by hand against filings before trusting any vendor.

Validate hardest: manual timestamp audits, session-aware execution, event-window cost stress.

Factor / rotation portfolios (monthly-weekly rebalance)

Mechanism: harvesting documented cross-sectional premia (value, momentum, quality, low-vol) via periodic re-ranking.

Backtest traps: point-in-time universe and fundamentals are the whole game (§03.2, §04) — restated financials and backfilled index membership fabricate most naive factor alpha; rebalance-day assumptions matter (everyone trades the same reconstitution days — model the crowding cost); and factor definitions have their own multiple-testing history (hundreds of published "factors," few survive out-of-sample — the field's own replication crisis is your prior).

Validate hardest: PIT data end-to-end, turnover-cost realism, long-horizon OOS (factors cycle over years, not months).

Cross-archetype rule Notice the pattern: each archetype's flattering bias matches its mechanism. Mean reversion is flattered by survivorship and fill fantasy; trend by regime selection and skipped-trade luck; seasonality by multiple testing; events by timestamp leaks. Before backtesting any strategy, ask: "if this family has a signature lie, what is it?" — then run that test first. It's the fastest route to a verdict, and §17's spec includes the mapping so your system can run the right gauntlet automatically.

12 Daily & Swing Specifics — Your Operating Domain

End-of-day signals, holds of days to weeks. The sweet spot for individual algorithmic traders: enough inefficiency to matter, slow enough that costs and latency don't dominate, small enough data to run on a laptop. Here are the domain-specific decisions.

Choose and freeze your execution convention

Pick one, encode it in the engine, and never compare strategies across conventions — the convention is worth tens of bps per trade and will masquerade as edge. And obey §03.7's granularity law: the data must tick finer than the strategy decides. Daily bars honestly serve daily decisions with next-bar fills; the moment your orders live inside the bar, bar data starts inventing fills that never happened.

Gaps are the tax of holding overnight

Roughly a third of a typical stock's variance realizes between close and open — where your stops don't exist. Consequences for design and testing: stops cap intraday loss only (model gap-throughs at the open, §03.7); position sizing must assume losses beyond the stop (size to gap scenarios, not stop distance alone — a −8% overnight gap on a 25% position is −2% of equity before your "risk management" wakes up); and earnings are the scheduled gap generator, which deserves its own policy:

• Avoiders exit or skip entries when an earnings date falls inside the expected holding window — the common choice for mean-reversion and breakout swing systems. Requires reliable forward-looking earnings dates live, and point-in-time historical ones in the backtest (§04, §11).
• Harvesters trade the event deliberately (drift, reaction) — a different strategy family with §11's event-driven burden of proof.
• What's not acceptable is accidental exposure: a backtest that never knew which bars were earnings bars is averaging two regimes and will surprise you live.

Universe construction: where swing backtests are won

A daily/swing system is defined as much by what it's allowed to trade as by its entry rule. The eligibility screen, evaluated point-in-time each day (§04): primary-exchange listed common stock (exclude OTC; decide policy on ADRs and ETFs explicitly), unadjusted price ≥ $5, median dollar volume ≥ $10–20M, not within N days of a known earnings date (if avoiding), optionally ≥ 1 year since IPO (young listings behave differently and stress data quality). Then verify the edge isn't a liquidity artifact: re-run with the price/volume floors doubled — real behavioral edges usually survive with modest degradation; data-artifact "edges" concentrate in exactly the names the tighter screen removes (§03.6).

Signals-to-orders bookkeeping

Small mechanical decisions that change results and must be frozen in config: ranking ties (two names, one slot — deterministic tie-break, seeded); simultaneous entry and exit signals (netting order); insufficient cash for all signals (priority rule: signal strength, liquidity, or diversification); partial fills from participation caps (carry the residual order or cancel); and re-entry policy after stops (cooldown or immediate). None of these is glamorous; all of them are the difference between a backtest you can reproduce and one you can't.

Realistic expectations (calibrate your suspicion)

Corresponding drawdown reality: a Sharpe-1 strategy should expect drawdowns around 15–25% over a decade of trading (and the Monte Carlo tail worse, §09). If a backtest shows Sharpe 1 with a 4% max drawdown, the window is short, the fills are fantasy, or the tail hasn't happened yet.

The swing trader's daily loop, automated Design the live system as the same five phases as the engine (§06): after close — ingest data, run validation checks (§04), compute signals; evening — construct orders, human-review if desired (a read-only checkpoint: you may halt on operational anomalies, but overriding signals invalidates the backtest's claim to describe your trading); pre-open — submit; during day — monitor fills and stops; after close — reconcile fills vs expectations into the §16 dashboards. If any step can't be automated and verified, it will eventually be skipped — on exactly the volatile day it mattered.

13 Machine Learning & the Advanced Toolkit

ML doesn't repeal any rule in this guide — it amplifies the penalties for breaking them, because flexible models exploit leaks that rigid rules can't even see. This section is the ML-specific leakage catalog, plus the modern labeling-and-filtering stack (López de Prado's toolkit) that makes ML tractable on financial data.

The ML leakage catalog

Each of these has produced a thousand beautiful validation curves and zero live profits:

• Shuffled splits. KFold(shuffle=True) or train_test_split on time-series data — the shuffle trap of §05, now with a model powerful enough to fully cash it in. Chronological, purged splits only.
• Preprocessing fit on the full sample. StandardScaler.fit(X) before splitting hands every training row the test period's mean and variance — a regime summary of the future. Fit scalers, PCA, encoders, everything, inside each training fold only.
• Overlapping labels. Predicting 5-day returns sampled daily means adjacent labels share 4 days of outcome. Random or unpurged splits then put near-duplicates on both sides of the boundary — the model "generalizes" to data it effectively saw. Purge and embargo (§09), and weight overlapping samples down (or sample non-overlapping events).
• Feature selection outside the CV loop. Choosing "the 12 best features" on all data, then cross-validating a model on those 12 — the selection already consumed the test set. Selection goes inside the fold (nested CV), or it's a leak.
• Hyperparameter search amnesia. Every configuration your tuner evaluated is a trial in §09's sense. Log the search budget to the registry; a "great" model found by a 2,000-point search is judged against the max-of-2,000 noise ceiling.
• Retraining fantasy. Backtesting one model fit on 2005–2020, when live you'll refit monthly — you tested a strategy you won't run. The walk-forward loop (§08) must wrap the entire pipeline: features, selection, tuning, fitting, prediction.
• Point-in-time violations in features. Restated fundamentals, current sector classifications applied historically, adjusted price levels — §04's rules apply to every feature, and feature stores silently violate them.
• Threshold tuned on the test set. Choosing the probability cutoff that maximizes test-set P&L is fitting to the test set, one parameter at a time.

Label engineering: the triple-barrier method

Predicting raw next-bar returns is fighting maximal noise for minimal structure. The triple-barrier method re-poses the question the way a trade actually resolves: from entry, which happens first — the profit target (+1), the stop (−1), or the time limit (0/sign of drift)? Barriers scale with prevailing volatility, so the label means the same thing in calm 2017 and violent 2022 — a fixed-percent barrier would stop on noise in one regime and never trigger in the other:

triple_barrier.py — volatility-scaled outcome labels (daily bars)

import numpy as np
import pandas as pd

def triple_barrier_labels(close: pd.Series, events: pd.DatetimeIndex,
                          pt_mult=2.0, sl_mult=1.0, max_hold=10,
                          vol_span=63):
    """Label each event date: +1 target-first, -1 stop-first, 0 timeout."""
    vol = close.pct_change().ewm(span=vol_span).std()   # daily sigma
    out = {}
    for t0 in events:
        sigma = vol.loc[:t0].iloc[-1]
        if not np.isfinite(sigma) or sigma <= 0:
            continue
        path = close.loc[t0:].iloc[1 : max_hold + 1]    # AFTER entry bar
        if path.empty:
            continue
        ret = path / close.loc[t0] - 1
        hit_pt = ret[ret >=  pt_mult * sigma].index.min()
        hit_sl = ret[ret <= -sl_mult * sigma].index.min()
        first  = min([x for x in (hit_pt, hit_sl) if pd.notna(x)],
                     default=pd.NaT)
        if pd.isna(first):
            out[t0] = 0                                  # vertical barrier
        else:                       # tie (both same bar) -> pessimistic -1
            out[t0] = -1 if first == hit_sl else 1
    return pd.Series(out, name="label")
    # Note: labels span time -> overlapping-label rules apply (§09).

Meta-labeling: separate the when from the how much

Instead of asking ML to find trades from scratch (hard, opaque, leak-prone), keep a simple, interpretable primary model that proposes direction — your momentum rule, your pullback trigger, a structural-break filter — and train a secondary classifier on one narrow question: given this setup and current conditions (volatility, spread, regime features, signal strength), is the primary signal likely to be right? Trade only when the meta-model concurs; optionally size by its confidence.

Why this architecture wins in practice: it attacks false positives — the primary rule keeps its recall while the filter lifts precision (and with it F1 and expectancy); risk control adapts without retraining the core logic; the system stays auditable (the primary rule is explainable; the ML is confined to a gatekeeper role); and the label for training it is beautifully natural — the triple-barrier outcome of the primary model's own historical signals. Every §09 rule still applies (purged CV over the meta-model's training, search budgets logged), but the surface area for catastrophe is far smaller than end-to-end ML.

Information-driven bars & fractional differentiation (know they exist)

• Volume/dollar/tick bars: time bars oversample dead hours and undersample violent ones, producing heteroskedastic, fat-tailed series that ML digests poorly. Bars that close on constant traded volume (or dollars) sample by information flow instead, yielding better-behaved returns. This matters when you go intraday or feed ML raw price structure; for EOD swing systems, daily bars are the market's own information rhythm and this is background knowledge.
• Fractional differentiation: prices are non-stationary; returns are stationary but memoryless. Fractional differencing (d ≈ 0.3–0.6) is the compromise — a series stationary enough to model that still carries long-range memory. Relevant when feature engineering for ML on levels; skip it for rule-based systems.

The coherent advanced stack (and when to bother) Assembled, the modern pipeline reads: sample events (e.g., CUSUM filter on prices) → engineer point-in-time features (regime-relative where possible, §05) → label with triple barriers → train a meta-labeler over a simple primary rule → validate with purged/embargoed walk-forward or CPCV → judge with DSR against the full search budget → deploy with drift monitors (§16). Adopt it in that order of value: triple-barrier labels and meta-labeling pay off early for a swing trader with a working rule-based system; bars and fractional differentiation pay off when you go intraday or model-heavy. And the eternal baseline discipline: an ML system must beat your best simple system out-of-sample, after its (much larger) search budget is penalized — most don't, and discovering that cheaply is the point of everything in §08–§09.

14 Tooling Landscape (2026)

The honest state of the Python ecosystem as of mid-2026 — what each tool is actually for, and the build-vs-buy decision for someone in your position. Verify maintenance status and pricing before committing; this landscape moves.

Backtesting engines

The build-vs-buy decision (for you, specifically)

For daily-bar equity systems, a custom engine is a small, well-bounded project — §06's skeleton plus §04's data layer plus §07–§09's reports is a few thousand lines of Python — and it's the path this guide's §17 spec assumes, for three reasons: every honesty rule (§03, §06) is enforced by your code rather than hoped about a framework's internals; the trial registry, evidence pack, and statistical gauntlet integrate natively instead of being bolted on; and building it is the single best education in backtesting mechanics available. The pragmatic hybrid most professionals converge on: vectorbt (or plain pandas) for idea screening → your own event-driven engine for validation → NautilusTrader or broker-API deployment when live parity starts to matter. Choose a framework instead if your time is the binding constraint and your strategies fit its assumptions — LEAN if you want data+engine+brokerage integrated, backtesting.py if single-name simplicity covers your designs.

Data (recap of §04's table)

Norgate or Sharadar for survivorship-free US EOD + point-in-time universes (the two names that come up again and again for individual systematic traders); EODHD for global breadth; Polygon/Databento when you need intraday; free sources for mechanics practice only.

Execution & paper trading

Tool-choice principle Tools implement honesty; they don't supply it. Every engine above will happily backtest a survivorship-biased universe with same-bar fills if you feed it one. Whatever you adopt, run your §06 tester-tests against it — buy-and-hold reconciliation, zero-edge null, split/dividend fixtures — before believing a number it prints. Frameworks earn trust the same way strategies do.

15 The Master Do's & Don'ts

The whole guide, compressed into checklists. Run them at three moments: designing the system, before trusting any result, and before committing capital.

Data

• ✓ Use survivorship-bias-free data with delisted securities and delisting outcomes; define universes point-in-time. (§03.2, §04)
• ✓ Carry adjusted and unadjusted prices; signals/returns on adjusted, levels/quantities/costs on unadjusted. (§04)
• ✓ Give every non-price record an "effective/knowable" timestamp and join everything as-of that time. (§03.11, §04)
• ✓ Run automated validation on every ingest: OHLC consistency, calendar gaps, unexplained >40% moves, stale series, duplicates. (§04)
• ✓ Store immutable raw snapshots and record a data-version hash with every run. (§04, §06)
• ✗ Don't validate strategies on free/survivor-only data — mechanics practice only. (§04)
• ✗ Don't key anything by ticker symbol across time; use permanent security IDs. (§03.2)
• ✗ Don't hand-patch bad prices; quarantine the series and log the exclusion. (§04)

Engine & execution modeling

• ✓ Enforce the timing contract: decisions from data ≤ close t, fills at open t+1 (or an explicitly justified variant). (§06)
• ✓ Expose data to strategies only through an as-of API — make look-ahead a compile-time impossibility, not a discipline. (§06)
• ✓ Fill pessimistically: limits only on trade-through, stops gap to the open, adverse intrabar ordering, participation caps on ADV. (§03.7, §06)
• ✓ Model costs per fill — spread + volatility-scaled slippage + impact when sized — and make the 0×/1×/2×/3× sweep one flag. (§03.5, §06)
• ✓ Account for everything: dividends on pay dates, splits, borrow fees, margin interest, T-bill yield on idle cash; assert cash + positions = equity every bar. (§06)
• ✓ Make runs deterministic and self-describing: config hash, data version, git commit, seed — the run manifest. (§06)
• ✓ Test the tester: buy-and-hold reconciliation, zero-edge null, known-answer fixture, split/dividend fixtures, look-ahead canary. (§06)
• ✓ Implement reset_state() and use it at every fold boundary. (§05, §06)
• ✗ Don't fill limit orders on a touch — that P&L is adversely selected fiction. (§03.7)
• ✗ Don't let a backtest fill stops at the stop price through a gap. (§03.7)
• ✗ Don't trust a vectorized result that the event-driven engine hasn't reproduced within costs. (§06, §08)
• ✗ Don't let any simulation trade through halts, or hold delisted names that silently vanish. (§06)

Research process

• ✓ Write the hypothesis — mechanism, predicted shape, falsifier — before touching data. (§08)
• ✓ Freeze IS / validation / lockbox splits before looking; budget and log every validation visit; touch the lockbox once. (§08)
• ✓ Log every run — including failures and abandoned branches — to an append-only trial registry. (§06, §08)
• ✓ Demand parameter plateaus; deploy from the center; apply the cliff veto to any candidate whose grid-neighbors collapse. (§08)
• ✓ Walk-forward with purge gaps, state resets, and locked parameters; report only the stitched OOS curve; track the resilience ratio (OOS/IS ≥ ~0.5). (§08)
• ✓ Test across regimes with fixed parameters; include 2008-shaped, 2020-shaped and 2022-shaped windows; report per-year tables. (§03.9, §05)
• ✓ Pre-commit acceptance gates (majority-pass + catastrophic veto) in a dated file before the lockbox run. (§08)
• ✓ Assemble the evidence pack: hypothesis, manifests, sensitivity maps, fold tables, MC distributions, trial count, gate results. (§08)
• ✗ Don't iterate against the backtest — tweak-run-look loops turn the test set into training data. (§02, §08)
• ✗ Don't re-run the lockbox after a failure with a "small fix" — the idea goes back to development. (§08)
• ✗ Don't compare strategies across different execution conventions or cost models. (§12)
• ✗ Don't keep the winner of {rolling, anchored} × {5 windows} × {3 conventions} and call it one trial. (§08, §09)

Statistics

• ✓ Judge the best result against the expected max of N noise trials; compute DSR with the honest registry count (raw and clustered). (§09)
• ✓ Demand t ≈ Sharpe·√years ≥ ~3 for mined strategies, or the FDR-adjusted equivalent. (§09)
• ✓ Purge and embargo every train/test boundary; use CPCV + PBO when ranking many candidates or using ML. (§09)
• ✓ Bootstrap (stationary blocks) for Sharpe CIs and the drawdown distribution; size to the bad tail, not the historical path. (§07, §09)
• ✓ Run the fragility suite: trade reshuffle, skip-trade, entry jitter, noise injection, random-twin percentile. (§09)
• ✓ Read Sharpe with skew/kurtosis and worst-week; suspect any smooth curve with negative-skew mechanics. (§07)
• ✗ Don't shuffle time-series data, ever — no random k-fold, no shuffled train/test splits. (§05)
• ✗ Don't report a Sharpe without its trial count and sample length — it's a p-value without degrees of freedom. (§09)
• ✗ Don't treat 60 trades as evidence; statistics on small trade counts are weather. (§07)
• ✗ Don't tune anything — even a probability threshold — on the data that judges it. (§13)

Sizing & deployment

• ✓ Backtest the sizing you'll trade; report fixed-notional and compounded views. (§10)
• ✓ Cap participation, cap position weights, cap correlated-theme exposure; stress the "all my signals are one signal" day. (§10)
• ✓ Estimate capacity before scaling; respect turnover's inverse relationship with it. (§10)
• ✓ Incubate 3–6 months of paper trading on live data; reconcile fills, slippage and signal timing against the backtest's assumptions. (§16)
• ✓ Go live at fraction size with pre-written kill criteria (drawdown, rolling-Sharpe floor, structural-break alarms) and a monitoring dashboard. (§16)
• ✓ Track implementation shortfall daily: live fills vs simulated fills on identical signals. (§16)
• ✗ Don't size anywhere near full Kelly on backtested estimates; quarter-Kelly is aggressive. (§10)
• ✗ Don't deploy a strategy whose historical drawdown you couldn't survive financially and psychologically at 1.5× depth and duration. (§03.12, §07)
• ✗ Don't override live signals ad hoc; if you must intervene, halt, log, and re-validate — an overridden system's backtest describes nothing. (§12, §16)
• ✗ Don't let a live system run without drift monitors and an automated flat-switch reachable from your phone. (§16)

16 From Backtest to Live — Closing the Loop

Deployment is not the end of validation; it's the start of the only test that counts. The design goal: make live trading a continuous, instrumented comparison against what the backtest promised.

Why live underperforms even honest backtests

Budget for degradation from four sources that no simulation fully captures: selection residue (even after §09's deflation, the strategy that reached deployment was the survivor of a search); alpha decay (edges are harvested by others and arbitraged thinner — the market adapts to its describers); execution reality (your fills, outages and latencies vs the model's); and regime novelty (the future contains environments your data didn't, §05). Hence the planning heuristic from §02: haircut the backtested Sharpe by ~50% and confirm the strategy still clears your bar. If it only works at full backtested strength, it doesn't work.

Incubation: what paper trading actually tests

Three to six months of paper trading a swing system yields maybe 30–100 trades — not enough to statistically confirm the edge (§07's sample-size arithmetic cuts both ways). What incubation genuinely tests, and what to measure:

• The pipeline: does data arrive on time, pass §04 checks, produce signals, generate orders — every day, unattended, including half-days and volatile opens? Count operational incidents; zero for a full month is the bar for real money.
• The cost model: compare simulated fills to paper fills (and later, real fills) on identical signals. This calibrates §06's slippage parameters with data — the single most valuable output of incubation.
• Distributional consistency: are live trade outcomes plausible draws from the backtest's per-trade distribution (win rate, average win/loss, holding periods)? A formal two-sample test is weak at these sample sizes, but gross mismatches — live win rate 38% vs backtested 58% — surface fast and demand explanation before scaling.
• You: did you follow it? Every manual intervention gets logged and post-mortemed; incubation is where the §03.12 conversation with yourself becomes empirical.

The live monitoring stack

Instrument the system from day one — retrofitting monitoring after a drawdown is how post-mortems get written. Four layers:

Kill criteria and the retirement decision

Written before launch, versioned with the strategy, executed without renegotiation:

• Hard stops (automatic flat): drawdown beyond 1.25–1.5× the Monte Carlo 5th-percentile max DD; a single-day loss beyond the modeled worst-case; broker/position reconciliation failure; data integrity failure at decision time.
• Soft reviews (de-risk and investigate): rolling 6-month Sharpe below the backtest's 10th-percentile band; drift detector firing; implementation shortfall trending; live expectancy below half of backtested for N trades.
• Retirement: edges die of crowding and regime change more often than of drama. A strategy that spends a year inside its "soft review" zone with a decaying resilience trend (§08) is telling you something; retiring it while modestly profitable is a win, not a failure. Pre-commit the review cadence — quarterly against the evidence pack — so the decision is procedural.

Pitfall — silent strategy mutation The live system will tempt you into "small" changes: a tweaked filter after a bad week, a manually skipped signal, a stop moved once. Each unversioned change detaches the running system from the evidence pack that justified it — after three such tweaks you are trading an unvalidated strategy that merely resembles a validated one. Rule: any change = new strategy version → registry entry, abbreviated re-validation (the §08 pipeline, fast-tracked), updated gates. The discipline sounds bureaucratic and takes an afternoon; the alternative is not knowing what you're running.

Scheduled re-estimation vs redesign

Distinguish the two maintenance modes explicitly (§05's passive vs active adaptation): re-estimation — refitting parameters on the walk-forward cadence you simulated — is part of the strategy and needs no new approval; redesign — new rules, features, universes — is a new strategy and re-enters the pipeline at Step 0. The trap is redesign disguised as re-estimation ("just widening the stop range this quarter"). If the change alters what the walk-forward would have done historically, it's redesign.

The loop, closed Live results feed back as research data: realized slippage recalibrates the cost model; live trades extend the OOS record and sharpen the DSR; drift alarms update the regime map; post-mortems seed the next hypothesis page. A year of disciplined live trading — win or lose — leaves you with something no backtest can produce: a calibrated research process, whose promises you can now price. That compounding of process quality, more than any single strategy, is the actual asset you're building.

17 Prompting Your System — A Ready-to-Use Specification

This section converts the entire guide into a build order. Below is a complete specification you can paste into an AI coding agent (Claude Code or similar) — or hand to a developer — to build a backtesting system that enforces everything above by construction.

How to run the build (read before pasting)

• Build in phases, in order. The spec defines seven phases, each with acceptance tests. Don't let the agent (or yourself) advance a phase until the previous phase's tests pass — the later phases assume the earlier honesty guarantees.
• Tests first, and you review two things personally: the timing contract (where can the strategy see data from?) and the fill logic (what fills, at what price?). These two files are where an entire system quietly becomes a liar; read every line of them yourself, against §03.1 and §03.7.
• Feed the agent context. Alongside the spec, paste the relevant sections of this guide when working on a component (§06 for the engine phase, §09 for the statistics phase). The spec says what; the guide says why, and agents write better code when they know why.
• Keep the spec as the contract. When you change requirements mid-build, edit the spec first, then implement — the same discipline §16 demands of strategies applies to the system itself.
• Start with synthetic data. Phases 1–5 run entirely on generated fixtures; you can build and verify the whole machine before spending anything on real data (§04's vendors slot in at Phase 6).

SPEC — paste this into your AI coding agent

# PROJECT: Honest event-driven backtesting system for daily US-equity
# swing strategies. Python 3.12+. You are building research infrastructure
# whose PRIMARY design goal is that dishonest results are structurally
# impossible — correctness and auditability outrank speed and features.

## 0. TECH BASELINE
- Python 3.12+, pandas (or polars) + numpy + scipy; pydantic for config;
  pyarrow/parquet for storage; pytest (+ hypothesis for property tests);
  matplotlib for reports. No other hard dependencies.
- Everything deterministic: same config + data + seed => bit-identical
  outputs. All randomness flows from a single seeded RNG in config.
- Type hints throughout; dataclasses/pydantic models for all records.

## 1. ARCHITECTURE (strict module boundaries)
- data/     DataStore: immutable raw parquet layer (unadjusted OHLCV,
            adjustment factors, dividends, splits, delistings, universe
            membership, earnings dates, T-bill rates), snapshot-versioned
            with a content hash. Derived layer rebuilt deterministically.
- feed/     AsOfFeed: THE ONLY data access path for strategies.
            API: history(sid, field, n, asof), open/close/adv/vol(sid, t),
            calendar(start, end). It is impossible to request data with
            timestamp > asof; attempting it raises LookaheadError.
- universe/ PIT eligibility: members(t) from membership data + filters
            (min unadjusted price, min median dollar volume, listing
            venue, optional earnings-window exclusion) evaluated as-of t.
- strategy/ Strategy interface: target_weights(view, eligible) -> dict;
            reset_state() clears ALL internal state. Strategies never see
            fills, cash, or anything post-asof.
- portfolio/ Sizer: targets -> orders under constraints (max weight,
            max positions, gross/net caps, ADV participation cap, whole
            shares, cash buffer). Deterministic tie-breaking, seeded.
- exec/     ExecutionSimulator (THE PESSIMIST):
            * market orders fill at next open * (1 + slippage), slippage
              sign always adverse;
            * limit orders fill ONLY on trade-through beyond an epsilon,
              never on touch;
            * stop orders: long stop fills at min(stop, open) when gapped;
            * intrabar stop+target both touched => stop first (adverse);
            * participation cap: fill min(order, cap*ADV); residual policy
              configurable (carry N days | cancel);
            * no fills on halted days; delistings convert positions via
              the delisting record (proceeds or write-off), never vanish.
- costs/    CostModel: commission/fees + half-spread + vol-scaled
            slippage + optional square-root impact Y*sigma*sqrt(Q/ADV).
            All parameters in config; global cost multiplier flag for
            0x/1x/2x/3x sweeps.
- account/  Accountant: cash ledger; dividends credited on pay date;
            splits adjust share counts; short borrow fees accrue daily
            (punitive default when no borrow data); margin interest;
            T-bill yield on idle cash. INVARIANT asserted every bar:
            equity == cash + sum(shares * unadjusted close).
- engine/   Event loop with the frozen timing contract:
            decide(close t) -> order(overnight) -> fill(open t+1) ->
            manage stops(bar t+1) -> settle+mark(close t+1).
            No component may read forward of its phase.
- registry/ Append-only JSONL trial registry. EVERY run auto-logs a
            manifest: run_id, UTC time, config hash, data snapshot hash,
            git commit, seed, full config, headline metrics. Includes
            aborted/failed runs. There is no code path that runs a
            backtest without logging it.
- report/   Metrics + report generation (see §4 below).
- research/ Walk-forward runner, Monte Carlo suite, statistics module,
            sensitivity/grid runner, evidence-pack builder (see §5-§7).
- cli/      Commands: run, sweep, walkforward, montecarlo, stats,
            report, evidencepack — all config-file driven (YAML).

## 2. CONFIG (single YAML, pydantic-validated; reject unknown keys)
strategy name+params; universe filters; execution convention
(next_open | next_close | moc_preclose); cost model params + multiplier;
sizing (method, vol targets, caps, participation); dates (IS/validation/
lockbox boundaries stored EXPLICITLY); seed; data snapshot id.

## 3. LEAKAGE DEFENSES (hard requirements)
- AsOfFeed raises on any future access (unit-tested).
- All rolling statistics computed with trailing windows only; a lint
  test greps strategy code for forbidden calls (full-sample mean/std,
  center=True windows, fillna(method="bfill"), shift(-n)).
- Signals computed on adjusted series; eligibility, share quantities and
  costs on unadjusted. Both series carried; mixing is a typed error.
- Purge-gap and state-reset support built into every fold boundary.

## 4. METRICS & REPORTS (one fixed report card, §07 of the guide)
Full-period: CAGR, vol, Sharpe (vs T-bill, from daily), Sortino, Calmar,
max DD + underwater duration, skew, kurtosis, worst day/week, exposure,
turnover. Trade-level: n, win rate, payoff, profit factor, expectancy
bps vs cost bps, holding periods, top-5-trade concentration. Views:
per-calendar-year table, rolling 12m Sharpe, underwater curve, monthly
heatmap. Benchmarks: total-return index, vol-matched index, alpha/beta
regression, random-twin percentile (exposure-matched random entries,
1000 sims). Cost sweep table 0x/1x/2x/3x. Output: HTML + JSON per run.

## 5. WALK-FORWARD RUNNER
Rolling or anchored; train/test lengths in config; purge gap between
train and test; strategy.reset_state() at each fold start; parameters
locked per fold with lock timestamps logged; outputs stitched OOS curve
+ per-fold table (IS Sharpe, OOS Sharpe, resilience ratio OOS/IS,
params). Grid/sensitivity runner produces plateau heatmaps and a cliff
report: for each candidate cell, worst one-step-neighbor Sharpe drop
and DD increase.

## 6. MONTE CARLO SUITE (each: config-driven n_sims, seeded, CI outputs)
a) stationary bootstrap of daily returns (geometric mean block length)
   -> Sharpe CI, max-DD distribution;
b) trade-order reshuffle -> DD distribution;
c) skip-trade (drop 10-20% of trades) -> profit concentration;
d) entry jitter (+/-1 day) -> timing-luck sensitivity;
e) price-noise injection (fraction of ATR, full re-run) -> brittleness.

## 7. STATISTICS MODULE
- PSR(sr_hat, sr_star, T, skew, kurt); expected max SR of N trials
  (Euler-Mascheroni form); DSR wired to the registry: reads ALL trials
  for a project tag, reports DSR with raw N and clustered effective N
  (cluster trial-return correlation matrix);
- minimum backtest length given N and target SR;
- optional CPCV (S blocks, all C(S, k) combos, purged) + PBO;
- t-stat = SR*sqrt(years) reported everywhere Sharpe is.

## 8. EVIDENCE PACK
One command assembles per-strategy: hypothesis file (markdown, authored
by the user), split design, all run manifests, sensitivity + cliff
report, walk-forward tables, MC distributions, regime/per-year tables,
trial counts + DSR, the pre-committed gates file, and gate evaluation
(majority-pass across folds + catastrophic veto). Output: a single HTML
dossier + JSON. Gates file is versioned; the pack records its hash and
commit date so "gates written before lockbox ran" is provable.

## 9. ACCEPTANCE TESTS (definition of done — write these FIRST)
1. Buy-and-hold reconciliation: 1 synthetic stock with dividends+split,
   zero costs => engine equity equals hand-computed total-return curve
   to 1e-9.
2. Zero-edge null: random signals, costs on, 100 seeds => mean return
   ~= -(cost drag), |Sharpe| small; distribution reported.
3. Known-answer fixture: sawtooth price + threshold rule => exact P&L
   matching a hand-computed table committed to the repo.
4. Look-ahead canary: a malicious strategy attempting future access
   fails with LookaheadError (and the lint test passes on examples).
5. Split/dividend fixtures: 2:1 split mid-hold => continuous equity,
   doubled shares; dividend => cash on pay date, not before.
6. Gap-stop test: stop above next open => fill at open, not stop.
7. Limit-touch test: bar low == limit => NO fill; trade-through => fill.
8. Delisting test: position in a delisting stock => converts via
   delisting record; portfolio never silently drops it.
9. Determinism: identical config+data+seed => byte-identical outputs.
10. Accounting identity asserted every bar in every test above.
11. Walk-forward: fold boundaries respect purge gap; state reset proven
    (a stateful dummy strategy's state does not leak across folds).
12. Registry: any CLI run appends exactly one manifest; a killed run
    still logs.

## 10. BUILD PHASES (each ends green on its tests before the next)
P1 data layer + calendar + synthetic fixture generator;
P2 AsOfFeed + leakage tests; P3 engine + execution + accounting
(acceptance tests 1-10); P4 metrics + report; P5 walk-forward +
grid + Monte Carlo + statistics (test 11) + registry (test 12);
P6 real-data adapters (Norgate/Sharadar/EODHD importers) + §04-style
data validation checks on ingest; P7 evidence pack + gates + polish.

## 11. NON-GOALS (refuse scope creep)
No intraday/tick simulation, no options/futures/FX, no live trading in
v1 (but Strategy/Sizer interfaces must be reusable by a future live
adapter unchanged), no GUI, no database server, no distributed compute.

## 12. STYLE OF COLLABORATION
Propose file layout first. Write tests before implementations. After
each phase: summarize what is guaranteed and what is NOT yet guaranteed.
Flag every place you made a modeling choice the spec left open, in a
DECISIONS.md. Never "improve" realism by making fills more generous.

After the build: the first month of use

Run the machine before trusting it with ideas you care about: (1) generate synthetic universes and confirm the acceptance suite stays green; (2) load real data and run §04's validation on ingest — expect it to find vendor problems, that's it working; (3) backtest a deliberately naive strategy (e.g., same-bar moving-average cross) and confirm the system's pessimism reduces it to roughly nothing — your engine should be hard to impress; (4) reproduce a known effect (12-month momentum, monthly rebalance) and check its shape against the literature — right sign, modest magnitude, 2009-shaped crash and all; (5) only then, bring your first hypothesis page and enter §08's pipeline at Step 0.

A companion file for this spec This specification is also provided as a standalone Markdown file alongside this guide, so you can paste it into an agent without extracting it from HTML.

18 Glossary

ADV (average daily volume) — Typical shares or dollars traded per day; the denominator of participation caps and impact models.

ADWIN — Adaptive Windowing — an online change-point detector that shrinks its window when old and new data disagree beyond a Hoeffding-bound threshold; used for drift detection and regime-aligned validation (§05).

Alpha / beta — From regressing strategy returns on a benchmark: beta is the market-exposure component (cheap to rent); alpha is the residual return your process added.

Anchored vs rolling window — Walk-forward training on all history from a fixed start (anchored) vs a fixed-length trailing window (rolling).

As-of join — Joining datasets on "the latest record knowable at time t" rather than exact timestamps — the mechanical defense against look-ahead in slow data (§04).

ATSCV — Adaptive time-series cross-validation: fold boundaries placed at statistically detected regime breaks instead of calendar intervals (§05).

Calmar / MAR ratio — CAGR divided by maximum drawdown; return per unit of worst pain.

Capacity — The AUM a strategy can run before impact costs consume its edge; hierarchy from implementation to terminal capacity (§10).

Concept drift / covariate shift — Two modes of market change: the input-output relationship P(Y|X) itself changing (concept drift) vs only the input distribution P(X) moving (covariate shift) (§05).

CPCV — Combinatorial Purged Cross-Validation — all combinations of sequential blocks as test sets, purged at boundaries, yielding a distribution of out-of-sample paths (§09).

Delisting return — What a holder actually received when a security left the market (merger proceeds, bankruptcy residual); omitting it is a core survivorship error (§03.2).

DSR (Deflated Sharpe Ratio) — Probability the observed best Sharpe exceeds the expected maximum Sharpe of N zero-skill trials, adjusted for short samples, skew and kurtosis (§09).

Embargo — Extra buffer of training data dropped after a test block to absorb market memory beyond label overlap (§09).

Equity curve — Portfolio value over time; the object every bias in §03 is trying to flatter.

Expectancy — Average P&L per trade (in currency or bps of notional); compared against per-trade cost in the break-even test (§07).

FWER / FDR — Multiple-testing error philosophies: probability of any false discovery (family-wise) vs expected proportion of false discoveries (§09).

Implementation shortfall — Live execution price vs the simulator's assumed price on identical signals; the live calibration signal for cost models (§16).

Kelly criterion — Growth-optimal bet size given known edge and variance; catastrophically sensitive to edge overestimation — used fractionally if at all (§10).

Lockbox — Final holdout data touched exactly once, under pre-committed gates, as the strategy's verdict (§08).

Look-ahead bias — Any use of information not knowable at decision time (§03.1).

Market impact — Price movement caused by your own trading; scales ≈ with the square root of size relative to ADV (§10).

Meta-labeling — A second-stage ML model that predicts whether a primary signal will succeed, gating/sizing trades rather than generating them (§13).

PBO — Probability of Backtest Overfitting — how often the in-sample best choice underperforms the median out-of-sample across combinatorial splits (§09).

Point-in-time (PIT) — Data exactly as it was knowable on each historical date — memberships, fundamentals, universes (§04).

PSR (Probabilistic Sharpe Ratio) — Probability the true Sharpe exceeds a benchmark given sampling error, skew and kurtosis (§09).

Purging — Removing training samples whose label windows overlap the test period (§09).

Regime — A persistent market environment (trend/chop, high/low vol, easing/tightening) within which strategy behavior is roughly stable (§05).

Resilience ratio / walk-forward efficiency — OOS performance ÷ IS performance per fold; ≥0.6 healthy, <0.5 red flag (§08).

Sharpe ratio — Annualized excess return ÷ annualized volatility; the industry's common currency and §07's most-abused number.

Slippage — Difference between decision price and fill price, excluding explicit fees (§03.5).

Stationary bootstrap — Block resampling with geometric random block lengths; preserves short-range dependence when generating alternate histories (§09).

Survivorship bias — Testing only on securities that still exist today (§03.2).

Total-return series — Price series with dividends reinvested; required for honest performance and benchmark comparisons (§04).

Trial registry — Append-only log of every backtest run ever executed in a project; supplies N for DSR (§06, §09).

Triple-barrier label — Trade outcome classified by which of profit-target / stop / time-limit was hit first, with volatility-scaled barriers (§13).

Turnover — How many times the portfolio's value trades per year; multiplied by per-trade cost, it's the annual friction bill (§07).

Walk-forward analysis — Sequential refit-then-trade-forward simulation matching how the system will actually be re-estimated live (§08).

19 Further Reading

The canon behind this guide, in a sensible reading order for your profile.

Start here (practical foundations)

• Robert Pardo — *The Evaluation and Optimization of Trading Strategies*. The original walk-forward analysis text; the source of walk-forward efficiency and much of §08's discipline.
• Ernest Chan — Quantitative Trading and *Algorithmic Trading*. Grounded, honest retail-to-small-fund perspective; strong on practical pitfalls, mean reversion vs momentum mechanics, and Kelly caution.
• Robert Carver — *Systematic Trading (and Advanced Futures Trading Strategies*). The best treatment of sizing, vol targeting, and designing systems you'll actually follow; deeply skeptical of overfitting in exactly §08's spirit.
• David Aronson — *Evidence-Based Technical Analysis*. A book-length assault on data snooping in technical rules; the philosophical companion to §03.3 and §09.

The statistical core

• Marcos López de Prado — *Advances in Financial Machine Learning*. The source for purged/embargoed CV, CPCV, triple-barrier labels, meta-labeling, information bars, fractional differentiation, and backtesting's procedural laws (§05, §09, §13).
• Bailey & López de Prado — "The Deflated Sharpe Ratio" (2014). The DSR paper behind §09; short and readable.
• Bailey, Borwein, López de Prado & Zhu — "The Probability of Backtest Overfitting" (2016) and "Pseudo-Mathematics and Financial Charlatanism" (2014). PBO/CSCV and minimum backtest length; the arithmetic of §09's noise ceiling.
• Harvey & Liu — "Backtesting" (2015) and "…and the Cross-Section of Expected Returns" (2016). Haircut Sharpe ratios, FWER/FDR frameworks, and the t ≥ 3 argument from the factor-zoo replication crisis.
• Lo — "The Statistics of Sharpe Ratios" (2002). What a Sharpe estimate's error bars really are, including autocorrelation effects (§07).
• Politis & Romano — "The Stationary Bootstrap" (1994). The block-resampling method behind §09's Monte Carlo.
• Bifet & Gavaldà — "Learning from Time-Changing Data with Adaptive Windowing" (2007). The ADWIN paper behind §05's drift detection.

Microstructure, costs and capacity

• Almgren & Chriss — "Optimal Execution of Portfolio Transactions" (2000) and Almgren et al. — "Direct Estimation of Equity Market Impact" (2005). The impact-vs-speed tradeoff and empirical impact curves behind §10.
• Obizhaeva & Wang — "Optimal Trading Strategy and Supply/Demand Dynamics" (2013). Temporary-impact shock-and-decay modeling (§10).
• Bouchaud et al. — Trades, Quotes and Prices (2018). The square-root law and order-flow microstructure, in depth.
• Vangelisti — "The Capacity of an Equity Strategy" (2006) and Kahn & Shaffer — "The Surprisingly Small Impact of Asset Growth on Expected Alpha" (2005). The capacity-definition hierarchy and turnover-capacity mathematics (§10).
• Platen — "Backtest of Trading Systems on Candle Charts" (IFTA Journal, 2016), Stasiak — "Candlestick — The Main Mistake of Economy Research in High Frequency Markets" (2020), and Jäkärä — From Candles to Ticks (Aalto University, 2023). The case against bar data for intrabar execution: OHLC reconstruction is formally undecidable, and the empirical live-vs-backtest comparison behind §03.7's evidence note.

Broader context

• Stefan Jansen — *Machine Learning for Algorithmic Trading*. Encyclopedic, code-heavy companion (its author maintains zipline-reloaded, §14).
• Bailey & López de Prado's online backtest-overfitting demonstrations. Interactive tools that let you feel §09's arithmetic by mining random data yourself.
• Andrew Lo — Adaptive Markets (2017). The ecological/evolutionary frame for why edges appear, get crowded, and decay — the intellectual backdrop of §05 and §16.

Scope and disclaimer. This guide is educational material about research methodology, compiled August 2026. It is not investment, legal or tax advice; markets involve risk of loss; verify vendor, broker, regulatory and tax specifics for your situation independently. Backtested performance — however honestly produced — does not guarantee future results, which is, in a sense, the entire point of this document.