Backtesting and Curve Fitting Explained
Backtesting means running a set of trading rules over historical market data to see what would have happened. It is a standard step in building and checking an automated strategy, and it is also easy to misuse. This page explains what a backtest can and cannot tell you, and what curve fitting is. It is part of our guide to NinjaTrader® automated strategies.
What a backtest does
A backtest replays old price data and applies the strategy's rules to it as if the strategy had been running at the time. The software records each simulated trade and produces statistics: number of trades, average trade, largest drawdown and so on. In NinjaTrader® 8, this is done in the Strategy Analyzer.
What a backtest cannot tell you
- It is hypothetical. No real orders were placed, so there were no real fills. A backtest assumes how your orders would have been filled, and real markets can differ.
- It is built with hindsight. The rules were chosen by someone who already knew what the market did.
- It does not predict the future. Markets change. A pattern that appeared in the past data may not appear again.
For these reasons, regulators require hypothetical results to carry a disclosure, and you should read any such results with care.
What is curve fitting?
Curve fitting, also called over-fitting or over-optimization, happens when a strategy's settings are tuned until they match the past data very closely. The strategy then looks excellent on that data, but it has learned the quirks and noise of the past, not a pattern that repeats. It tends to disappoint when it meets new data.
A simple picture: imagine drawing a line through a scatter of points. A smooth line captures the general direction. A wiggly line that touches every point captures every accident of that particular set of points, and is useless for predicting the next one.
Warning signs
- A great-looking result that only appears at one very specific combination of settings, and falls apart when a setting is changed slightly.
- Many adjustable settings compared with the number of trades in the test.
- Results shown only for the period the strategy was tuned on.
- No costs (commissions and slippage) in the numbers.
How traders reduce the risk
- Out-of-sample testing: tune the strategy on one stretch of data, then test it on a later stretch it has never seen.
- Walk-forward testing: repeat that process in a rolling way across time. NinjaTrader® 8's Strategy Analyzer includes walk-forward optimization.
- Robustness checks: change settings a little and see whether the results stay in the same neighborhood.
- Forward testing: run the strategy on a simulated account in real time before using real money.
Next steps
Use these ideas with our checklist for evaluating an automated trading system, and look at how our NinjaTrader® strategies are described on their product pages.
Futures trading contains substantial risk and is not for every investor. An investor could potentially lose all or more than the initial investment. Hypothetical performance results have many inherent limitations. No representation is being made that any account will or is likely to achieve profits or losses similar to those shown. Past performance is not necessarily indicative of future results.