Backtesting 101: Verify With the Past Without Being Fooled By It

In earlier articles we said that received wisdom such as “a golden cross means it will rise” or “buy when RSI hits 30” does not in fact work well. But how do we know whether something “works” or “does not work”? The tool that answers that question with data is the backtest. You apply a trading rule to real historical data and verify “what results would this rule have produced in the past?” It is a powerful tool, and at the same time the tool that most reliably deceives beginners — and sometimes experts too. This article covers what a backtest is, and why you should not take its dazzling results at face value.

What a backtest is

A backtest consists of four steps. First, you define the trading rule precisely (something like “buy when the price breaks above the 20-day moving average and sell when it breaks below”). Second, you apply that rule to real historical price data, case by case, and run the trades in simulation. Third, you measure the results of that run from several angles — cumulative return, maximum drawdown (MDD), win rate, volatility and so on. Fourth, you look at those measurements and judge “did this rule work in the past?”

The appeal of a backtest is obvious. Instead of a vague subjective feeling that “this method seems good”, you can test a rule against the evidence of historical data. In verifying with data rather than hunches or rumours, the backtest is a core tool of evidence-based investing. Indeed, all of what is called quantitative — systematic, numerical — investing stands on rigorous backtesting. The problem is how easily the tool is misused.

Trap 1 — Overfitting, the most common and most fatal

The most dangerous trap in backtesting is overfitting. If you keep adjusting a rule’s settings this way and that until the best possible performance emerges from the historical data, what you end up finding is “the combination that happened to fit the past by chance”. A rule fine-tuned to maximise past returns — a moving average of 23 days rather than 20, a stop-loss of 6.5% rather than 7% — has not predicted the past; it has memorised it.

The diagram above shows that trap. Over the historical window (the data used for verification), a finely tuned rule produces dazzling performance, but it collapses once it enters the actual future window. Conversely, a simple and honest rule is less spectacular on the past but more robust in the future. The paradox that the more perfect a rule is on the past, the more likely it is to collapse in the future — that is the first lesson to carve into your mind when dealing with backtests.

Trap 2 — Look-ahead bias

Look-ahead bias is the mistake of quietly using information in a backtest that could not have been known at the time. For example, the rule “if that day’s close is above the moving average, buy at that day’s open” contains a contradiction: that day’s close is only known once the session has ended, yet the purchase is made at that day’s open. Calculating as though you knew the results before they were announced, or using later-revised data at an earlier point in time, is the same error. With this bias present, backtest performance comes out unrealistically good.

Trap 3 — Survivorship bias

Survivorship bias is the mistake of running a backtest using only the stocks that have survived to today. If you verify the past using only currently listed stocks rather than the full universe as it stood at some point in the past, the companies that were delisted or went under along the way drop out — even though those failed companies were also buy candidates at the time. The result is performance that looks better than it really was. An honest verification requires that the list of “what if I had bought with this rule back then” also include the companies that later disappeared.

Trap 4 — Trading costs and slippage

If a backtest leaves out trading commissions, taxes and slippage (being filled unfavourably rather than at the price you wanted), performance is inflated. This is especially true of rules that trade frequently, where these costs accumulate: profits on paper are often entirely eaten up by costs in practice. The liquidity problem covered earlier in the candlestick article carries through here too — in thinly traded stocks slippage is large, so however good the backtest, in live trading you cannot buy at that price.

Trap 5 — Data snooping and multiple testing

If you try enough rules and settings, something that fits the past well purely by chance is bound to appear. It is the same as the fact that if you toss a coin a few thousand times, there will be runs of consecutive heads. This is called data snooping, or the problem of multiple testing. Backtest 100 strategies and a few of them will look excellent by accident, but results that were good by accident are not reproduced in the future. This is why the statement “this strategy backtests well” is effectively meaningless until you also know how many attempts it was selected from.

Return alone is not enough — the many faces of performance

In backtest results, beginners see only the cumulative return. But the same return can be reached by an entirely different path. That is why maximum drawdown (MDD) must always be viewed alongside it. However high the final return, if you had to endure −60% along the way, most people would have abandoned the rule and sold during that stretch. As covered in investment risk, failure usually comes when the plan is discarded at the worst moment.

So a backtest has to be read for drawdown, volatility and risk-adjusted return (the Sharpe ratio and so on) alongside the return itself. The win rate is a useful reference too, but it has a trap — even with a 90% win rate, if the losing 10% loses heavily the total can be negative, while even with a 40% win rate, if the wins are large the result can be profitable. Performance is honestly assessed only when its several faces are viewed together, not as a single number.

Regime change — the market is not the same market

One fundamental limitation of backtesting is that the market of the past and the market of the future are not the same. The interest-rate environment, the participants, regulation and the speed at which information spreads all differ from era to era. This is called regime change. There is no guarantee that a rule which worked in a low-rate era will also work in a high-rate one. In particular, a rule that has latched onto a phenomenon valid only in a specific period collapses once that period ends.

Markets also have a self-extinguishing quality. When someone discovers a rule that works and it becomes widely known, the effect disappears as many people follow it. So the more a rule worked in the past, the more likely it is that it no longer works now. A backtest only shows you the past, and the moment you assume that past will repeat, you may be wrong.

With a small sample, chance is mistaken for skill

Reaching a statistically trustworthy conclusion requires a sufficient sample. From a handful of trades or a backtest over a short period, you cannot tell whether the performance was skill or chance. It is the same as the fact that winning 7 out of 10 trades does not let you call it a strategy with a 70% win rate. An honest backtest therefore has to secure a sufficient number of trades across a long period, many stocks and several markets.

A small sample particularly increases the risk of the data snooping described above, because with little data a rule that happens to fit well emerges far too easily. That is why the claim “this rule has been perfect on this stock for the past six months” is close to meaningless. Six months and one stock is a sample short enough and narrow enough for chance to dominate.

Why a genuinely unbeatable strategy is never sold

There are plenty of places selling a “backtest-verified, can’t-lose trading method”. One piece of common sense is worth remembering here. If a rule really did make large sums easily, there would be no reason to sell it to anyone else — you would simply use it quietly yourself. And if such a rule were sold widely and used by many people, then as noted above its effect would soon disappear.

So the phrase “verified can’t-lose strategy” is close to a contradiction in itself. What genuinely works is usually simple and already known, and even that is hard to stick to consistently. When you see a dazzling backtest curve and an advertised high return, it is safer to think first — before being impressed — about which of the traps covered in this article it might have fallen into: overfitting, survivorship bias, omitted trading costs, data snooping.

So how do you use a backtest honestly

There are a few principles for using backtests properly. First, out-of-sample verification. Separate the period used to build the rule from the period used to verify it, and check whether it also works on “data you did not see while building it”. Second, keep it simple. The fewer settings there are to adjust, the lower the risk of overfitting. Third, allow for trading costs generously. Fourth, take a large sample (many stocks, a long period, several markets) to filter out chance. Fifth, look at the results with a sceptical eye — if they are too good, something is probably wrong.

The human psychology a backtest misses

A backtest assumes the rule was followed perfectly. But real trading is done by people. In a backtest the data passes indifferently through a −30% drawdown, but living through a stretch where your own money is down 30% and sticking to the rule without emotion is an entirely different matter. Most people abandon the rule in fear during that stretch. So even a backtest that looks good on screen leaves the question of whether a human being can actually adhere to it as a completely separate problem.

The conditions for a good strategy therefore include not just “is the return high” but “can I hold to it through the bad stretches”. A strategy with a drawdown you cannot bear ends in surrender in live trading, however dazzling the backtest. A strategy beyond your own psychological limit is, for you, a bad strategy. This is the decisive gap between practice and theory that the backtest numbers alone can never show.

Fundamentals need verification too

Backtesting is not used only for technical rules. Fundamental strategies — “did low-PER stocks beat the market?”, “did high-ROE stocks generate excess returns?” — are also verified with data. In fact the long-run performance of factors such as value, profitability and size has been studied in academia for a long time. The same traps apply here too, though: there is no guarantee that a factor that worked in the past will work in the future, and once it becomes widely known the effect weakens.

So whether fundamental or technical, the same attitude is required when you encounter any rule: ask “has this been verified honestly?” That is why this site, instead of advertising dazzling backtest returns, stakes out a scenario for each individual stock and builds a record by updating its status quarterly. An honest record verified against the future is more trustworthy than a finely tuned curve that memorised the past. The real lesson of backtesting is: “verify — but do not be deceived by your verification.”

Individual investors and backtesting

So how should an individual investor treat backtesting? You do not need to run elaborate backtests yourself, but the spirit of it is essential. It is the attitude of asking, when you come across a trading method or an indicator, “has this really been verified, and if so, did that verification avoid these traps?” You should be strongly sceptical in particular of anywhere selling a “backtest-verified, can’t-lose strategy”, because if there were a simple rule that genuinely worked, it would already be widely known and its effect would have vanished.

The fact that this site stakes a falsifiable core scenario on every stock analysis and updates it quarterly in stock tracking is also a way of applying the spirit of backtesting to the future. Instead of memorising the past, it verifies over time whether the logic set out today holds up against actual data going forward. This “forward test” is free of overfitting — because a future that has not yet arrived cannot be memorised.

One analogy helps. A backtest is like looking back over the road you have travelled and drawing the optimal route, saying “it would have been quicker to go this way”. A route drawn after you already know where all the potholes were looks perfect. But on the road ahead you do not know where the potholes will be. The more exactly a route is fitted to the road already travelled, the easier it is to stumble on the road ahead. That is the essence of overfitting.

So the real purpose of learning about backtesting is not “to find the perfect rule” but to develop “an eye that is not fooled by a plausible one”. Recalling the traps in the face of a dazzling performance curve, asking whether the verification was honest, doubting what looks too good. That balance of scepticism combined with respect for data is the heart of evidence-based investing. To verify, and to stay humble about the verification — that is the most valuable lesson backtesting gives us.

Paper trading — the bridge before live trading

If a backtest is verification against the past, paper trading (simulated investing) is verification aimed at the future. Before putting real money in, you trade the rule you have set out with virtual money and test it against data from now on. This is a form of the forward test described above: it is free of overfitting and has no look-ahead bias either, because it verifies with data that has not yet arrived.

Paper trading has its limits, though. Virtual money is not real money, so the fear of a drawdown and the greed of a profitable stretch do not operate as they do in reality. As a result it is common for someone who kept to a rule perfectly in simulation to break it in live trading. Even so, paper trading is a useful bridge for spotting logical flaws in a rule in advance and for getting the rule into your hands before going live. Using verification against the past (backtesting) together with verification against the future (forward testing) is safer than relying on either alone.

To sum up, backtesting, paper trading and forward testing are all tools born from the same spirit of “confirming with data whether my judgement is right”. They differ only in direction (past or future) and in method. This spirit of verification is the root of investing that is not swayed by hunches and rumours, and it is why this site attaches a logic to every stock analysis and leaves it on the record.

Frequently asked questions (FAQ)

Q1. If a backtest return is high, isn’t it a good strategy?

Not necessarily. It may have been inflated by overfitting, survivorship bias, omitted trading costs and so on. The better the result, the more it should be doubted. Whether that backtest also works out of sample, and whether it avoided the traps, matters more than the return figure.

Q2. So is backtesting useless?

No. Done honestly, it is very useful for falsifying and filtering out rules. It is simply stronger at “excluding bad strategies” than at “proving good ones”. And you should look at whether it avoided the traps rather than at how good the result was.

Q3. Is forward testing better?

It has the great advantage of being free of overfitting, because it verifies with future data that has not yet arrived and so cannot memorise the past. It simply takes a long time. This site’s practice of staking a core scenario and updating its status quarterly in stock tracking is a form of forward testing.

A checklist for dealing with backtests

1. Out-of-sample verification: does it also work on data not seen while building it?
2. Simplicity: the fewer the adjustable settings, the lower the overfitting risk.
3. Costs included: commissions, taxes and slippage, generously.
4. Sample size: a long period, many stocks, a sufficient number of trades.
5. Doubt: if it looks too good, something is wrong.

In summary

A backtest is a powerful tool for verifying a trading rule against historical data, but it is riddled with traps: overfitting, look-ahead bias, survivorship bias, omitted trading costs and data snooping. That is why dazzling backtest results are usually either memorisations of the past or products of chance. An honest backtest keeps to out-of-sample verification, simplicity, generous cost assumptions, a large sample and an attitude of doubt. Above all, a good backtest does not promise the future; it falsifies a rule. In the next article — the last of the technical track — we deal with the liquidity trap to watch for especially when looking at the charts of overlooked stocks outside the market’s attention, and we join the fundamental and technical tracks into one.

Investment study · technical analysis. This article is for information purposes and is not a recommendation to buy or sell any particular security. The example charts use hypothetical data for the purpose of explaining the concepts.

Related reading

First Month of the Higher Delisting Market-Cap Threshold — 174 Below the Line, 465 Next YearScreening Methodology for Overlooked Stocks — How 2,765 Companies Became 207Technical analysis, from the ground up — a six-part learning roadmap