A backtest replays a trading strategy over historical prices to see how it would have performed. Done carefully, it is the cheapest way to discard bad ideas before they cost real money. Done carelessly, it is the easiest way to convince yourself a bad idea is a good one. This guide covers the habits that separate the two, and ends with an honest account of how we tested, and rejected, several strategies before choosing the one SignalEye uses today.
Replay the past one candle at a time
The most important rule is that every decision in the backtest may use only information that existed at that moment. A walk-forward replay steps through history candle by candle; at each close it calculates the indicators from past candles only, decides whether to trade, and then lets the following candles determine the outcome.
Breaking this rule is called look-ahead bias, and it sneaks in easily:
- Entering at a candle's open using a signal that is only known at that candle's close.
- Indicators or normalisations calculated over the whole data set, including the future.
- Choosing which coins to test based on how they performed later. Testing only today's surviving, popular coins quietly flatters any strategy.
If a result looks too good, look-ahead bias is the first thing to check.
Count every cost
Fees are small per trade and large in total. At 0.1% per side, every round trip costs 0.2%, and a strategy that trades often can lose its entire edge to them. Add an allowance for slippage too: breakout entries usually fill a little above the signal price and stops in fast markets a little below. Always report results after costs.
Resolve ambiguity against yourself
Historical candles only tell you the open, high, low and close, not the order in which prices were reached. When a single candle's range touches both your stop and your target, you can't know which came first. The conservative assumption is that the stop was hit. Apply the same pessimism elsewhere: assume entries at the signal price or worse, never at the candle's best price. A strategy that only works under optimistic assumptions doesn't work.
Keep some data you haven't looked at
Split history into an in-sample period, where you develop and tune the strategy, and an out-of-sample period, which you test only once the rules are fixed. In-sample results are always flattering, because the rules were chosen to fit that data. Out-of-sample results are the closest a backtest gets to a real forecast of performance. If you tune again after seeing the out-of-sample result, it becomes in-sample data.
Beware of testing too many ideas
Every extra setting you try is another chance to find something that worked by luck. If you test around 1,900 combinations of lookbacks, stop distances, targets and filters, as we did, the best in-sample result is almost certainly partly luck, even when no setting has a real edge. This is the multiple comparisons problem, and it is the core of overfitting.
Some defences:
- Prefer simple rules with few parameters.
- Look for broad regions of settings that all work, not a single sharp peak.
- Expect out-of-sample results to be clearly worse than the best in-sample result, and judge the strategy on the former.
Compare against the mirror image
A useful control is the reversed-direction test: replay every trade in the opposite direction, with the stop and targets mirrored, and the same costs. If a strategy has a genuine edge, it should clearly beat its mirror. If the reversed version does as well or better, the "signals" are noise, or are pointing the wrong way.
Break the results down
A single overall average can hide a lot. Check the result year by year and coin by coin. A strategy whose entire profit comes from one year or one coin is a bet on that period repeating. You also want enough trades in each slice for the numbers to mean something; a handful of trades proves little either way.
Two summary numbers are worth reporting alongside the average: the profit factor (gross profit ÷ gross loss, where 1.0 is break-even) and the distribution of outcomes, so readers can see how trades actually ended.
Expect most ideas to fail
Most strategies don't survive honest testing. In our own work, short-selling variants and mean-reversion strategies (buying sharp dips and selling sharp rallies) looked acceptable on parts of the data but did not hold up out-of-sample. That is a normal outcome, not a failure of the process. A backtesting process that never rejects anything isn't testing.
Case study: how SignalEye chose its strategy
We used 5 years of 4-hour candles from Binance, October 2021 to October 2026, across 12 coins: BTC, ETH, SOL, XRP, ADA, PEPE, SHIB, TON, BNB, DOGE, LINK and AVAX. Fees were charged at 0.1% per side, and every test used the rules above.
What failed: an AI forecasting model
Before launch, SignalEye tested a pretrained AI forecasting model for candlesticks, sending its signals to the team only (paper trading). Replayed over 60 days it produced 110 signals; 39% hit the first target, and the average result was −0.62% per trade after fees. The reversed version averaged +0.28%. In other words, it had no edge, and doing the opposite would have been better. We dropped it.
What survived: a breakout strategy
The strategy we chose is a buy-only breakout trend-following system with trend and market-regime filters and volatility-based stops. Over the five years it took 557 trades, averaging +1.07% per trade after fees, with a profit factor of 1.36. The reversed control averaged −0.86%, and the strategy was profitable on 11 of the 12 coins.
| Year | Trades | Average per trade |
|---|---|---|
| 2021 (Q4) | 21 | +0.78% |
| 2022 | 11 | −0.38% |
| 2023 | 142 | +0.41% |
| 2024 | 180 | +2.21% |
| 2025 | 159 | +0.04% |
| 2026 (to October) | 44 | +2.73% |
How the 557 trades ended:
- 29% hit the stop loss.
- 6% reached all targets.
- 11% reached the first target, then closed at the entry price or at expiry.
- 54% were closed at the 72-hour limit: 28% in profit and 26% at a loss.
So only about one trade in six reached the first target. The strategy makes money because full losses are limited to under a third of trades and the winners are larger than the losers, not because it is usually right.
The caveats
- The edge is modest and uneven. Most of the profit came from 2024 and 2026; 2025 was essentially flat. Long flat or losing stretches should be expected.
- Some data snooping. The market-regime filter, which keeps the strategy out when Bitcoin is below its 200-day average, was added after seeing how badly long breakouts did in 2022. That makes 2022's result look better than a truly blind test would have.
- The coin list was chosen today, which carries a degree of survivorship bias.
- Past results don't guarantee future results. Markets change, and a backtest is a simulation, not a record of real fills.
For those reasons the strategy is paper-traded live, on prices it has never seen, before its signals are published, and every published signal, including losses, appears on the public track record. That live record, not the backtest, is what you should judge us by.
Reading someone else's backtest
When a provider shows you backtested results, ask:
- Are fees and slippage included?
- How were trades where the stop and target were touched in the same candle counted?
- How many settings were tried, and is there an out-of-sample period?
- How does the strategy compare with its reversed version or another simple baseline?
- What do the results look like year by year and asset by asset?
- Is there a live record since the backtest that you can check?
A provider that can't answer these gives you little reason to trust the headline number. Our guide on how to evaluate a crypto signal provider covers the rest of the checklist.