With fast moving markets and every old trading strategy's life becoming shorter by the day back testing has become norm more than experiment
. Newer ideas and strategies are developed rapidly and thus back testing becomes part and parcel of this regime.
However back testing has its own perils. I have highlighted 3 of them in the article.
1. Error using Historical Data,
As you are using historical data and in future various decision points in past data may change based on current or future data. To illustrate the same say you have a strategy which uses highest high or a strategy which uses close of the day. Well the both this data are of past and projecting the same to make entry decision on future difficult. Best solution is to fine tune the strategy to use the data up to a historical point for testing only. For entry point one will have to use open of the day or open of the next tick as entry point and not close. Also high is only up to last tick so one has to check past ticks only.
2. Complicated Rules and Variables.
There is a myth that more complicated the system lesser is loss and more is profit. Thus many indicators and variables are added for testing. This may work perfectly fine for past data as we are back fitting those indicators which already worked. However in live data we do not know what would each variable have impact on our trading system on its own. If one checks systems of great traders (Turtles or Wizards) they made more money with simple system with best money management tools.
3. Black Swan events.
As mentioned by Nassim Nicholas Taleb black swan events happen and are opportunities for making maximum money. However the same events are grave yard for back testers whose strategies have worked so well in orderly markets.
What is the solution ?
Prepare and test your system for such events. Example if you are testing system for a particular amount of volatility brute force it and test for extreme conditions and see if the system still performs.