Getting bad backtesting results? Read this.

Are you writing automated strategies that seem to test well, but then fail when run in a live account?  There may be a number of reasons this is happening, so today we are going to look at one possible cause for poor back-testing results.

But first, let’s go over a few things we already know about back-testing.  First, back-testing is the process of running your automated strategy on historical data.  That’s the definition. Obviously, it would be foolish to write a strategy, and then run it live without testing. Back-testing is a required step in the trading system development strategy.

Second, back-testing is difficult; it requires meticulous attention to detail, careful record-keeping, and knowing a lot about your strategy, your parameters, which data you are using to tune performance, and which data you are testing “blind” on.  That is, which data is in sample, and which is out of sample.  I’ve written before on in-sample and out-of-sample issues.

But back-testing is deceptively difficult because it looks so easy.  In the case of the popular Metatrader 4 (MT4) currency trading program, it is so easy to open up the strategy tester, choose a strategy, set some parameters, choose a time frame, and then click on Start.  Literally, you can start a back-test in about 8 clicks.  The Strategy Tester will produce a nice graph, and a nice formatted report with detailed metrics on how well your strategy did.  If your graph slopes up and to the right, you might think you’re done.  Done. Profitable. What more do we need to do?

The deception is that you are most likely using bad data.  I hesitate to call it bad, because it’s not completely bad.  It’s just that it could be so much better.  The historical data that you test against in Metatrader 4 is approximate.  We’ll save the details of why the data is approximate for another time.  For now, it is sufficient to say that the data that comes with Metatrader cannot give you an accurate picture of how your strategy will behave in a live market.

Here’s an example: go into Metatrader’s History Center, and find a currency pair and time frame, say, EUR/USD 15-minutes, and export that data to a CSV file. If you open that CSV file in Excel, you’ll find one line per 15-minute block of time.  That one line will contain the Open, Close, High and Low values for that 15-minute time period.  In a live market, there will be hundreds if not thousands of data points during that 15-minute time frame.  The default data that you get from the History Center leaves most of the market activity out.

If you are testing an intra-day strategy, and your system relies on indicators that do math calculations (they all do), then you will see a remarkable difference between the calculations on one or two data points versus the calculations on hundreds or thousands of data points.  It should be easy to see that the more data points in your historical data, the better your indicator calculations, and the better the indicator values, the more likely you’ll know if your software is making good choices.  It’s literally the difference between profit and loss.

One other thing I can point out is in the Metatrader 4’s Back-tester report.  At the top of the report you’ll see a section that says “Ticks Modeled”.  If you are using the default free data, note the number of ticks modeled and the number of bars in the test; if you test with high quality data, the same time frame, and the same number of bars, you will see that the “Ticks Modeled” count will be much higher for the quality data — even 15 times higher.

So, what to do about this?  Thankfully, there are several options, both free and commercial that can give you better back-testing results.

The commercial solution I use is called Live Tickdata Client, and I got it from The Tickdata Client allows you to download historical data from about a dozen different currency brokerages, all major currency pairs, any common time frame, and in either “tick data” format, or “1-minute” format.  We didn’t go into the details of how tick data is stored compared with how one-minute data is stored, mainly because Metatrader cannot import tick data natively.  If you import data into MT4, you must use OHLC format, which is how the 1-minute data file is stored.  The point is that you get high quality data to test with.  Using the TickData Client is easy; it downloads the data, unpacks it, and sends it over to MetaTrader where you can test with it.  The cost of using Live TickData client starts as low as $15 to $20 per month, and goes up from there, depending on if you pay in advance, and some of the optional features.  I encourage you to take a look at this.

Free solutions are available too. The broker offers Forex data downloads for free; the download is throttled, and you can’t ask for too much at a time; but you can get the data if you are patient.

Most other trading platforms, such as TradeStation, AmiBroker, JForex, and ThinkOrSwim, have high quality tick-level data available natively, so this problem seems to be prevalent mainly in the currency trading world of Metatrader.

Now you know the importance of high quality data, and how to get a hold of good data.  Make sure you are not making the mistake of relying on poor quality data to test your strategies.


Did you enjoy this article?
"Get my free programmer's checklist"
The checklist gives you step-by-step instructions on how to create the perfect automated trading system.


Leave a Reply

Your email address will not be published. Required fields are marked *