Getting bad backtesting results? Read this.

Are you writing automated strategies that seem to test well, but then fail when run in a live account?  There may be a number of reasons this is happening, so today we are going to look at one possible cause for poor back-testing results.

But first, let’s go over a few things we already know about back-testing.  First, back-testing is the process of running your automated strategy on historical data.  That’s the definition. Obviously, it would be foolish to write a strategy, and then run it live without testing. Back-testing is a required step in the trading system development strategy.

Second, back-testing is difficult; it requires meticulous attention to detail, careful record-keeping, and knowing a lot about your strategy, your parameters, which data you are using to tune performance, and which data you are testing “blind” on.  That is, which data is in sample, and which is out of sample.  I’ve written before on in-sample and out-of-sample issues.

But back-testing is deceptively difficult because it looks so easy.  In the case of the popular Metatrader 4 (MT4) currency trading program, it is so easy to open up the strategy tester, choose a strategy, set some parameters, choose a time frame, and then click on Start.  Literally, you can start a back-test in about 8 clicks.  The Strategy Tester will produce a nice graph, and a nice formatted report with detailed metrics on how well your strategy did.  If your graph slopes up and to the right, you might think you’re done.  Done. Profitable. What more do we need to do?

The deception is that you are most likely using bad data.  I hesitate to call it bad, because it’s not completely bad.  It’s just that it could be so much better.  The historical data that you test against in Metatrader 4 is approximate.  We’ll save the details of why the data is approximate for another time.  For now, it is sufficient to say that the data that comes with Metatrader cannot give you an accurate picture of how your strategy will behave in a live market.

Here’s an example: go into Metatrader’s History Center, and find a currency pair and time frame, say, EUR/USD 15-minutes, and export that data to a CSV file. If you open that CSV file in Excel, you’ll find one line per 15-minute block of time.  That one line will contain the Open, Close, High and Low values for that 15-minute time period.  In a live market, there will be hundreds if not thousands of data points during that 15-minute time frame.  The default data that you get from the History Center leaves most of the market activity out.

If you are testing an intra-day strategy, and your system relies on indicators that do math calculations (they all do), then you will see a remarkable difference between the calculations on one or two data points versus the calculations on hundreds or thousands of data points.  It should be easy to see that the more data points in your historical data, the better your indicator calculations, and the better the indicator values, the more likely you’ll know if your software is making good choices.  It’s literally the difference between profit and loss.

One other thing I can point out is in the Metatrader 4’s Back-tester report.  At the top of the report you’ll see a section that says “Ticks Modeled”.  If you are using the default free data, note the number of ticks modeled and the number of bars in the test; if you test with high quality data, the same time frame, and the same number of bars, you will see that the “Ticks Modeled” count will be much higher for the quality data — even 15 times higher.

So, what to do about this?  Thankfully, there are several options, both free and commercial that can give you better back-testing results.

The commercial solution I use is called Live Tickdata Client, and I got it from The Tickdata Client allows you to download historical data from about a dozen different currency brokerages, all major currency pairs, any common time frame, and in either “tick data” format, or “1-minute” format.  We didn’t go into the details of how tick data is stored compared with how one-minute data is stored, mainly because Metatrader cannot import tick data natively.  If you import data into MT4, you must use OHLC format, which is how the 1-minute data file is stored.  The point is that you get high quality data to test with.  Using the TickData Client is easy; it downloads the data, unpacks it, and sends it over to MetaTrader where you can test with it.  The cost of using Live TickData client starts as low as $15 to $20 per month, and goes up from there, depending on if you pay in advance, and some of the optional features.  I encourage you to take a look at this.

Free solutions are available too. The broker offers Forex data downloads for free; the download is throttled, and you can’t ask for too much at a time; but you can get the data if you are patient.

Most other trading platforms, such as TradeStation, AmiBroker, JForex, and ThinkOrSwim, have high quality tick-level data available natively, so this problem seems to be prevalent mainly in the currency trading world of Metatrader.

Now you know the importance of high quality data, and how to get a hold of good data.  Make sure you are not making the mistake of relying on poor quality data to test your strategies.


Stop wasting time. Scan!

How much time to do you spend every day looking for a trade setup?  If you have a good system, and you know your setups well, do you spend a long time looking through a dozen stocks or currency pairs, checking to see if your setup is there in each one?

I’m getting more and more excited about the idea of scanning.  With scanning, you only need to program in your setup criteria, which financial instruments to scan, and let the scanner tell you which stocks, futures or currency pairs look like they’re getting ready for your set up.

The trick is, of course, to write the scanner, but that’s what we’re here for, right?  A scanner is a lot like writing an indicator for a trading platform, but instead of drawing it on the screen for one chart, the scanner checks multiple charts, and just lets you know which chart might be the ones you should look at.

In John Person’s book Mastering the Stock Market, he spends a bit of time talking about scanning.  It’s in the section on Relative Strength, which I wrote about last week.

scanning-jpm-chartSo, a quick recap here.   On the chart, the top graph is the stock we want to trade; in this case, JPM.  The lower graph shows the relative strength of the JPM stock with the financials XLF Exchange Traded Fund.  Additionally, on the bottom graph, there’s a 9-period moving average that gives us some cross-over points.

Where ever the moving average and the Relative Strength lines cross, that’s a change.  If Relative Strength is above the moving average, that means our stock is doing better than the rest of its industry, and we want to be in that stock.  When the Relative Strength line falls below the moving average, that means it’s doing less well than the rest of the industry, and we want to be out of that stock.

So, how about scanning?  The scanner needs to do the following for each stock (let’s just stick with stocks for now, and let’s assume we’re using daily charts)

  1. Look up the associated ETF or index for the given stock
  2. Load the daily chart for the stock
  3. Load the Relative Strength indicator for the the stock and the paired ETF
  4. Apply a moving average to the Relative Strength buffer
  5. If the moving average has crossed the Relative Strength in the last few days (configurable), alert me.

It would be fairly simple to do this with a variety of platforms, and we looked at something like this from Howard Bandy in his Quantitative Trading Systems book.  We’ll look at that more closely in an upcoming blog post.


Comparative Relative Strength

I’ve been reading some of John Person’s books lately.  If you haven’t read anything by him, you owe it to yourself to get a hold of his trading books and read them.  Seriously, they are some of the best material that I’ve read on trading.

I recently picked up Mastering the Stock Market again in preparation for my interview with John, and one chapter in particular caught my eye–the chapter on Comparative Relative Strength.  (Note that this shouldn’t be confused with the “RSI” indicator, which is the “relative strength indicator”.  The two are completely different.)

The basics of Comparative Relative Strength (CRS) is that you take one chart and divide it by another. The result is a simple line graph of the relative strength of the two markets.  The line graph is the “difference” between the two charts.

For example, you might take the chart for JP Morgan (JPM) and do a relative strength comparison to the XLF financial sector ETF.  What this does is it tells you how those two markets compare to each other.  This is specifically interesting when you compare a stock and a sector ETF, because it tells you how well the stock is doing compared to the rest of the stocks in its sector.  This is one way to do sector analysis.

Another way to use this would be to compare an individual stock with the S&P 500, which will tell you how the stock is doing compared with the broader market.

So, if the stock and the market we are comparing it to were exactly aligned, the result would be a straight line.  Not interesting.  However, when we compare JPM and the XLF, we see places where the stock is outperforming the rest of the financial market where the CRS line is trending up.  That’s good to know, especially when evaluating a stock. You not only want to know how it’s doing individually, but especially how it’s doing compared to its sector.

I set up this exact pair on both the ThinkOrSwim platform and the Amibroker platform, and it works beautifully on both platforms.  With ThinkOrSwim, the indicator is called RelativeStrength, and you set up the one stock in a chart, and set up the Relative Strength chart in a lower sub-graph.  In Amibroker, it’s a function called RelStrength.  It appears to be common enough that it is likely to be in other major platforms as well.relative-strength

Here’s the best part of the chapter: you can apply a moving average to the Relative Strength line and use that as a buy or sell signal.  The same way you might have used a moving average on just a stock in the past, you can now use a moving average on the relative strength of a stock within its sector.  Because moving averages act like automated trend lines, you can program a system with a moving average to alert you of changes in the trend.  That means when a stock begins to outperform its sector, you can be alerted of this fact (and buy); when the stock performs poorly compared to its sector, you can be alerted that the trend is over.  John describes how you can use this CRS with a moving average system to write a tool to automate the signals.  Though not a full trading system, it would be used to alert you of stocks that are outperforming the rest of the market, as well as stocks that are under-performing.   The automated alert system would scan for the change in relationship to let you know when a stock was trending higher than the market or ending that trend.

If I were to pick one project to work on next, this would be it.   Sector analysis is powerful, and it should definitely be part of your tool box.