Getting bad backtesting results? Read this.

Are you writing automated strategies that seem to test well, but then fail when run in a live account?  There may be a number of reasons this is happening, so today we are going to look at one possible cause for poor back-testing results.

But first, let’s go over a few things we already know about back-testing.  First, back-testing is the process of running your automated strategy on historical data.  That’s the definition. Obviously, it would be foolish to write a strategy, and then run it live without testing. Back-testing is a required step in the trading system development strategy.

Second, back-testing is difficult; it requires meticulous attention to detail, careful record-keeping, and knowing a lot about your strategy, your parameters, which data you are using to tune performance, and which data you are testing “blind” on.  That is, which data is in sample, and which is out of sample.  I’ve written before on in-sample and out-of-sample issues.

But back-testing is deceptively difficult because it looks so easy.  In the case of the popular Metatrader 4 (MT4) currency trading program, it is so easy to open up the strategy tester, choose a strategy, set some parameters, choose a time frame, and then click on Start.  Literally, you can start a back-test in about 8 clicks.  The Strategy Tester will produce a nice graph, and a nice formatted report with detailed metrics on how well your strategy did.  If your graph slopes up and to the right, you might think you’re done.  Done. Profitable. What more do we need to do?

The deception is that you are most likely using bad data.  I hesitate to call it bad, because it’s not completely bad.  It’s just that it could be so much better.  The historical data that you test against in Metatrader 4 is approximate.  We’ll save the details of why the data is approximate for another time.  For now, it is sufficient to say that the data that comes with Metatrader cannot give you an accurate picture of how your strategy will behave in a live market.

Here’s an example: go into Metatrader’s History Center, and find a currency pair and time frame, say, EUR/USD 15-minutes, and export that data to a CSV file. If you open that CSV file in Excel, you’ll find one line per 15-minute block of time.  That one line will contain the Open, Close, High and Low values for that 15-minute time period.  In a live market, there will be hundreds if not thousands of data points during that 15-minute time frame.  The default data that you get from the History Center leaves most of the market activity out.

If you are testing an intra-day strategy, and your system relies on indicators that do math calculations (they all do), then you will see a remarkable difference between the calculations on one or two data points versus the calculations on hundreds or thousands of data points.  It should be easy to see that the more data points in your historical data, the better your indicator calculations, and the better the indicator values, the more likely you’ll know if your software is making good choices.  It’s literally the difference between profit and loss.

One other thing I can point out is in the Metatrader 4’s Back-tester report.  At the top of the report you’ll see a section that says “Ticks Modeled”.  If you are using the default free data, note the number of ticks modeled and the number of bars in the test; if you test with high quality data, the same time frame, and the same number of bars, you will see that the “Ticks Modeled” count will be much higher for the quality data — even 15 times higher.

So, what to do about this?  Thankfully, there are several options, both free and commercial that can give you better back-testing results.

The commercial solution I use is called Live Tickdata Client, and I got it from The Tickdata Client allows you to download historical data from about a dozen different currency brokerages, all major currency pairs, any common time frame, and in either “tick data” format, or “1-minute” format.  We didn’t go into the details of how tick data is stored compared with how one-minute data is stored, mainly because Metatrader cannot import tick data natively.  If you import data into MT4, you must use OHLC format, which is how the 1-minute data file is stored.  The point is that you get high quality data to test with.  Using the TickData Client is easy; it downloads the data, unpacks it, and sends it over to MetaTrader where you can test with it.  The cost of using Live TickData client starts as low as $15 to $20 per month, and goes up from there, depending on if you pay in advance, and some of the optional features.  I encourage you to take a look at this.

Free solutions are available too. The broker offers Forex data downloads for free; the download is throttled, and you can’t ask for too much at a time; but you can get the data if you are patient.

Most other trading platforms, such as TradeStation, AmiBroker, JForex, and ThinkOrSwim, have high quality tick-level data available natively, so this problem seems to be prevalent mainly in the currency trading world of Metatrader.

Now you know the importance of high quality data, and how to get a hold of good data.  Make sure you are not making the mistake of relying on poor quality data to test your strategies.


Multiple time frame use in trading systems

When we first start to trade, one of the rules of trading is “Don’t try to trade on multiple time frames.  Stick to one time frame.”  For a beginning trader, that makes a lot of sense.

When I first started trading I wanted to use a dozen indicators, and have multiple time frames up so that I could see every possible tool that would tell me which way the market was moving, and thus ensure I was placing my trade in the right direction.

However, what ended up happening (you can see this coming, right?) is that instead of all the indicators and time frames aligning, I got conflicting information.  The 15-minute chart would be showing a nice uptrend, whereas the 1-hour or 4-hour chart would be trending down; moving average cross-overs would be telling me it’s time to buy, but the RSI would be telling me to sit still.  More information is not better for a beginning.

What I needed to be told was “watch the market and learn how it moves.”  Use a single time frame, and one indicator, and learn it.  Even if the indicator doesn’t provide great buy or sell signals, it’s so valuable to learn to watch for certain signals, patterns, and market behavior; once you’ve figured that out, you only need one or two indicators, and the rest is just observing and watching for what happens in your trade setup.  Expert traders I’ve studied definitely do use multiple time frames, watching for “confluence”, where a higher time frame confirms a support or resistance level seen on your current time frame.

On the other side of things is the automated system developers.  I’ve seen a number of books and articles that describe how to use multiple time frames in a strategy to confirm a trend or use the higher time frame to determine what kind of market we are currently in.  Howard Bandy uses a higher time frame in several of his strategies in the two books I have,  Quantitative Trading Systems and Mean Reversion Trading Systems.

Because we’re peeking at multiple time frames inside the program code, it’s a little easier to do.  We don’t need to worry about getting confused or having too much going on at the same time, because the computer will be just fine with multiple time frames.  It’s not a good idea to just throw it in the program anywhere; obviously, it needs to be part of the greater strategy you are trying to implement.  Trading strategies can often benefit from extra confirming checks using multiple time frames.

It’s a little tricky to get the trading platform to see two time frames at the same time.  In Metatrader, many function calls require an “int timeframe” parameter, and we use a zero there to indicate the current time frame.  If we are writing code that uses multiple time frames, it is likely you will know the time frames in advance (say 15 minute and hour charts), so in each indicator, you will need to specify which time frame that calculation is being done for.

If you’re organized, it shouldn’t be hard; using the additional longer time frame will give you the confirmation you’re looking for.


Bitcoin: Trading a virtual currency

Have you heard of Bitcoin?  If you haven’t, here’s a quick overview.

Bitcoin is a kind of currency, but it isn’t backed by any government.  Instead, it’s generated, tracked and traded electronically, peer-to-peer, without any intermediate financial currency.  It’s based on an extremely complex encryption algorithm. The servers that keep track of Bitcoins are called bitcoin miners, and these servers also create new bitcoins electronically (but quite slowly).  Wikipedia has a really good article on Bitcoin, as they are known to do for techie types of things.

Here’s a few interesting things about Bitcoin: you can trade it like any other currency.  So, several small brokers allow you to trade Bitcoin against other major currencies like the Dollar and the Yen.  One brokerage, AvaTrade has been making the news recently as they allow you to trade Bitcoin with Metatrader 4, but it appears they are not licensed to operate in the US.  There may be other brokerages that operate in the US that offer Bitcoin.

Another interesting fact is that a court case in Texas just ruled that Bitcoin can be treated legally like a currency.  The case involves an individual who committed fraud in the form of a Ponzi  using Bitcoin.  The judge noted that Bitcoin is more like a precious metal than a currency, but the fact that it has value legally is interesting.

Another interesting point is the volatility of the currency.  Just read the Wikipedia article to bitcoin-chartssee how far the value goes up and down:  $13 at the beginning of 2013, then $230 in April of this year.  If you’re looking to trade a really wild currency, maybe this is the one for you.One of the confusing things about looking at the Bitcoin charts is that there are dozens of “markets” or actually brokerages that trade Bitcoin with other currencies.  They appear to have different slightly different values for Bitcoin.  (Maybe an opportunity for arbitrage?)  But if you want to trade Bitcoin on Metatrader, you pick a broker that supports both, set up your charts, do your technical analysis, and even run automated strategies on it.  This sounds like fun.

Have you traded Bitcoin?  Have you used a trading platform? If so, what’s your experience been like?


My Favorite Forex Book

One of the first books I read on Forex trading was John Person’s Forex Conquered.  I was just re-reading parts of that book this week, and I wanted to highlight some of the best parts of the book.

John Person has a whole section on Trading Systems in which he explains how some of his tools and strategies from earlier in the book can be applied to automated trading systems.  Person’s strategies are excellent for automated systems because they are clear-cut; he has discrete rules for entries and exits, as well as for profit targets and stop losses.  In fact, the almost every chapter of “Forex Conquered” has knowledge that can improve your automated trading systems.

Person has a couple trademark tools he describes in almost all of his books: Pivot Points, seasonality, and the Doji chart pattern.  The Trading Systems chapter covers Pivot Points, and touches on seasonality with respect to automated systems.

Person takes three separate automated trading systems and compares their results.  He first explains a system that uses Pivot Points and the Stochastics Oscillator; he describes the system’s sell logic in that it waits for price movement through Pivot Point R1, R2 or R3, and then looks for the Stochastics Oscillator to cross up over the 80 percent level, and then back down over 70 percent level.  (Buy signals are similarly calculated.)  He also explains some of his thought process on setting stops, and reasonable profit targets.  And best of all, he provides some actual code to look at.  In this example, he shows the system written in TradeStation’s EasyLanguage programming language.

At the end of this section of the chapter, Person presents some statistics showing net profit, profit factor, winning percentage, and the standard stats you would expect to see from a trading system.  Just for a point of reference, the Profit Factor of this system is 1.86.

The second system that Person explains is one that uses Pivot Points and the MACD histogram indicator; the program logic is essentially the same as the Stochastics program, with just the indicator switched to the MACD.   Again, Person presents the statistics for the MACD version, and compares the results with the Stochastics version.  The Profit Factor for this MACD system is 1.35.

He concludes that the Stochastics version is a bit better, because the MACD’s signals lag a bit behind the price movement, and thus we are not able to enter a trend at the beginning or exit at just the right time.

The third system Person looks at is his Moving Average Pivot Point system (which he calls Defcon III). This is his trademark system, and it has served him very well over the years.  When Person teaches trading to new traders, the Moving Average Pivot Point system is usually a part of it.

John Person describes much of how the moving average automated system works, and gives nice hints as to how it should be programmed.  However, he does not provide code for this system, and in a conversation with him last year, he mentioned that was intentional.  It’s left as an exercise for the reader.

I spent some time writing this system for Metatrader 4, and it is working quite well.  It’s not perfect yet, and I need some more time on it, but I like the results I see.

At the end of this chapter, Person shows the statistics for the Moving Average Pivot Point system, and as you might expect, the stats are much better for this system, compared with the other two.  The profit factor for this system is a nice 2.03.

Even though the book has been out for a while, if you haven’t read “Forex Conquered”, I recommend that you do.  The Pivot Point Moving Average system is explained in a number of chapters throughout the book, and I believe the concepts should be part of every trader’s toolbox.

Is the Market Up, Down or Flat?

When writing a trading system, one of the problems I encounter is determining what kind of market we are in, and what kind of market the software will do well in. Some trading systems do well in smooth trending markets, but do poorly in volatile markets, and other systems need the volatility to do their work.

Author Van Tharp divides market types into several different categories of bear, bull or neutral, and then a volatility qualifier for each one.  He uses a 100-day period of the S&P 500 to determine the market type.  Tharp explains his system and the reasons behind it in several of his books.

An article from the March 2013 issue of Futures magazine gives a different approach.  The author, Billy Williams, explains how to use a 20-period moving average and a 40-period moving average in his article “What’s your market type?”  In his method, he uses the weekly charts of the financial instrument he is trading to determine its market type, with these rules:

  • When the 20-period moving average is above the 40, the market is bullish.
  • When the 40-period moving average is about the 20, the market is bearish.
  • When there’s no clear winner, the market is going sideways.

Other authors I have read use different tools such as pivot points and average true range to determine the general market type, before making trading decisions.  The point is that determining the market type is critical information.  You need to know what kind of market you are in before you do anything else.

Once you determine the general market type you are in, you can make better decisions in your code; if your trading system works really well in trending markets, then you can decide to sit out a volatile period, or perhaps switch strategies until the market returns to a trending state.

Take a look at Van Tharp’s market classification system and Billy Williams’ market type rules, and use these in your code.  Both Tharp’s and Williams’ systems are easily coded, using simple math to make their classifications.