How To Know If A Trading System Will Likely Continue To Work In The Future

I've seen plenty a great looking, back-tested trading system turn into a big, steaming pile of horse manure when traded with real money...

So, let's drop some SCIENCE on the blog and look into a formula for calculating if a trading system will likely continue to work into the future!

More...

what are the basics of a good Trading System?

Elements Of A Good System Design:

  • Makes lots of money!
  • Small drawdowns, losing money is bad!
  • Lots of samples
  • Out of sample testing looks like in sample testing
  • Factor in commission and slippage
  • System trades a with lower number of shares in volatile markets

Trading system design is an exercise in minimization.  

You have to have certain minimums thresholds or you completely toss out the system! 

Here are my minimums for my trading models:

Minimum Requirements For A Robust System:

  • Minimum 100 trades
  • At least 10 years of data
  • Statistical significance of 30 ( profit factor * number of samples ^ 0.5 )
  • 20% of data used for "out of sample" (OOS) testing
  • OOS Profit Factor  / In Sample Profit Factor >= 70%
  • Net profit / Max drawdown >= 10
  • Profit factor >= 2

During system development, if anyone of these criteria are not met, it's onto the next idea.

But if a system does pass the first volley of tests; I then rank them according to this formula:

System Score = Profit Factor * Number Of Trades ^ ( 0.5 )  * ( Net Profit / Max Drawdown ) * ( Out Of Sample Profit Factor / In Sample Profit Factor )

The first part of the equation is all about statistical significance taken from signal processing; we want this value to be above 30 roughly:

Profit Factor * Number Of Trades ^ ( 0.5 ) 

Profit factor is basically the measure of signal to noise ratio. 

Profit factor explained.

If you have a lot of signal, but a small number of samples, you might have some significance.

You can also have significance with a low amount of signal and a large number of samples.

Again, you want this value to be somewhere above the 30 mark.

I want to get to my trading Goal ASAP!

I don't want to lose $800,000 before going on to make $1,250,000.

Thus, we want to divide our net profit by the worst drawdown.

( Net Profit / Max Drawdown )

Making this number a ratio gets rid of raw profit.  

If we don't do this, a person risking 3% per trade would seemingly have better numbers than a trader risking 1% per trade.  

Amateurs look at raw profits only.

I want to reward a system that has matching (or better) parts

Next comes the ratio between the out of sample (OOS) and in sample profit factors.

I want a system that has similar (if not better) profit factors on the data it has "seen" than the data it has not "seen" aka:

( Out Of Sample Profit Factor / In Sample Profit Factor )

It's a beautiful thing when this ratio is greater than one, because then the OOS data profit factor is greater than the in sample profit factor.

Awesome.

Let's look at a few examples:

The S&P 500 Swing trading system:

Trading system on the S&P 500 with a mean-reverting system

Trading the S&P 500 with a mean-reverting system

In 2012, when I wrote the computer code for this system, I used the Emini S&P 500 futures contract data.

Futures contracts are traded almost around the clock and are the best source of data for trading the general stock market.

All the data before the purple line is the data I used to create this system.

Between the purple and blue lines is the data I ran the system over that it had not seen yet, the out of sample data - the green equity curve keeps going up as you can see.

Then the master stroke, everything after the blue line is REAL TIME trading.

I can't overstate that enough.

All three sections look identical to each other; this is exactly what you want to see in your trading.

Now, let's plug and chug the values into our ranking equation above:

Profit Factor * ( ( Number Of Trades ) ^ ( 0.5 ) ) * ( Net Profit / Max Drawdown ) * ( Out Of Sample Profit Factor / In Sample Profit Factor ) -->

4.17 * ( ( 306 ) ^ (0.5) )  * ( 505,000 / 14,000 ) * ( 3.5 / 3.8 ) =  2423.5

Now, that is a BIG mathematical: "OMG I want to use this system!"

The Gold Trend Following System:

Trading system used on gold with a trend following system

Trading gold with a trend following system

You can see the huge difference between trading the S&P 500 and gold immediately.

This is due to the fundamental way each market works internally.

The S&P 500 is a mean-reverting market and gold is a trending market.

You must use the correct trading method (mean-reverting trading or trend following ) with the right market.

(i.e. a mean-reverting trading system does not work with a trend following market)

You'll also note that the green equity curve for trading gold looks choppy, not as nice and clean as the S&P 500 system.

This again is due to the trending nature of gold; you have to put up with a lot of little losses while waiting to catch the massive trends higher. 

Let's run the numbers and see how gold trading compares to S&P 500 trading (I'm sure you already know it's going to rank lower just by looking at that chart).

Profit Factor * ( ( Number Of Trades ) ^ ( 0.5 ) ) * ( Net Profit / Max Drawdown ) * ( Out Of Sample Profit Factor / In Sample Profit Factor ) -->

2.8 * ( (109 ) ^ (0.5) )  * ( 457,000 / 25,000 ) * ( 2.8 / 3 ) = 498

This is a mathematical: "Damn, it's hard to trade gold, but still statistically worth it."

Pretty large scoring difference between the two trading systems.

Nonetheless, any system that scores above 400 is still worth trading.

But why trade gold in the first place?

Trading different, non-correlated asset classes (Stocks, Gold, Oil, etc.) smooths out your portfolio growth over time.

When one system is zigging, the other is zagging.

The only real Holy Grail in trading is the use of multiple systems trading different asset classes.

And now we have a scientific way to measure systems against each other.

Use the above equation on your own systems and see how they stack up against each other.

You can use the equation on any system over any time period and on any time-frame (like day-trading systems).

Conclusion: Good Trading System Design

  • Good systems make lots of money in the real world
  • Have large net-profit to drawdowns ratios
  • Have lots of trade samples
  • Out of sample testing looks like in sample testing
  • Have commission and slippage factored in
  • Trading different, non-correlated asset classes (Stocks, Gold, Oil, etc.) smooths out your portfolio growth over time

  • Trade with a lower number of shares in volatile times (risk management)

Let me know how your systems score in the comments below.

Remember: IF YOU CAN'T TEST IT, DON'T TRADE IT!

About the Author

Hello, I'm Kurt Bruch and I'm the Relaxed Trader guy writing most of the stuff on this website. Feel free to ask questions or leave comments on my posts. I love talking to fellow traders. Remember if you can't test it you shouldn't trade it!

Leave a Reply 0 comments

>