Cesar’s First Rule
May 20, 2018

Much of what I know about the practical aspects of backtesting I learned from two of my colleagues at Connors Research, Cesar Alvarez and David Weilmunster. One particularly important lesson was Cesar’s observation that “if the results seem too good to be true, they probably are”. This situation comes up often enough that I now think of it as “Cesar’s First Rule”.

Not long ago, a client asked me to verify some very strong backtesting results produced by another researcher. I began by looking through the other developer’s AmiBroker AFL code, but didn’t immediately find any errors. However, as any of you who’ve done a significant amount of coding are aware, it can be challenging to dive into someone else’s work and fully grasp all the nuances.

Next I started to rewrite the strategy using my own coding templates, and sure enough, my results were not nearly as good as those from the other developer. It turned out that he was always exiting at the profit target price, even when the price had declined and the trade should have resulted in a stop loss! Since AmiBroker’s default behavior is to restrict the price based on the high and low of the exit bar, some trades still came through as losers  but with a smaller loss than they should have. This had the effect of masking the problem, because almost anyone would have recognized that 100% winners must be an error.

Sometimes even my own results are too good to be true. Recently I was working on a presentation about building adaptive strategies: quantified trading systems that use different rule parameters depending on the current market condition. I was quite pleased when the adaptive version of the strategy outperformed every variation of the static-parameter version over the same time period (2013-2016). But then, Cesar’s first rule started nagging at me. I checked all my code, and all the adaptive parameters, but everything looked OK. Finally I realized that I had selected the adaptive parameters using results from the current time period (2013-2016), which is basically “cheating” by looking into the future. When I selected the parameters based on past results (2000-2012) as I should have done, my adaptive strategy did not perform nearly as well.

Everyone wants to find their own silver bullet of trading. We all hope that we have the ability to make a brilliant discovery. The cold, hard truth is that in most cases if the results seem too good to be true, then there’s a mistake lurking somewhere. Thanks for the lesson, Cesar!

Copyright 2024 Quant Alpha – ‘Quant Alpha’ and ‘Quantive Alpha’ are Trade Marks of Quantive Alpha – all rights reserved.

All the information contained on this website is general in nature and does not constitute personal or investment advice. Quant Alpha produces algorithms and software only and does not trade or arrange any trading on your behalf. Quant Alpha will not accept liability for any loss or damage, including without limitation, any loss which may arise directly or indirectly from the use of, or reliance on: its algorithms; the information on this site; or information provided by its managers, partners or affiliates. You should seek independent financial advice and conduct your due diligence prior to acquiring any Quant Alpha technology. Quant Alpha is neither a registered investment advisor nor an investment advisory service and does not provide any recommendations to buy or sell particular financial products. 
Before engaging in any trading activities, you should understand the nature and extent of your rights and obligations and be aware of the risks involved. Don’t trade with money you can’t afford to lose. Your trading and investing decisions are entirely your own responsibility. All securities and financial product transactions involve risks. Where Quant Alpha provides hypothetical representations of what the technology has achieved in the past, this has been done with the greatest know-how, data and expert technology that is available, but still, Quant Alpha cannot guarantee that these results have any likelihood whatsoever of being achieved in future. Where records have been provided of how the software has performed on management’s own accounts, whilst these are an accurate and true record of what has taken place in the past, they are not necessarily indicative of future results – the future is as unknown to Quant Alpha management as it is to anyone else. The past performance of any trading system or methodology is not necessarily indicative of future results.