I’m a Survivor
For the last couple of years I’ve routinely stated how important dealing with survivorship bias is in testing. I hope you agree with me that it’s a critical issue that makes it nearly impossible to run a historical test and then being able to repeat those results in the future. The only problem has been that it has historically been really hard to run tests without survivorship bias. Well, finally we have the ability to do this built into the next major release of Optuma (due Q2 2018). Normally I’d wait with this, but the testing I’ve been doing has made it so abundantly clear why this is such an important issue and I wanted to share that with you straight away.
First – what is Survivorship Bias?
Survivorship bias or survival bias is the logical error of concentrating on the people or things that made it past some selection process—and overlooking those that did not—typically because of their lack of visibility. This can lead to false conclusions in several different ways.
The initial outworking of this was in World War II when the Engineers were looking at bullet-riddled planes which had returned from missions. They focused on strengthening the armour in the locations that had the most bullet holes. But, these planes were the ones that made it back, so obviously those areas were not critical to the survival of the plane. Instead, the armour should have been added to places where none of the survivors had been hit in the hope that it meant that more planes would survive the mission.
In our application, when we run a back test on equities, we often say that we want to focus our test on the members of a popular index like the S&P500, FTSE100 or ASX200. We collect together the securities that make up that index and do our test on those over the last ten years. The trouble is that members of the S&P500 today are not the same as the members ten years ago. Lehman Bros ring a bell? How can we run a test without including “LEH” in the list?
Similarly, there are a number of current names in the S&P that were not included ten years ago. E.g. Netflix was added to the S&P500 in 2010, so we should not be considering any signals in Netflix before 2010 when it was not part of the index.
Both these conditions—ignoring companies that are no longer in the index and including those that had not yet “made it”—leads to survivorship bias which skews our tests positively.
Now for the results
Ok, so now that that is out of the way, let’s have a look at this in real life. Following is a simple test of a 50-period moving average crossing above a 200-period moving average run over the last ten years. Here is the script:
MA(BARS=50) CrossesAbove MA(BARS=200)
Remember in this chart the Blue shaded plot is our equity from the test. Obviously not a lot of alpha, but it shows a moderate return over the index (red line). The issue is that we have only used the current 504 stocks in our test. We need to set this up to include all the companies that were ever in the index. Not only that, but we also need to adjust our script so we tell Optuma to only take a signal if the company was in the index when we got our signal.
In the properties of the test I need to change the new “Membership” property from “Current” to “Historical”. Note that this will only show for Optuma Symbol Lists where we have set up the survivorship data. At this stage, all we are doing is including all the 700(ish) companies that were in the S&P 500 over the last ten years.
To make sure that we only take a signal when the company was actually in the Index, I need to update my script to add the new “IsMember()” function. Note that we don’t change the exit script since we want to exit regardless of membership in the index (although you could exit on removal from the index if you wanted to).
Here is that script:
MA(BARS=50) CrossesAbove MA(BARS=200) AND IsMember()
Suddenly this does not look so good anymore. Our idea did not “beat” the market. Anyone who has ever tried trading a MA crossover like this knows that it’s a great strategy in theory, but the results are really hard to replicate. Finally, the tests are telling us what we already know.
The positive side of this is for those who are working on Short or Long/Short strategies. Removing survivorship bias gives a “lift” to Short results.
The main point of this is to highlight to you how important survivorship bias is, and to ensure that you don’t ignore it in testing. If you have ever been frustrated by your inability to repeat test results in real-life, then this will help you see why that has happened.
A simple rule of thumb, for when you don’t have access to correct survivorship bias-free data, is to subtract around 3% per annum from your results. That will give you a better idea of what you can expect. Just don’t plan your trading strategy by only looking at the survivors. Make sure you properly consider the securities that didn’t make it.
Note: at present this will only work for S&P500, and only back to 2009. This is due to the large manual process for setting this up as we have to manually add every entry and exit of an equity in the index. Now that we have the infrastructure built into Optuma, we can get more data into the S&P500, and also start to add other indexes as we get the history. We’ll release a “how-to” when the update enters beta testing for anyone who wants to use this in their tests.
Mathew Verdouw, CMT, CFTe
CEO / Founder Optuma
As a Computer Systems Engineer, Mathew started Market Analyst (now Optuma) within 18 months of completing his degree. From that point on, Mathew has made it his mission to build the very best software tools available.
Since 1996 Mathew has been learning about all aspects of financial analysis, and in 2014 earned the CMT designation (Chartered Market Technician). In 2015, he was also awarded the CFTe designation. In 2017, Mathew started to teach the required content for the CMT exams at learn.optuma.com. He is the only person in the world who teaches all three levels due to his broad exposure to all forms of financial analysis.
As someone who has dedicated his life to find better ways to analyse financial markets, Mathew is set to drive innovation in this sector for many years to come.