Optuma Testing Errors

Quantitative Testing is one of the most important things we can do as Technical Analysts. It allows us to have a degree of confidence about the strategies we put into place.

As a company, building quality testing tools is one of our highest priorities because we see that it causes many people so many problems. Did you know starting with a Back Test is one of the worst things you can do? I didn’t for many years. I had no idea how I was sabotaging my ideas through flawed testing.

Because this is an important issue, we are putting together a course on all of the testing tools in Optuma and how they can be used. I will be teaching on the theory of the testing methods and explaining how you can implement them in Optuma. More on that in a moment.

The other thing that I am doing is thoroughly checking everything. Optuma has been developed mainly by our clients telling us what they want to see. In the early days, our partners would tell us the formulas to use, and we would take their word for it. Unfortunately, some of what we were told was not correct. Because of that, and because I am going to teach on this, I want to be 100% sure of all the calculations that are in the testers.

There are a couple of items which I have already found that Optuma users need to be aware of right now (all of these items will be fixed in the next major update - Optuma 1.4):

• Annualized Returns in Back Tester and Signal Tester were using a wrong calculation. Monthly returns were multiplied by 12 rather than being compounded.
• Monte-Carlo analysis in the Back Tester was diminishing the results of the sample distribution which gave us erroneous p-values. It made poor strategies look more significant than they were. In fixing this I also took the opportunity to make it even faster!
• Back Tester only starts reporting results from the first trade taken rather than the beginning of the test period (I need to know if my strategy takes two years to make a trade!). This is not an error, but it does make it hard to compare strategies.
• Short strategies in Back Tester work, but using margin on them produces erroneous results. Do not rely on these. As of writing I still have not addressed this.
• Back Tester will not take a Short position if a Long position is open.
• Trade Tester - a new version of Signal Testing that we introduced in Optuma 1.3 is inflating results making strategies look sensational. That is because the results at the end of the test period are only considering the trades that had not closed out yet. Those are the handful of trades that obviously were doing great because they had not triggered an exit.

The first image shows the inflated results. That would be nice, but as they say, “if it’s too good to be true…….”

Optuma Trade Tester

The solution was to keep all closed trades in the calculations of the averages. You can see how this is more realistic here.

Optuma Trade Tester

These are the issues that I have found (with some help through great questions from others using these tools). This is a major priority for me and we are getting all of these items fixed asap (in fact most of them are already done and are just waiting to be included in an update).

Testing Course

As I mentioned earlier, I am working on a significant course on testing that will cover all the different ways that we can test, and the theories behind them. As an introduction to testing, I have recorded a presentation that I have done at a number of Technical Analysis societies around the world. The overview session - available now - will give you an idea of how the material is presented. You can read more about that by clicking on the image below. If you are interested in this video course, it is on sale right now while we are in the building phase.