Trendiness / Choppiness Index
It has long been my contention that markets have become less kind towards trend following models in recent years. My view is that markets do change and that models therefore need constant adaption to survive, witness the recent demise of JW Henry and the dreadful performance in general of trend following models over the last two years.
One concept that I have often admired but never tested is that of Perry Kaufman’s Efficiency Ration to be found on page 134 and onwards of his book “Smarter Trading” which can be found in the Listings section here at Traders’ Place.
A trend follower’s wants and needs to avoid false signals due to noise. A trend following model will reap the greatest and cleanest profits when a market breaks out in one direction and never looks back – a perfect (non-achievable!) trend would go from point A at the bottom left hand corner of the chart to point B at the top right hand corner in an absolutely straight line with no retracements.
Kaufman’s Efficiency Ratio has values ranging from 0 when markets are very noisy and a theoretical +1 when markets are perfectly directional.
For a given day, Kaufman’s ratio is calculated as: the absolute value of net price change over time (e.g. 120 days) / the sum of the absolute value of all day to day price ranges over the same time period. You can readily see that if a price goes smoothly upwards from day to day with no retracement (ever!) the index for any given day will equate to 1.
I based my code on true range rather than close to close price change (arguably not what Kaufman intended). I ran the code over an entire portfolio of over 100 instruments from 1970 to date. Each day I added each individual instrument efficiency ratio for the whole portfolio and divided it by the number of instruments for which I had a price on that day.
While it has taken me a few hours fiddling about coding this up today, I by no means expect my code to be bug free and may need to confess to having to make an amendment or two in the future. Nonetheless, for what it is worth, see the chart set out below for the combined Efficiency Ratio at portfolio level on a daily basis. Arguably, an average (perhaps 10 days) might have made the trend more evident but the chart is probably clear enough as it stands.
It would seem to indicate that indeed markets as a whole have become less trend friendly/ more noise over time and that the last two years look particularly “inefficient” and trendless on the 120 day calculation period I used. I could of course (and probably shall) calculate the aggregate trendiness indicator for different periods: 1, 3, 12 months for instance. But this is an interesting start.
I really do not know what to what uses such an indicator may be put. In his book, Kaufmann uses it in combination with his adaptive moving average to vary the speed - a faster moving average in clean markets, a slower moving average where noise dictates wider stops and the need to keep out of the way of random retracements.
I suspect that like all indicators, this one is very useful with hindsight - in describing the state of the market in historical terms. As can be seen from the chart, according to this index, 6 month trends have been historically very noisy these past two years. Confirming what we have seen in actual trading results.
My suspicion is that the index has as little predictive ability as any other but as you say, you could certainly see what happens (in back testing at least) if you use it to rank the order in which new entries are taken. This would of course only be relevant where you operate sector and or overll portfolio risk limits.
Looking at an expanded version of the chart, what it shows is that the efficiency of trends (at least 6 month trends as defined by this index) improved to well above average between mid 2008 and 2009 and then slumped to an historic low for much of the period since.
As with all indicators I have come across, they are great at giving historical perspective and putting the past into context. We can see that the past couple of years have been horrible for trend following by the briefest perusal of CTA results for those years. And this indicator tells us why this has been the case (in technical terms at least).
What this indicator does not do is to allow us to predict whether the next day, month or year will be any kinder to a trend following strategy. It shows that trends have become less efficient year by year over the past 40 years but does not tell us what will happen over the next 40.
It's a very interesting concept. As I understand it, you've measured a 120 days rolling period since 1970. It would be interesting to to measure and compare with other rolling periods such as 20, 50, 90, 150 and 200 days as to check if we can infere anything about the properties of "trendiness" and "market noise" in the portfolio.
Itamar, yes that is exactly what I have done. The only reason I have not yet posted for other rolling periods is that I am not quite satisfied yet with my priming alignment in coding. I believe I am probably as much as two or three days out of sync which does not matter with a 120 day period but becomes much more crucial obviously as you reduce the number of days in the rolling period. I'll thoroughly check it out in the morning, make any necessary coding corrections and post some further results. Incidentally, thanks for the comment on resolution and sizing for images - I posted the same chart on the Trading Blox Forum and it was easily readable and decipherable but as you can see, Andreas has improvements high on his priority list.
Thanks for your interest in this topic. I can't believe I have not bothered to code this indicator over the past decade. I have owned Kafman's books fo a long time and this efficiency ratio of his makes perfect sense. To me, it is the best definition of noise/choppiness and by contrast good clean trends I have ever come accross.
Perhaps I should have phrased that slightly differently but what I meant is that Andreas and I really want to build this site into a place where professionals genuinely share ideas and discuss their methods and to that extent if you felt able to share your other idea with this community, we would be very grateful.
I would very much appreciate it if you would be kind enough to share your other idea with us!
Anthony, I didn't want to take away from the topic of this thread but I'm happy to share. It's actually not my idea but one found by doing an internet search. I've found that if you know the right questions to ask the answers just pop right out. I don't always know the right questions but in this case I seem to have stumbled onto a fairly good idea. It is interesting to note that as I've been searching for this topic your thread also popped up on the radar and so I had a chance to read the concept from Kaufaman.
How to measure smoothness of a time series in R
Pay particular attention to the first answer. Essentially it is the coefficient of variation, in this case differenced. The basic form is Std Dev / Mean, and the form presented is done with the differences: sd(diff(x))/abs(mean(diff(x)))
The explanation seems clear enough on the answer. This formula seems to have some favorable properties with regards to finding a solution to my optimization problem with regards to combining multiple symbols into a basket.
Thank you so much Itamar - I had intended to post the images myself but you have done a much better job.
I must say however that the results of my further testing with this index has given me cause for considerable concern over traditional trend following as practiced for 30+ years by the likes of Dunn, Henry and the rest of the crew. Each of these rolling periods shows a marked trend of deterioration over the 40+ years of the test. By these measures (and I feel this index is a very good one - thank you Mr Kaufman) trends have indeed become less efficient and much more noisy on a consistent downward trend since 1970 with the most marked deterioration being in the past two years. No doubt things will improve somewhat but the nagging question remains that the glory days for this kind of strategy may be behind us. I hope I am wrong.
Indeed, all of them show a channel downtrend at the least. I wonder which might be the reasons and the possible conclusions if any:
1. 1972 ~ 1978 were the "paradise" years for traders?
2. Central Branks have been successfully taming volatility and inflation?
3. Unintended consequences of the ever growing number of "arbitrators" (traders) in the markets, the proliferation of derivatives and computerized or quantitative trading?
With regard to the nagging question of markets possibly becoming less and less benign to "trend following" I wonder if this allows us to conclude that they are becoming more and more benign to some other strategies, such as "fading the trend" or "HFT".
All questions to which I would very much like an answer but can provide none. Continued research I guess may reveal some pointers but I do indeed wonder what other stragies one may be able to devise (or whether there are ways to improve trend following strategies). Or am I just mistaken and making a fuss about nothing?