Evaluating a Manager's Skill vs Luck

Updated: Mar 2, 2020

We just completed our semi-annual Investment Team “cross-sectional review” of our investment managers in which we individually and collectively assess the potential of our active managers to add value in the future. It’s a chance to recalibrate our views on manager performance potential in a systematic and measurable way. If you care to dive into the details, our letter of February 2018 outlined the processes and procedures we use—send us a note if you would like to see that again. In this letter, however, we wish to outline the hard logic behind luck and skill, and the inescapable statistics of assembling a winning portfolio. In our experience, our ability to create portfolios that truly add value on a risk-adjusted basis is sustainable only because we remain mindful of the hard logic of luck and skill—to wit, always be prepared to deal with the fickle moods of chance.

The vagaries of luck. The ability of any manager to add value is always a combination of skill and luck. This simple statement bears some disassembling. “Adding value” can be hard to define. For example, a long‐only manager that seeks to beat the MSCI EAFE Index can add value simply by returning more than the index. But the manager may also be adding value by matching the performance of the index with less volatility. How about a hedge fund? A hedge fund may add value by achieving a return of, say, 8% if that manager did so with no stock market exposure and no correlation with the broader market index. Let’s assume that we can measure “adding value” through a clever application of precisely the right benchmark (recognizing that in practice this is sometimes easy, and sometimes hard). For the EAFE manager it may simply be beating the EAFE Index; for the hedge fund it may be an absolute return hurdle.

With that in mind, the ability of a manager to add value can be represented by a simple bar chart:

Luck, even for a skillful manager, is much more important than skill over the short term, meaning over a quarter or even a year. There is simply too much variability caused by unforecastable events, such as stock‐specific events, or the news of the day. It stands to reason that over time periods as short as a day, the driver for outperformance for even the most talented manager (excluding high‐frequency traders) is nearly 100% luck. For longer periods, the random nature of short‐term risk tends to cancel itself out, and true manager skill tends to prevail. This will be true both for talented managers and for managers with

weak skills—that is, over the long term, a manager with inferior stock‐selection skills will tend to underperform, as its repeated mistakes inevitably draw down its performance.

We believe the ability of great managers to add value over longer periods (say, a decade) can be attributed to upwards of 80% investment skill—perhaps even 90% (but never 100%). It is true that if you have enough monkeys flipping coins, a certain percentage of them (actually about 0.098%) will flip 10 heads in a row, but that is where experience comes in—with sufficient ongoing due diligence (i.e., interviews, analysis, and reference checking) one can separate the lucky monkeys from the superior investment minds. We believe that great managers have added value over long time periods because they are truly skillful.

So, that begs the question of how do you find these managers that have this wonderful lopsided ability to add value based much more on skill than luck? First, recognize that it is a difficult task—in our crowded universe of managers with great track records there will be some lucky monkeys. There will also be managers whose next 10 years will likely be very different than the last 10 years (due to factors such as growing too large, key people changes, getting complacent, or other afflictions of time and age). So how do you estimate a manager’s ability to add value through skill or luck? Two other insights help set the stage.

All great managers have a well‐calibrated self‐assessment of their own skills. One of our first steps is to figure out the manager’s own assessment of its ability to add value through skill. After all, they are the ones on the inside—they should know better than anyone else their own skill level. One would think so, but funnily enough, managers (especially young managers with great track records) tend to overattribute success to their own skill. Put another way, the 0.098% of lucky monkeys generally think they are brilliant. In order to weed these managers out (and thus separate the lucky from the skillful), it is fair to probe managers concerning their “edge” and how much outperformance they think they can

attribute to skill. Thoughtful managers will have an answer that recognizes the influence of luck, at least over short time periods. The best managers will demonstrate low hubris, providing answers that acknowledge the difficulty of measuring skill and of separating skill from luck. These managers may cite other portfolio factors (such as having more growth or value stocks, or more small‐cap or large‐cap stocks, etc.) as additional contributors to their performance—it is important to acknowledge such factors when assessing manager skill. Now it’s true that there are virtually no managers that under‐assess their own skill, so low hubris is probably a necessary but not sufficient criteria for manager excellence. Ideally,

you want a manager who accurately self‐calibrates. Such managers will use regular feedback loops” that test and assess past decisions. For example, some managers will compare the performance of their larger, “high‐conviction” stock positions to that of their smaller, “low‐conviction” names with the idea that if larger positions really reflect their “best ideas” then the outperformance of these best ideas can be tested. Others, especially small shops, may review their performance less systematically, but may do so through introspection or thoughtful discussions with a peer. It can be difficult to measure this selfcalibration ability, but it remains an essential first step in evaluating a manager’s ability to add value. After all, how can you rely on a manager that has a wildly inaccurate view of their own abilities!

All great manager‐selectors (people like us) also have a well‐calibrated assessment of their own skills. In ways that are similar to understanding manager skill, our manager‐selection skill is subject to some logical conclusions:

  • Our ability to evaluate a manager’s ability to add value through skill is never 100%. You can never know everything about a manager—you always learn a bit more in each successive assessment. Let’s say that given sufficient time, sufficient data points, and sufficient interviews we can determine a manager’s ability to add value through skill with 90% accuracy.

  • Bear in mind that the common case (even with managers with great track records) is that our assessment of their skill differs from their own assessment.

  • Ideally, we find managers that can add value through skill; they have an accurate self‐assessment of their abilities; and our assessment closely matches their assessment. It’s hard to form a successful investment partnership without satisfying these three criteria.

The math works out as follows. To assemble a great portfolio with great managers:

  • We must find very skillful managers whose ability to add value over a long time period is high, say 90%.

  • We perform ongoing, deep due diligence and interview the managers to make this assessment and to understand the manager’s own calibration of that assessment.

  • We create a portfolio recognizing that our own calibration abilities are not perfect. Perhaps given enough time and data points we can get it to 90%.

  • Then, th