A common retort that many in the hockey analytics community have gotten is: “have you ever played the game?”

The insinuation, of course, is that if you haven’t played hockey at a high level, let alone in the NHL, then you can’t possibly understand the game. Certainly not as well as those who have. And that when it comes to evaluating players or making decisions on how best to improve a hockey team, the former players and lifelong hockey men that populate the league’s front offices can always fall back on their instinct for the game in ways that no one else can.

But let’s talk about relying on your gut instincts to make decisions.

What Is Intuition?

The notion that you can trust your instincts on a topic is a powerful one with a deep-seated origin in our evolutionary history. Our primitive brains developed in an environment where the ability to instinctively recognize and react to danger meant the difference between life and death. So it’s natural to accept that these innate abilities that helped our pre-historic ancestors survive have now evolved to help modern leaders make better decisions. Research has shown that there can be a biological reaction in response to environmental cues that are otherwise imperceptible; however, situations where gut instincts can drive better decisions are rare.

Before we go any further, it’s important to distinguish between instinct and intuition. The two are often used interchangeably, but there is a slight difference. Instincts are hard-wired and instantaneous physical reactions to something in your environment, while intuition is really just a summation of your accumulated experience. Both can be used to inform decisions, but when we talk about gut instincts, we are usually talking about intuition. Certainly that is what the “but have you even played the game?” retorts are based on.

In hockey, instincts are what someone who actually has played the game might rely on out on the ice. The problem is that same former player will be relying mainly on intuition, not instincts, when he is inevitably hired into a front-office position after he retires. Or sooner.

So, our real interest here is in exploring the role of intuition in guiding decision-making in the NHL.

Conditions for Development of Intuition

In a 2009 article published in American Psychologist, Gary Klein and Daniel Kahneman conclude that there are two conditions necessary to create an environment where expert intuition can lead to better decision-making: 1) it needs to be a “high-validity” environment with stable links between observable cues and subsequent outcomes, and 2) you need direct feedback on the quality of a decision after it has been made, i.e. you need the opportunity to learn the relevant cues. As Klein and Kahneman put it, if a situation provides “valid cues and good feedback, skill and expert intuition will eventually develop in individuals of sufficient talent.”

The question that arises is whether the world of professional hockey meets these two conditions. To examine this, let’s go back to the examples that Klein and Kahneman use to describe a “high-validity” environment:

[It] is very likely that there are early indications that a building is about to collapse in a fire or that an infant will soon show obvious symptoms of infection. On the other hand, it is unlikely that there is publicly available information that could be used to predict how well a particular stock will do—if such valid information existed, the price of the stock would already reflect it. Thus, we have more reason to trust the intuition of an experienced fireground commander about the stability of a building, or the intuitions of a nurse about an infant, than to trust the intuitions of a trader about a stock.

Applying this to a hockey context (or any sport), it can be argued that with enough time and effort a skilled observer could learn to identify the observable cues that point to future success for individual players. However, professional sports and the development of athletes are very complex systems with multiple variables overlapping and interacting to produce the end result. Human cognitive capacity is not necessarily well equipped to differentiate between valid and invalid relationships between observations and outcomes. In addition, most roles in professional sports do not provide sufficient opportunity to truly learn and develop intuition, which requires “prolonged practice and feedback that is both rapid and unequivocal”.

The exception here may be in coaching, where decisions on tactics and deployment provide immediate feedback on their efficacy. On the other hand, this is not the case for scouting staff or front office roles, where decisions impact results over much longer time frames. In these roles, it will be much more difficult to develop reliable intuition, and Klein and Kahneman point specifically to an example involving professional sports when discussing imperfect intuition:

[In Moneyball, Michael Lewis] described the weaknesses in the ability of baseball scouts and managers to judge the capabilities, contributions, and potential of players. Despite ample opportunities to acquire judgment skill, scouts and managers were often insensitive to important variables and overly influenced by such factors as the player’s appearance—a clear case of prediction by representativeness.

That is not to say that there aren’t scouts or hockey executives that have put in enough time and effort to develop an intuitive ability to identify top quality prospects. But this is going to be more the exception than the rule. And it took more than simply having played the game to get there.

Whether they ever did play the game at a high level is not only irrelevant to their skill as talent evaluators, but it could even be a hindrance. Experience as a player is more likely to increase the likelihood of their own personal experiences introducing biases in their judgment and ability to differentiate between valid and invalid links with regards to observations and results.

The distinction between a hockey scout and front office manager serves to highlight another important factor when considering the merits of relying on intuitive decision-making. At its core, intuition is very much a form of pattern recognition or memory recall. This means that skilled intuition does not translate to different contexts: it is specific to the training environment. For example, a firefighter that can rely on his intuition to know when a house is about to collapse will not necessarily have the same abilities when fighting a fire in a high-rise apartment building.

This concept also applies to hockey. A scout that has become proficient at identifying junior age players that might succeed at the NHL level will not have the same abilities to identify professional players that might make good additions to an NHL team. Not only are they two different contexts, one could argue that hockey play at the professional level does not meet the requirements for a high-validity environment. But even if it did, the intuition developed at the amateur level does not translate. You can see this in action on a team like the Vancouver Canucks, where the General Manager comes from a scouting background. The result has been that Canucks have had some success at the amateur level, but have failed miserably in their professional scouting and talent evaluation.

This same problem is exactly why having “played the game” is actually a big detriment to developing valid intuition. There is nothing to indicate that being able to play hockey at a high level should offer any kind of advantage to evaluating players or making sound decisions. If anything, those personal experiences as a player are more likely to create biases that impair judgement. Being a former player won’t give you a head start, other than opening a few doors, that is. You would still need to make the most of these opportunities to build up the observation and evaluation experience necessary to develop reliable intuition.

The Problem of Confidence

Another aspect of having played the game that can hinder the development of valid intuition is one of over-confidence in decision-making abilities. In some ways this derives out of the “appeal to authority” inherent in the assumption that just because someone can play the game, they can also identify the characteristics that it takes to succeed in the NHL in others. With this fallacy as the starting point, it’s just a short jump to subjective confidence in the strength of their intuitive abilities. But there is no link between subjective confidence in judgments and the validity of those judgments.

Thus, while we all use intuition to make decisions all the time, it is very difficult for anyone, even experts, to recognize when our intuition is actually valid and when it is not. Even when evaluating the results of a decision, it is possible to make the right decision on faulty intuition. So the problem with trusting subjective confidence is that it can quickly lead to overconfidence through nothing more than a little bit of luck. As Kahneman puts it:

“[L]ucky risk takers use hindsight to reinforce their feeling that their gut is very wise. Hindsight also reinforces others’ trust in that individual’s gut. That’s one of the real dangers of leader selection in many organizations: leaders are selected for overconfidence. We associate leadership with decisiveness. That perception of leadership pushes people to make decisions fairly quickly, lest they be seen as dithering and indecisive.

So what’s the alternative? How can decision-makers guard against the tendency to be overconfident about their intuition and gut feelings? Well, Kahneman and Klein given some examples of professionals that are actually pretty good at staying within their area of expertise, i.e. they know the bounds within which their intuition is valid, and when to call on an outside perspective:

Weather forecasters, engineers, and logistics specialists typically resist requests to make judgments about matters that fall outside their area of competence. People in professions marked by standard methods, clear feedback, and direct consequences for error appear to appreciate the boundaries of their expertise. These experts know more knowledgeable experts exist.

It is therefore prudent to make a deliberate effort to counter overconfidence and insular perspectives, especially in less certain or structured environments, such as professional sports.

Avoiding the Pitfalls

One way to do this would be to ensure that a wide range of perspectives are provided as input into the decision-making process. And by wide-range, I mean a range of backgrounds and experience; having five former players around a table deciding who to target in free agency doesn’t meet the criteria. Another might be to force a more structured approach to your decision-making processes by adhering to a checklist every time you are making a critical decision under uncertainty. There are any number of decision-making checklists out there that you can adapt and build on, but the key is to question:

• the quality and sources of information you have;
• the assumptions you are making;
• whether the decision outcome was pre-determined;
• if there is group-think going on; and
• can the decision be deferred until you have more data?

Another approach that has been popularized by Klein is that of a pre-mortem. I won’t go into too much detail on this technique here, but in short the idea is to imagine that a decision turned out to be disastrously wrong and then come up with all the reasons why it turned out that way. This can help identify faulty assumptions of gaps in the information going into the decision-making process.

Beyond this, there are other, more analytical approaches, that can help overcome some of the issues discussed above.

Giving Intuition a Boost

Given the difficulty in developing intuition that you can trust (if this is even possible), it’s useful to consider ways that analytical approaches to aid in decision-making.

In a low-validity environment, such as the one in which most NHL decisions are made, the two most common pitfalls for decision-makers are (1) the inability to recognize weak links between observable cues and results, and (2) being unable to discern valid links from misleading links between cause and effect. To make things worse, human cognitive decision-making processes are terrible at remaining consistent. When you mix this inability to separate valid from invalid cues in a noisy environment with inconsistent judgments, the decisions are going to be much less rational than you would expect or desire.

It is in these types of situations that Klein and Kahneman suggest a more analytical approach:

A statistical approach has two crucial advantages over human judgment when available cues are weak and uncertain: Statistical analysis is more likely to identify weakly valid cues, and a prediction algorithm will maintain above-chance accuracy by using such cues consistently.

That last part is important. This is not saying that algorithms will not make accurate predictions, just that they will typically be better than a coin flip. Unfortunately, even that low bar means that under conditions of weak validity and inconsistent cognitive decision-making processes, algorithms will significantly outperform humans. As an aside, algorithms will also outperform humans when validity is very high, largely due to their inherent consistency.

The unwillingness to accept that algorithms are still useful in these low-validity environments despite their limited accuracy is probably the biggest barrier to widespread acceptance. There are certainly others, but that is the one that critics will immediately point to. However, data analysts should resist the temptation to counter this resistance by designing aggressive models that do tremendously well in-sample but fail once tested in the real world (a phenomenon called overfitting). You won’t be serving anyone’s interests by claiming that you can predict the results of a seven-game playoff series in the NHL with an 85% accuracy.

So When Can You Trust Your Intuition?

Whether you have ever played the game or not, we all tend to use intuition to make decisions. The real questions are: when is that intuitive decision-making valid, and what can we do to improve our intuition? Andrew Campbell and Jo Whitehead over at McKinsey have put together a test to help you determine whether you really should be trusting your gut. As they put it:

[I]t is tempting to argue that leaders should never trust their gut: they should make decisions based solely on objective, logical analysis. But this advice overlooks the fact that we can’t get away from the influence of our gut instincts. They influence the way we frame a situation. They influence the options we choose to analyze. They cause us to consult some people and pay less attention to others. They encourage us to collect more data in one area but not in another. They influence the amount of time and effort we put into decisions. In other words, they infiltrate our decision making even when we are trying to be analytical and rational.

In order to combat the biases and emotional responses that will sub-consciously affect your judgment and decision-making, they suggest putting yourself through these four tests:

1. The Familiarity Test: have you frequently experienced identical or similar situations?
2. The Feedback Test: did you get reliable feedback in past situations?
3. The Measured-Emotions Test: are the emotions you have experienced in similar or related situations measured?
4. The Independence Test: are you likely to be influenced by any inappropriate personal interests or attachments?

Failing any one of these tests means you are at risk making a poor decision. You may still make the “right” decision, i.e. one that gives you the best outcome, but your decision-making process is not as rational as you think it is, and is susceptible to influences that your are not aware of. In these situations, a much more structured and rigorous approach to collecting, analyzing, and integrating data and information into the decision-making process is warranted. More on that in a future post.

This is the second installment in a series on applying best-practices in data-driven decision-making to the world of professional hockey. You can find the previous installment here.

Follow me on Twitter: Follow @petbugs13