We’re bringing it back! Every once in a while I will rant on the concepts and ideas behind what numbers suggest in a series called Behind the Numbers, as a tip of the hat to the website that brought me into hockey analytics: Behind the Net. My ramblings will look at the theory and philosophy behind analytics and their applications given what is already publicly known, keeping my job safe while still getting to interact with the public hockeysphere.

I’m back and here to ramble on things like models, sheltering, and environmental impacts on the results we measure.

Let’s set the stage with some basics of analytics and the theory behind it before we look at how nuance needs to be observed in making assumptions from all statistics.

Results-oriented evaluation and decision making matters. The game of hockey is decided by the outcome of events; the team with the most goals wins. By measuring those events, correlating events, and events that are part of the process, we can be better informed on what is going on.

All of these events are a combination of multiple factors. They are the outcomes of players’ actions, abilities, decision making, and effort, but also environmental factors such as quality of linemates, competition, shift starts, systems, chemistry, life, psychology, etc.

I believe the numbers and models we currently have do a very good job in removing some of these environmental variables with how they tend to impact players. However, I also believe there is a range to how environmental variables impact each specific player. Players’ reactions may change, or they may make different decisions when pushed into tougher or easier minutes.

You don’t have to take just my word for it. Micah McCurdy made a similar rant on Twitter.

Regression methods excel at removing environmental impacts and regressing impacts towards average (it’s in the name!). Currently, public models do a decent job in factoring in the majority of the impact on things like shot quantity, shot quality, zone starts, linemates, linematching, rest, and leverage.

That is not everything. Still, I want to stress that models still are really good… just not perfect.

Thought experiment:

You’re a NHL defender (Player A) and halfway through the season your defensive partner changes (B->C), but nothing else. The new partner (C) is worse than the one you had previously (B) and as expected the results with you on the ice trend downward as well. Regression models know previously how you performed with your last partner (A+B), how the new partner performed previously (C), and how you are performing now (A). It also knows how your former partner (C) is performing now and how your new partner’s old partner (D) is currently performing (this is becoming complicated!).

The model would then assume that the historically weaker player carries the lion’s share of blame for your (A) poorer performance.

Now, let’s say the results are worse than what the model, and what one reasonably should, initially expect. How does that influence things?

The model will assume it was slightly off in it’s prediction on how good you, your former partner, and new partner was and make adjustments on everyone. But maybe that’s wrong. Maybe there’s more to it.

I believe there likely is.

One missing variable is chemistry.

Now, I often dislike how chemistry is used in hockey, by the industry, media, and fans alike, but what I call “true chemistry” is very real.

We know that players are not equally adept at all parts of the game. Player B may be better than Player C overall, but Player C may still have areas where they are superior.

Moving back to our thought experiment. Perhaps you (A) are more of a so-called “defensive defender.” You do well in your defensive zone assignments, excel in board battles, and can easily strip the puck from the opponent, but you’re only a middling puck handler. Your old partner (B) was not just better than your current partner (C), but also had a different skill set. The former was highly adept at handling and moving the puck, while the latter individual is similarly defensive but an even worse puck handler.

So now the bulk of the weight in carrying the puck out of the defensive zone and starting the breakout lies on your shoulders, where before you could leave that responsibility to your old partner.

Results are not just different because of the overall quality difference in skaters, but also because your decisions (Player A) have changed.

Results aren’t strictly about abilities, but those abilities mixed with the decisions players make.

As an aside: this is a thought experiment only. Please do not think that this means teams should default to having defensive pairings where one is defensive and the other is a puck mover. Sometimes the difference in overall player ability is still what matters most, not “chemistry.” Also, the various different skill sets of players, roles, and how they interact is much, much more complex than “puck mover” and “safe and defensive.”

These are simply tropes I’m using to illuminate how “chemistry” can be an impact on results aside from what current public regression models already observe. This impact is something we’ve already seen and studied in cursory in the public domain.

The theory of chemistry: players having differing skill sets produces outcomes better or worse with different linemates than one would expect if overall talent was strictly the only variable.

These differing skill sets for players impact more than just chemistry. The impact of environmental factors is messy when mixing with player abilities and skill sets. The human effect, chemistry, and environmental factors can compound in non-linear and confounding manners.

As one NHL executive said to me “there’s a million things going on and you can’t possibly get everything.”

Currently public models do a decent job in telling you how well a team performed with a player given their role, but that does not mean it was the optimal role for the player. Nor can all models compensate for the fact that not all variables impact all players in the same manner.

Rather than going over multiple thought experiments, I shall merely raise some further questions:

• Should we expect all players to equally struggle mentally when moving from a more sheltered to a more exposed role?
• Should we expect all differing skill sets to equally struggle when moving from a more sheltered to a more exposed role?
• Should we expect a pairing of two “defensive defenders” to equally struggle when moving from a more sheltered to a more exposed role than a “defensive defender” paired with a “puck mover?”
• Should we expect all players to equally deal with the pressure of leaping up in the depth charts or experiencing greater leverage deployment?
• Should we expect all forwards to improve equally when playing with a superior defensive group behind them?
• Should we expect all defenders to improve equally when playing with a superior forward group in front of them?
• Should we expect all players to react the same as they start noticing performance regression with age?
• Should we expect all defenders of equal level react to a decrease in sheltering on a team with strong forwards top-to-bottom vs a team with weak forwards top-to-bottom?