“Young Hockey Players” by Piotr Alberti, via Wikimedia Commons

Hockey statistical analysis isn’t really capturing all of hockey, or seeking to package it; it’s about getting as close we can to the essence of the thing. All the ideas, conclusions, best practices that we’ve cobbled together over the years give us an approximation of the actions a team, a player, or a fan could make going forward to better grasp the game.

Within this fact lies the greatest bone of contention for the hockey stats crowd, and the frequent refrain of critics who can only chirp from the sidelines. “Have you considered measuring this? Have you considered measuring that? Have you removed the games when the Rangers lacked sufficient compete level? Have you adjusted for Hamburglar’s pre- and post-lifetime gift certificate to McDonald’s?” While some of these adjustments may be worthy, and others utterly ridiculous, “shot quality” has been a persistent critique of the use of all shot attempts.

Admittedly, there are some interesting developments in Ryan Stimson’s work on puck movement, which might shed some light on an area yet explored. Though it’s not necessarily his focus, I think his data can give us an idea of how possession is maintained effectively. The remainder of shot quality, or at least the way it’s being conceptualized, lies in these remaining areas: type of shot, where shot is located on net, screened/tipped/direct/clear-look shot data, shooting talent, and where on the ice the shot is taken from. The former two, according to Gabe Desjardins, didn’t really demonstrate themselves when he came across the data (nor when I asked him a month ago). Shot location has already died a partial death by Desjardins, who found it seems to have minimal impact on save percentage, though he also found a team talent component, to the tune of differences ranging up to 0.7 feet.

Let me put the location stuff to bed the rest of the way.

First off, long-time readers will already recall that Garrett has broached shot quality a few times, both in summation and with regards to goaltending (here and here). I also used shot plots from top and bottom teams to show their similarities in the location scatterplot, and earlier this week used a comparison of supposed coaching “systems” for Bruce Boudreau, Alain Vigneault, and Lindy Ruff to make a similar point. Most damning is that the remaining shot quality proponents in the analytics world have had to shift over to the puck movement research of Stimson and/or await something concrete from Chris Boyle’s efforts or the “high danger” shot work at War on Ice — because there’s just not much left here.

But, because I had the data, and had never seen it presented, let’s look at some league-wide figures for shot location, taken from Greg Sinclair’s stellar Super Shot Search site.

Hmmm, so already there should be some alarm bells going off here. We’re seeing significant differences in the proportion of shots in-close (3-4% change across 60,000 shots), yet the shooting percentage is holding much more characteristic steady progression downward. A bit of this has to do with the counting numbers in this proportion; on average, 0 to 10 foot shots account for around 5,500 of the 60,000 shots. In other words, looking at just 9% of all shots does not seem to bump the major needle (shooting percentage).

Okay, now we’re grabbing 30% of our 60,000, and we’ve improved, but we’re still not seeing the needle move on shooting percentage in a way that matches our proportion of high danger shots. From 2008-09 to 2009-10, 1,200 shots moved out of the high danger area, yet league shooting percentage went down one one-hundredth at evens. Factors like variance and error are knocking around enough to offset league-wide differences of nearly 0.7% in its highest-danger shots — shots which are accounting for 30% of total. With this amount, and this kind of data, that’s a pretty big indicator of why location’s impact gets greatly minimized. Note: for a little more on this, see Eric Tulsky’s piece on scoring chance data.

Put another way, let’s revisit Gabe’s observation that it seems a team might have a demonstrable talent to the tune of 0.7 feet closer on shots-against than other teams. Well, unfortunately, from one year to the next the entire league varies about…0.7 feet!

Whether the impact is recording or league talent, those demonstrable differences might very well be swallowed up by the variance of the league from one year to the next. No matter which way you slice it, the signal does not come through clearly, if at all, for the simple location component of shot quality.**

So that leaves us with puck movement,* tip/screen/clear-look shots, and team shooting talent as the remaining potential components of “shot quality” (unless somebody would like to include stick quality…please don’t). To this end, it’s important to point out that the question is not about whether shot quality exists; it does. But how much of the explanatory pie are those three components fighting over? (Answer: a heavily-regressing, at-maximum sustainable difference of about 1% above or below league-average PDO; Tulsky is also instrumental here). Would any one of the three be willing to say the other two don’t exist? I highly doubt it. But as we slice the pie thinner, we move further away from primary drivers of the team game, and we run the risk of putting too much attention on something that isn’t repeatable or actionable. If we’re returning a more predictive logarithm, are we sure we’re including something we can reasonably add in a real environment?*** If it’s something that is greatly impacted by variance and error, we could easily find ourselves chasing shadows.

It’s all a part of the game of approximation we’re playing in analytics, trying to pick the colors that will paint a portrait that resonates. And shot quality is glitter glue.

—

Postscript

Steve Burtch brought up a good point on the data, drawing attention in particular to the evolution of goaltending.

Here is a quick explanation why the increase in HD shots isn’t driving up SH% at 5v5: pic.twitter.com/rBgPGhiKWY

— Stephen Burtch (@SteveBurtch) May 8, 2015

Ok so the R^2 btwn Goaltender Avg Height and HD SV% from 06-15 is 61%… so… yeah – goaltender’s getting bigger is definitely signfiicant

— Stephen Burtch (@SteveBurtch) May 8, 2015

This adds an important wrinkle to our understanding: while goaltenders are fairly static in their success stopping shots from 20 feet and further, they are improving in the “high danger” zone 20 feet and in. This is the kind of “evolution” that, in retrospect, I should have written more about — though I still contend that we have variance and error in there as well, even if partially muted by goalie evolution. Steve suggests goaltender size is a factor, and certainly there are other factors with regard to parity, technique, and usage improvements. This ends up being another shot quality quandary: try to force in shot in closer? But we’ve just seen from studies in the past (and in the distance segment above) that teams are not really exhibiting that talent very well. Shot volume through improving possession and offensive zone time is the short-term, easier-to-implement answer, but eventually that inefficiency will go away as well. As a league, this will be a difficult issue to address, but for shot quality it’s just another area where pushing for shot quality will bring diminishing returns.

Bruce McCurdy also voiced skepticism about the data from 2008-09 in particular, and I think it’s possible some of the data skews to greater differences in the proportion of shots at distance. At the same time, keep in mind the curve Burtch shared above: we do know that league-wide shooting percentage (including high danger shooting percentage) came in guns a-blazing in 2005-06, then fairly quickly dipped to the plateau and steady decline from around 2009-10 onward. It is conceivable that the salary cap has imposed a pretty rapid push towards parity, and that 2008-09 was catching the last curve up to the plateau. That theory does seem to match Burtch’s as-yet-unpublished War on Ice high danger shots data, which suggests that high danger shots jumped from 27.7% of all shots in 2008-09 to 28.8% in 2009-10. But Bruce also posited that perhaps recorders recorded tipped shots more and more frequently as being from in-close, whereas they had recorded them as the distance of the original shooter in the past. We won’t really know until somebody, or some company, looks back through the footage and does the dirty work, but it is a theory worth including.

*Of the three, as I mentioned above, puck movement might have the greatest utility for coaching and understanding how puck possession is maintained and made productive — even if that isn’t necessarily Ryan’s focus.

**Full data, all at even-strength; first table, percentage of all shots at selected distances, second table the shooting percentages at those distances:

***Keep in mind that goal-scoring and prevention, to the sufficient degree and consistency of being a talent, is conspicuous (i.e. expensive).