On Monday, Kyle Alexander and CAustin (aka the Puckologist) wrote a post on Raw Charge titled “It’s still okay for an NHL team to draft goaltenders.”  This is a topic that isn’t exactly new in the hockey analytics community – on this site alone Garret and myself have written a few posts about how unpredictable goalies are and the general consensus in the hockey analytics community being that goalies are simply not worth drafting in the early rounds of the draft, due to the variability on their results compared to other skaters (particularly forwards).

The Raw Charge guys in their post don’t totally disagree, but do think the talk of avoiding goalies is a bit exaggerated by some, concluding:

However, the gap between goalie drafting and forward drafting isn’t nearly as stark as it’s been made out to be. It’s much more worthwhile to make drafting and development at all positions better than to attempt to specialize in elite forwards to the exclusion of other positions.

Essentially, the Raw Charge guys argue:
1.  The Gap between skaters and goalies’ success and failure rates isn’t as big as people think – most evaluative measures used in such studies disfavor goalies by using metrics such as GP by a certain age, where goalies rarely get opportunities to meet such thresholds.
2.  The response to whatever gap there actually is should be to try and improve goalie evaluation – similar to how Swedish and Finnish goalie federations’ improved early goalie training to improve their goalie crop – rather than to eschew goalies altogether.
3.  The failure of goalies may also have to do with poor development processes rather than bad evaluation.

While all three points do have merit, I think they’re both quite a bit overstated.

On the first point, even if you try and use objective measures of performance instead of simply accounting for “success” vs “failure” (as the RC guys do in the post) there is still a huge gap in how goalies perform relative to draft spot as compared to skaters.  As Garret pointed out in one of this site’s first posts: Compare this for skaters:

To this for goalies:

Both of the above images courtesy of Matt Pfeffer with the Y axis being based on GVT.

GVT is being used here not just through the first few years of a career, where goalies are indeed disadvantaged (When I did my goalie Marcel projections, there was a single goalie under peak age (24) with enough sample for the projections – Robin Lehner).  And it’s pretty clear here the gap just makes goalies drafted prior to the 3rd round at best an unworthy gamble.  Not that this disagrees with what the RC guys are saying – but it should be clear the gap between skaters and goalies is real and VERY relevant.

Where I really disagree is on the second point.  There’s no question that goalie evaluation can improve at all levels – at the NHL for instance, we just had a contending team act as if Ryan Miller was a massive improvement over Halak/Elliott/Allen (nope) and stake its trade deadline on that move.  And at the amateur level, Rhys J (That’s Offside!) has noted that teams often seem to ignore save percentage when evaluating junior goalies, which obviously is kind of silly (ignoring data in making an evaluation is pretty much always going to be bad – especially with goalies for whom the eye test can be VERY misleading).

That said, goalies are pretty much always going to be harder to evaluate than skaters and it’s hard to see it getting that much better (especially since it’s not like skaters can’t be evaluated better at the junior level either!) for the same reason why goalies are such a problem at the NHL Level:  Goalie performance is just naturally more variable due to sample size.  And scouting goalies isn’t ever going to really get a decent sample size like you can get from a few years’ data in the NHL – the stats are going to cover a smaller amount of games (during which goalie skill is more likely to actually change than in the NHL) as is the scouting.  Whereas it may be possible for national programs to improve the development of goalies by improving training methods, evaluative methods are not going to get much better samples with which to work.

As to the third point – it’s definitely true that development processes may get better – hell, that’s why goalies have gotten so much better since the 80s such that they’re where they are now.  At the same time, again, it seems more likely that the failure of so many goalies has to do with the fact that scouting them pre-draft involves dealing with tiny samples for both eyes and stats in a position where small sample size flukes are most notable.

Overall, it would be crazy to say not to draft goalies at all in the draft in favor of skaters throughout the entire draft, but it just seems unlikely that goalie evaluation and development will improve to the point that drafting goalies early will ever make sense.