Is Jacob Markstrom still good?

Whether you come at hockey from the numbers or from traditional scouting, finding NHL-quality goaltending is a challenge. In order to have a good sense of a goalie’s talent (as measured by even-strength Sv%), you need to observe about 4,000 shots worth of work. On average, a goalie needs to play over three seasons as a starter (or eight seasons as a backup) to see that many shots. If they play poorly, few netminders will ever get close to that amount of playing time and most goalies are entering age-related decline by the time they’ve seen that many shots. As such, teams usually make decisions on goaltenders long before they’ve seen 4,000 shots and, unsurprisingly, teams make mistakes.

This is usually framed as “the Cam Ward problem”: what to do about goalies who excel over a short stretch of games, particularly late in the season or during the playoffs? Goalies like this are often rewarded with multi-year contracts but few are talented enough to live up to them. In contrast, there’s also “the Jacob Markstrom problem.” Markstrom, currently a Vancouver Canuck, is best known as an excellent SEL and AHL performer who’s struggled badly in NHL action. We don’t often discuss the risk of prematurely abandoning a talented goalie based on a stretch of poor early play but it almost surely happens. How often do we see these situations in action?

This isn’t an easy question to study with goaltending data. Whenever poor early play costs a young talent his spot in the NHL, we don’t get to observe the rebound. More generally, team decisions regarding their goaltending rarely come down to save percentages alone and circumstances will differ from team to team. Still, if we want to make a more general analysis of how often good or bad luck might lead a team to misjudge its goaltending options, a simulation involving fake goalies can show us how often the Cam Ward and Jacob Markstrom problems might occur through random chance.

Let’s assume a hypothetical team with three young goalies. The first has a true talent corresponding to a 0.925 even-strength Sv% (i.e., a slightly above-average NHL starter); goalie #2’s talent corresponds to a 0.915 5v5 Sv% (making him a passable backup); and the third goalie is a replacement-level 0.905. I assumed binomially-distributed Sv%s and created 10,000 random sequences of 4,000 shots for each goalie.

To illustrate the challenge of differentiating goalies without observing a lot of shots, the figure below depicts 95% confidence intervals around the average cumulative Sv% for each goalie. The reds represent lower and upper bounds for our hypothetical starter, and the blues indicate bounds for the backup. At 4,000 shots, we should expect to differentiate the starter and the replacement, but a backup goalie still has a decent probability of looking like either one.

The following table makes this clearer. The nine columns on the right represent a goalie’s observed Sv% conditional on his true-talent Sv%. For example, the second-to-last column indicates how often a starter-quality goalie will have a backup’s track record. One takeaway from these estimates: with a career 0.907 Sv% on 1,100 shots, things are not looking good for Markstrom as an NHL starter. Another: a replacement-quality goalie can sustain a great run of play in the short term, but (as Carolina found with Ward) handing him a starter’s workload of roughly 1,250 shots against per season is not likely to end well. More generally, even through 4,000 shots, 12.3% of backup-quality goalies will look like starters, and 13.6% of replacement-level goalies will perform like a decent backup. If a team makes a decision after, say, 1,000 shots, they’re even more likely to misjudge their goalies. At this point, 30% of backup-quality netminders look like replacement goalies, 27% of backups have starter-quality numbers, 23% of replacement goalies look like backups, and 25% of starters look like backups.

The graph below illustrates this in a different way. If a starter-quality goalie can get a full season of work (i.e., 1,250 shots), there’s only a 5% probability that he’ll have a lower Sv% than a replacement goalie, and he’ll have a roughly 80% probability of a better Sv% than a backup. In a backup workload, however (on average, about 500 shots a season), our backup plays worse than a replacement-level goalie about 27% of the time. After two seasons of an average workload, the starter is most likely outperforming the other goalies. Yet the backup is still in danger of playing like an AHLer, having only seen about 1,000 shots.

One takeaway from all of this is that the Jacob Markstrom problem may not be a big deal. If a goalie gives you replacement-level play early in his career, it’s unlikely that he’ll have starter-level numbers by the time he’s seen 4,000 career shots. Similarly, so long as you don’t react to a hot streak with a huge contract offer, you’re not likely to make a Cam Ward-type error. On the other hand, it also suggests that it’s not hard for teams to make mistakes on the margin. By 4,000 shots, the majority of goalies will have the track record you’d expect based on their talent. But before that, when most teams are faced with important decisions about their tandem, it’s not hard to end up with a backup-quality goalie in a starting role, and vice versa. Given their lesser workloads, it’s especially easy to miscast a replacement goalie as your backup, and to send a capable goalie to the AHL.

Practically, all this emphasizes the need for teams to keep their options open when it comes to goaltending. If you have a goalie who performs like a good starter over a full starting workload, and based on his age is likely to keep doing so, it’s probably safe to assume he belongs in the NHL. And similarly, if a goalie plays like hell, even for under 1,000 shots, it’s very likely that he isn’t a future starter. Apart from these edge cases, though, is a great deal of gray area. Without a large sample of shots, teams should be careful about overcommitting to a starter, as well as giving up on a prospect too quickly.