(This piece was written as a collaboration between Carolyn Wilke and Chris Watkins)

What makes a good general manager in the NHL?

It’s a hard question, plagued by subjectivity, by bias, and by lack of transparency. It’s complicated by league mandates like the expansion draft and the hard salary cap. It mixes the weight of process, results, and vision into one big stew, where it can be difficult to distinguish the meat from the sauce.

It’s a question, that unlike many others, is difficult to quantify with even the most advanced of stats.

And it’s one that the league has no desire to answer definitively, as that would only hurt the men currently in those roles.

Fortunately for you, Hockey Graphs loves tackling the hard questions.

In the following articles, we will attempt to rank all 31 of the NHL’s GMs, as objectively as possible, according to seven important criteria. They each painstakingly researched trade histories, draft selections, and salary cap management, coming up with a final score for each.

While this process still was subjective, in that these scores are not quantitatively derived, it was an extremely holistic process, and both of us were forced to confront some of our own biases.

Criteria:

1. Drafting Accuracy (Weight – 17%)

While a bit difficult to determine success as it’s happening, doing well at the NHL Draft is one of the most important aspects of a general manager’s job. In this category, we attempted to look at a GM’s draft history and asked ourselves “Did they add players who projected to do well at a high level?”

Beyond that, did they add the best player possible?

While there are models to help project juniors players, most famously Cam Lawrence and Josh Weissbock’s PCS, many of these are not public, so admittedly we were going based on internal models as well as prior draft analysis.

2. Developing Drafted Players (Weight – 17%)

As important as drafting well is, it doesn’t matter much if you can’t get those young players up to speed and putting up points on your NHL team. Development of prospects can be all over the map, with some teams keeping blue-chip guys in the AHL for too long, and some rushing kids from juniors straight to the NHL before they’re ready.

Of course, this is a bit easier to quantify than the draft – you can easily tell which team has players they drafted still playing in their jersey, and of those players, whether they’re handling their workload well.

3. Extending Rostered Players (Weight – 17%)

At some point, a player on the current roster will need a new contract, whether it’s to keep them from hitting free agency or because their ELC is expiring. Extensions are the largest part of a team’s salary cap expenditures, and a good GM will be able to identify who is worth keeping around, and sign them to a deal that accurately values their performance.

For this category, we considered both the quality of player a GM extended, and whether the contract given matched the value of that player.

4. Overall Salary Cap Maintenance (Weight – 17%)

Unlike most other sports leagues, the NHL is faced with a hard salary cap. This means that no matter what, a good GM will be skilled in navigating this reality.

Hindsight on a bloated extension? Trade the player.

Aging vet with a no movement clause? Buy him out.

Predecessor overvalue a defensive blackhole? Send him to the AHL.

While this is one of the more nebulous concepts of our criteria, it is also one of the most important. Being able to recognize and own their team’s salary cap situation allows a GM to be nimble with their roster, adapting to new scenarios in an ever-changing NHL.

5. Trades (Weight – 13%)

Trades are a huge part of the NHL roster landscape and can really make – or break – a GM’s career. But as you may have noticed, we gave them a lower weight in our hierarchy because if a GM excels in the first four criteria, trades are only going to supplement a core, not build it.

A good trade meant a GM got equal or greater value than the piece he was giving up, whether that was other players or draft picks.

6. Unrestricted Free Agent Signings (Weight – 13%)

Much like trades, signing free agents is important – but a good GM will only use them to complement the core they already have in place. As a rule, UFAs tend to be overpaid, which makes wading into those waters far more dangerous than extending players already on the team.

Like extensions, we tried to take both the quality of player signed, as well as the contract relative to player value into account when grading UFA signings.

7. ELC-Level Free Agent Signings (Weight – 6%)

While the least important category, both of us recognized that to stay relevant in a cap world, making the most of entry-level contracts is key to a team’s success. And if you’re not drafting well, there is one more way to add this kind of cheap talent to a team – young free agents, either undrafted, from the NCAA, or who came over from European leagues.

Because these are all ELC deals, we mostly looked at who attracted this kind of talent, and the quality of players they signed to these deals.

Methodology:

It was important for us to put a timeframe on what we were doing, as general managers, like players, go through peaks and valleys in their careers. The trades made in 2012 may not be relevant in 2017, etc. A natural breaking point for us was the signing of the newest CBA, as that had a profound effect on both the salary cap as well as extensions that could be given.

Therefore, our research and rating process only dealt with moves made since January of 2013.

Next, we individually assigned a rating to each GM for each category, on a scale of 1-5, with 1 being the worst and 5 being the best. Then we calculated our individual overall score according to each criteria’s weight. Then, as an attempt to alleviate some of our individual biases, we averaged our two scores together, making one master rating.

We also sent out a survey to the other members of Hockey Graphs, asking them to rate the three best and three worst GMs in each of our criteria to see how it matched up with our findings.

As a final sanity check, we utilized DTMAboutheart’s Goals Above Replacement metric to better estimate the on-ice talent level of teams before and after the hiring of the current GM. We looked at Net GAR (Current Team GAR – GAR in last year of previous GM) and Max GAR ranking (the team’s best finish in total GAR under the current GM). This attempted to quantify what we were already doing subjectively, though can’t account for all of our criteria

While it was impossible to eliminate subjectivity in a task like this, we certainly approached it in a fact-based manner, and used our statistical knowledge to help guide our ratings. If nothing else, it was an interesting (and difficult) endeavor, and we hope it drives home this one point:

Being a GM is hard, and the margin between average and excellent — or terrible — is thin.

Tomorrow we will post our ratings of all 31 GMs, so stay tuned.