It’s not easy creating a data-driven decision-making culture in any organization, let alone one as bound in tradition and lore as the NHL, where hockey men are imbued with mythical powers of observation and judgement just by virtue of having played the game. And yet, the NHL is clearly moving in that direction. It may be at a glacier-like pace, but I suppose that makes sense, what with the ice and all. Despite some early stumbles, it’s probably safe to say that it is only a matter of time before data-driven decisions are the norm rather than the exception. Whether that happens while we still have glaciers is another matter.
But even when there is a managerial will and top-down direction to move toward a data-driven culture, it is often difficult to introduce data analysis into the existing decision-making process of an organization. It’s not just deep structural changes that are necessary, but also staff will need a robust change management process. It’s hard enough to get people to accept change, but a new culture requires that they go beyond acceptance and embrace it as a new way of doing things. This is a difficult process in any organization. However, it is made more difficult in the hockey world where many in positions of authority are in those roles precisely because they “played the game” and understand the traditional way of doing things.
But what if you could start from scratch and build something from the ground up?
The new NHL franchise in Seattle certainly had the opportunity to nurture a data-driven culture by embedding it into the organizational structure and decision-making processes right from the start. But starting from a blank slate will only take things so far. Despite what some of the old-school hockey men will tell you, the roles in that organizational structure are still being filled by people. People that bring their own preconceived notions of how things should be done and what role, if any, data analysis should play in the decisions they make.
Math is hard, but change is harder. And people hate it.
They especially hate change to systems and processes that they don’t understand. There’s a reason that an entire change management industry has grown out of nowhere in the business world. The NHL is rife with examples of where the shift to data-driven decision-making failed to take root, either by design (Edmonton, circa 2014) or by lack of commitment to the strategic direction (Florida, circa 2016).
But the league has also seen a few successes, most notably in Toronto, where the shift has been managed through a structured process over time, and also Carolina, where owner Tom Dundon has brought a belief in the value of data-driven decision-making with him from the business world. The same is true in Colorado, where Kroenke Sports & Entertainment rely on data-driven analysis across all of their sports holdings.
Those are just a few examples, but there are many more. Hockey Graphs’ own Shayna Goldman has been keeping a running list of data-focused roles across the NHL, with the latest version is in this piece over at the Athletic: Eight lessons Rangers GM Chris Drury can learn from the NHL’s final eight contenders.
It’s one thing to assemble an “analytics” team, but it’s quite another to actually integrate them into the decision-making process. In a previous piece at the Athletic, Shayna laid out a quote from an NHL data analyst that perfectly encapsulates the situation:
“In this day and age, you can’t swing a dead cat without hitting someone who works in data analysis, but to ‘move the needle’ it’s important to find those that both can understand and speak about both hockey and data.”
Building A Team of Ts
You should go read that article for many more great insights from analysts working in NHL front offices. But the common thread you will find in most of the quotes in there is that the surest way to fail is to assemble a team of one-dimensional statisticians or computer scientists. What you need is what the business world calls a team of Ts:
These are people who not only have a specialized knowledge in their area of expertise, but who are also comfortable working and communicating with others that might have adjacent or complementary domain expertise. Contrast that with a team of I’s that are subject matter experts in a very narrow field, but that can’t communicate that knowledge to others outside of that field.
The ideal data science team needs not only specialized skills, but also the ability to translate and communicate with decision-makers in the organization. This is especially important when the decision-makers don’t have the same level of expertise when it comes to data management, data analysis, or even basic statistics and probabilities. This is true in the business world, and it is certainly true in the hockey world.
The Decision Support Team
Ultimately, the role of a data science or data analysis team in any organization should be to help decision-makers make better decisions. Anything else is just window-dressing.
It isn’t enough to have a team of highly specialized data analysts, even if they are comfortable working in adjacent fields. You need a well-rounded team with subject matter expertise in a number of areas in order to generate actionable insights in ways that decision-makers understand:
- Decision science, to help formulate research questions, develop models, and structure analyses;
- Data analysis, to collect and clean data, manage databases, build queries, and write code;
- Communication and visualization, to distill results into insights and build interactive tools and visualizations to deliver the message, simulate scenarios, or provide self-serve data access; and
- Domain expertise in the business at hand, whether that is professional hockey or selling health supplements.
In addition to having the requisite skills, the Decision Support Team will need to function seamlessly together to be able to synthesize the needs of the organization into research objectives and that can deploy advanced analytical and statistical techniques to identify insights, evaluate options, model scenarios, and answer questions. All in support of allowing the organization to make better decisions.
Ultimately, you can break the Decision Support Team into a variety of roles, again with deep subject matter expertise specific fields, but enough lateral knowledge and comfort working with similar SMEs in adjacent roles.
These roles can be generally defined as specialists in the following areas:
- A Research Manager with data science expertise that can formulate statistical models and identify the analysis techniques and data sources required to meet the research objectives.
- Data Analysts to source and structure the data for analysis, or stream it for dynamic visualizations or apps.
- Visualization Specialists to create interactive visualizations to present and communicate the results in ways that are easily digestible by the business, i.e. Hockey Ops, so that might be the front office, pro or amateur scouting, or even the coaching staff.
These are your Ts. They have a deep expertise in their respective fields, but they also have an understanding of the related pieces of the whole. And to truly glue this team of specialists together, you need a generalist that understands the various functions on the team, as well as the nature of the business:
- A Team Lead that is not so much an expert in any one area as a communicator with an ability to translate the needs of the business into research objectives and also communicate the results of the analysis back to the business.
Decision Support Team Workflow
So now you’ve got your team, but how do they integrate into the rest of the organization? Especially an organization composed of hockey men that have always made decisions in a much different way? This is why at the very least the Team Lead needs to have some domain expertise in the field. That role needs someone that could fit right in at those old school BOGSAT decision-making sessions:
In many ways that’s exactly what Bill Armstrong did in bringing Lee Stempniak into the Coyotes’ organization to sit as a buffer between the analytics team and hockey ops:
“But Armstrong wanted to give Perri and his staff a conduit to hockey operations, one who could translate complex data into hockey language while filtering out information that had no application, or was simply off base. In addition to Perri, the Coyotes have hired former player Lee Stempniak as their hockey data strategist. Stempniak played 14 seasons for 10 teams (including the Coyotes) before retiring in 2019.“
But the ability to fit in is not enough. The goal here should be to shift the nature of the BOGSAT sessions in hockey ops to a more structured decision-making process that is grounded in sound, decision science based principles. And that will take more than just a former player to serve as a buffer and to provide the translation function. It needs someone with skill and experience in decision science and change management.
But that’s a topic for another post. Baby steps here.
In terms of integrating into an existing organizational structure, or even a new organization like Seattle, which is still largely made up of people that have cut their managerial teeth in the culture of traditional hockey decision-making processes, that team lead needs to be able to operate equally well in both worlds, and to act as a translator between them.
A typical workflow could start either through a specific request from the business, i.e. hockey operations, or perhaps a need that the team lead identifies through interactions with the hockey ops staff.
This question that hockey ops might have is then translated into a research objective and examined from a decision science perspective to determine how best to structure the analysis. From there, an approach to data collection and analysis would be developed. This might include building a model or applying statistical analysis techniques to transform the raw data into actionable insights. Finally, the results would be packaged into a format that could be easily communicated back to the business, both in terms of format and presentation, as well as translation to language and terms that the decision-makers can understand and that directly addresses the initial request.
While a formal workflow to respond to requests is always necessary, any innovative organization should always allow room for serendipitous discovery of insights from anywhere within the business. In this case, there should be opportunities for the Decision Support Team to conduct prospective analyses with available data in order to try and identify actionable insights outside of “official” requests for information. You might have someone just playing around with the data find something interesting that bears further investigation, and the workflow should be able to accommodate those situations.
Operationalizing the Team
I want to wrap up with some thoughts about how you actually put this type of decision-support infrastructure in place.
Whether it’s a hockey organization or any other business with a strong culture that stresses traditional approaches and ingrained knowledge, any move to a structured, analytical decision-making process will be met with resistance. Any early missteps in operationalizing a data-driven approach could set the process back years, if not lead to outright abandonment.
In this context, there can’t be any half-measures. Introducing a bare-bones team, or one that is not well-rounded into this type of organization is likely to lead to failure. Having only one or two of these key skill sets on the team will produce suboptimal results at best, and, at worst, will be completely ignored by decision-makers.
It takes a well-rounded, fully staffed team with the time and resources to gain enough of a foothold in the decision-making process to start adding value. So whether you’re trying to set up this type of team and function in a sports organization or a vitamin supply business, put some resources behind it and give it a real chance to succeed.
This is the third installment in a series on applying best-practices in data-driven decision-making to the world of professional hockey. You can find the previous installment here.
Cover image from courtesy of Dataedo, used under Creative Commons Attribution-NoDerivs 3.0 License.
Follow me on Twitter: Follow @petbugs13
Hockey Graphs 
I agree but you can’t make decisions without causal inference. A decision is something you do because you hope it has a causal effect on some outcome. That is what’s missing in basically all of hockey analytics.