Hockey Talk is a regular series where you will get to view the dialogue among a few Hockey-Graphs contributors on a particular subject, with some fun tangents.
This week we had a spontaneous discussion on black box data and why analytics departments may be misused.
@soccerquant stating edge in football no longer comes from access to data, but now more leveraging that data. #SSAC16
Carolyn Wilke This is where the NHL is having issues. Yes, there’s more data out there. Lots more data out there. but they’re not even (as a whole) using what we have yet.
Ryan Stimson Lots of teams will get tricked and/or just fall down pits of meaningless data.
asmean Yeah, and some GMs and coaching staff will foam at the mouth for data like “offensive taps after face-off wins” provided by these techs.
The most frustrating thing is most of these black box companies won’t open up their data so we could survey it. So we won’t really get the chance to prove how useless it may be. Until then, teams will continue to drink the Kool-Aid. But honestly even if we did prove how useless it really is, teams are still going to be incredibly stubborn in their beliefs.
Carolyn Wilke I honestly don’t know how many teams are drinking the Kool-Aid. I feel like a lot of them are paying for this data “because it’s what you do”. I think a lot of them are still managing from the “gut”, yanno?
Domenic Galamini I’ve heard that Sportslogiq may at least partially open up their database at some point down the line.
Matt Cane If you have a good analytics team, you could make something of the data provided by black box companies.
Carolyn Wilke Right.
Micah Blake McCurdy Yeah, I agree.
Carolyn Wilke For many black box companies, the priority rests in tracking and not in necessarily analyzing the data.
Matt Cane Precisely, Carolyn. You’d have to just take the raw data and do something with it yourself.
Ryan Stimson Isn’t Sportslogiq selling the data to teams? They aren’t analyzing it for them, right? [name redacted] told me he was going to buy it to see if he could do anything with it.
Matt Cane Yes, but I’m not sure what level they’re selling it at. There’s a big difference even between raw data and summary by game.
Domenic Galamini We plan on getting the software in Hamilton (Bulldogs) next season. It’s free for OHL clubs if you allow Sportslogiq access to the data (I’m assuming it will be sold to NHL teams).
Ryan Stimson That’s good if they offer to do that for free, especially since most OHL clubs aren’t rich.
asmean I still think some GMs and coaching staff, if not most, will continue to defy the counsel of their analytics team because they’ll continue to be stubborn and overvalue the wrong things.
Carolyn Wilke That’s true in any industry, not just sports. It’s just harder to “go under” in sports because the networks are all so cloistered in comparison to, say, a tech start-up.
Your point is a valid one. just…contextually, it’s missing something. Change is hard. Hockey hasn’t been pushed to change like other industries or other sports. The degree with which new practices will be adopted will be slow and probably grudging, because they don’t have to change or die.
And honestly, when you have a network like hockey ops, one that’s half made up of blood relatives, a lot of things are let slide that wouldn’t be in other industries. It’s unlikely so-and-so’s brother (or cousin or uncle) is fired over misusing analytics.
asmean The other problem I foresee down the line (if it’s not already happening) is that front offices will sometimes value and use their analytics team’s advice, and sometimes not.
This can be dangerously suboptimal because you’ll end up overvaluing/undervaluing different things at different times for different players. As in, you can’t just pick and choose data that you like and vice-versa. If you play this game, your decision-making will essentially be arbitrary even though you can fool yourself into thinking your decisions are perfectly data-driven and thus evidence-based.
Until you formulate basic tenants and stick to them, you’re running this risk. And this applies to analytical departments as much as it does for front offices.
Carolyn Wilke Yeah, definitely.
asmean The question is, what are those basic tenants? Is anyone working on something like the Big Tenants Of How To Evaluate Players Optimally? Or, more broadly, Optimal Hockey Strategy?
I see this exercise as being helpful in changing how front offices and coaches make decisions. And that’s really the change that needs to occur if a team wants to use their analytics department to its full potential.
Domenic Galamini Consistency is everything.
asmean How can you measure if you’re being consistently optimal if you don’t know what optimal really is?
Domenic Galamini I was just emphasizing your previous point but I agree that they go hand in hand. You could consistently use unreliable statistics to evaluate talent, but you’ll get nowhere.
You could have appropriately weighted, accurate measures of sustainable output but sometimes bend too much during the integration process thanks to dissenting preconceptions.
But I guess you could even consider the integration process as part of the established optimal strategy.
Note: Some edits were made to this conversation in the spirit of confidentiality.