Check out my first contribution to hockey-graphs.com HERE
Lately I have been receiving numerous emails and questions on twitter regarding HERO and WARRIOR Charts. Instead of responding to everyone individually, I have decided to write this post that hopefully answers everyone’s questions and also clarifies some misconceptions regarding both evaluation tools.
Question: Which performance measures are used, how are they calculated and why are they important?
Both visuals use these same seven metrics to evaluate a player’s effectiveness at 5-on-5…
% of Available 5v5 Time On Ice (5v5 TOI%)
Description: 5v5 TOI% measures the percentage of a team’s 5v5 ice time that a skater is given.
Significance: Using this information, we can get a sense of how much faith a skater’s coach has in his ability as well as the extent of exposure a skater receives in a typical game.
Limitations: Ice time is often thought to have a significant impact on performance measures despite evidence to the contrary. Changes in TOI from year to year are not related to changes in measures of production and possession from year-to-year. Because of this fact, TOI% is limited with respect to explaining skater output.
Goals per 60 minutes of ice time (Goals/60)
Description: The rate at which skaters score, expressed as a rate per 60 minutes of 5v5 ice time.
Significance: The importance of measuring a skater’s ability to score is pretty obvious. That being said, one must separate output from opportunity in order to accurately assess a skater’s goal-scoring ability. This can be accomplished by expressing goal production as a rate statistic (per 60 minutes in this case).
Limitations: Goals are heavily influenced by the volatility of shooting percentage (we will get to how WARRIOR charts adjust for this later on in the piece) and the quality of a skater’s linemates.
First assists per 60 minutes of ice time (First A/60)
Description: The rate at which skaters record a first assist, expressed as a rate per 60 minutes of 5v5 ice time.
Significance: In addition to goal-scoring, a player’s ability to set-up linemates contributes to his overall individual production. The best way to measure this ability is by using first assists in the form of a rate statistic. Secondary assists are excluded as they lack repeatability (especially when skater’s switch teams) and therefore aren’t considered to be talent-driven.
Limitations: First assists are susceptible to the same issues we encounter when using goals to measure ability – the variable nature of Sh% and the quality of one’s linemates have a significant impact on results.
Primary points per 60 minutes of ice time (Primary P/60)
Description: Goals per 60 minutes of ice time + first assists per 60 minutes of ice time
Significance: See Goals/60 and First A/60.
Limitations: See Goals/60 and First A/60.
Corsi For per 60 minutes of ice time relative to teammates (CF60 RelTM)
Description: The rate at which shots occur with a player on the ice, relative to the rate his linemates manage when they are away from him.
Significance: Shot generation is an important aspect of a player’s ability to outscore their opposition long-term. Accounting for a skater’s quality of teammates allows us to best isolate his contributions to generating chances for himself and his linemates.
Limitations: The calculation for CF60 RelTM is… CF60 minus Linemates’ CF60 away from the player of interest (weighted by time on ice with the player of interest). Although this method does a better job of dealing with collinearity than traditional on-ice/off-ice relative measures, it has its limitations. The first issue is sample size. Some skaters spend a large % of ice time with a particular linemate. This means that the sample of TOI away from the player of interest for that particular linemate will be susceptible to noise – potentially skewing RelTM measures.
Another issue is the fact that talent distribution is uneven throughout a lineup. For example, a third line center’s most common winger may spend a significant portion of his ice time with the team’s elite first line centre when he is away from the player on interest. This will unfairly penalize that particular third line center’s output simply due to the fact that he plays on a team with an elite first line center. Because of this, it is more appropriate to think of RelTM metrics as a measure of a skater’s ability relative to teammates playing the same position.
Corsi Against per 60 minutes of ice time relative to teammates (CA60 RelTM)
Description: The rate at which a team concedes shots against with a player on the ice, relative to the rate his linemates manage when they are away from him.
Significance: In addition to shot generation, shot suppression is another skill that contributes to a skater’s ability to outscore his opposition long-term.
Limitations: See CF60 RelTM
Corsi Differential per 60 minutes of ice time relative to teammates (CD60 RelTM)
*Sometimes expressed as Corsi For % relative to teammates (CF% RelTM)
Description: A player’s shot differential relative what his linemates manage away from him. CF60 RelTM – CA60 RelTM.
Significance: See CF60 RelTM and CA60 RelTM
Limitations: See CF60 RelTM
Question: Why do WARRIOR charts and HERO charts sometimes display different results for the same skater?
The main reason is that WARRIORs and HEROs are very different in how measures are aggregated across seasons. The following chart breaks down some of the key differences between the way both visuals function…
Question: How does the WARRIOR chart regression function work?
The method used to regress metrics is very similar to the method I used when calculating my Marcel projections.
Step 1: Seasons are weighted by temporal proximity…
Step 2: Metrics are broken down into components…
Goals/60 = Shots/60 * Shooting%
First A/60 = Teammate Shots/60 * Teammate Shooting % * First Assist % on teammate Goals
*TOI%, CF60 RelTM, CA60 RelTM are not broken down into components
Step 3: Components are regressed to the mean to an extent that is dependant upon year-to-year persistence…
The average forward in this data set had a sample of 196 shots on goal. Average expected regression was 63% to the mean. In order to achieve 63% regression to the mean on average, we must add an additional 331 shots on goal (at league average Sh%) to each forwards weighted sample.
The same method was applied to all components.
Step 4: Finally, the weighted and regressed components are reassembled
Question: Do WARRIOR Charts and HERO Charts tell us everything we need to know about a skater’s effectiveness at even strength?
Okay, this isn’t exactly a FAQ but I think it needs to be addressed. Pro scouts can exhale – the answer is NO! These visuals were created with the original intent of providing a quick resource to attain a surface look at a player’s effectiveness when trades/signings occur. As explained earlier, the metrics used to evaluate talent are far from perfect. Contextual factors are impossible to completely account for and player evaluation requires a deeper look into those factors before drawing any conclusions. Despite these imperfections, I believe that both Warrior and Hero charts satisfy my original intent.
Another goal was to simplify modern statistics in hockey and capture the interest of casual fans. I hope that some of those fans are encouraged to do some of their own research and help improve our understanding of the game.
If you have a question that wasn’t answered in this piece, or maybe even a question that was answered but not sufficiently so, feel free to email me at email@example.com
In the realm of baseball statistics, Tom Tango (@TangoTiger) is well known for his simple yet effective forecasting system dubbed The Marcel the Monkey Forecasting System (or Marcels for short). Why “Marcel the Monkey”? Tango relates his projections to stock prices. Like a stock price, Marcel forecasts are based on all known information. Predicting that a player will outperform or underperform his Marcel projection is as accurate as a monkey picking a stock to improve. The implication is that Marcel forecasts model a player’s true talent level and fluctuations away from those forecasts can be largely attributed to randomness.
A Marcel projection is determined by the following procedure…
- A player’s past performance is weighted by temporal proximity (recent seasons are valued more than earlier ones).
- Projections are regressed to the mean to an extent that is dependent on the sample size of the player’s weighted past performance
- An age adjustment is applied.
We can use Tango’s Marcel methodology to predict player performance in hockey by following the same three-step process. I have done exactly that.
Continue reading if you are interested in the process of calculating Marcel projections. Skip to the RESULTS section of the piece if you simply wish to see player forecasts for the 2015-16 regular season.
STEP 1: Weighting Past Performance
We can improve our ability to forecast future performance at the player level by using data that encompasses multiple seasons versus using a single season’s worth of information. There is room for improvement still, by weighting past seasons based on their recentness. The question is how to determine the value of a player’s stats in 2014-15 relative to his stats in 2013-14 and 2012-13. The answer: Multivariate regression.
Using season’s n-3, n-2 and n-1 as our independent variables and season n as our dependant variable, we can run a regression and determine the appropriate weightings for each of a player’s last three seasons. Running a regression for all forwards to play 30+ games in each of their last 3 seasons and 60+ games in their immediate season produces the following coefficients for individual Shots/60…
Considering our objective, the intercept is irrelevant. P-values <0.05 suggest that each of a forward’s last 3 seasons are statistically significant when predicting future performance. Based on the coefficients, it’s reasonable to weight a forward’s last three seasons 5/2/2 (a simpler alternative to 0.50/0.17/0.20)…
The same regression is performed on TOI/Gm, Shooting%, Assists/60 and CF% Rel. Doing so gives us the following weights…
STEP 2: Regressing to the Mean
Simply weighting a player’s past performance doesn’t necessarily reflect his true talent level. In hockey, a large portion of what we observe is the result of randomness. Because of this, we should expect a player’s true ability to be closer to the mean than his statistics indicate. The extent to which we regress to the mean depends on sample size (smaller sample = regress further to the mean).
If you look at projected individual Shots/60 calculated using our weights determined in Step 1, you will notice that a player’s subsequent season typically yields results closer to league average…
*Each bucket contains 100 forwards
As you can see, a forward projected to manage 11.0 Shots/60 only averages 10.1 in his next season. Given the total sample of forwards, projected Shots/60 best reflects future performance when regressed 16% to the mean. The average forward in the data set had an existing weighted sample of 775 minutes. Therefore in order to regress 16% to the mean on average, we need to add an additional 148 minutes with a league average Shots/60 to their existing weighted sample.
What is a weighted sample? It is the resulting TOI from applying Marcel weights for Shots/60 on past seasons.
Ex. 5-2-2 weights are applied to last 3 seasons which consist of 450, 550, and 200 minute samples. [(450*5) + (550*2) + (200*2)] / (5+2+2) = 406 minute weighted sample
How did I arrive at 148 minutes?
Average weighted sample of 775 minutes x (16% / 84%) = 148 minutes.
A forward with a projected Shots/60 of 11.0 after 775 weighted minutes should be expected to manage a Shots/60 of 10.4 in his next season. ((775*11)+(148*7.2))/(775+148) = 10.4
The same method can be used to regress other metrics…
STEP 3: Adjusting for Age
The last and most simplistic step is adjusting for age. In this case, we will limit our sample to forwards and defensemen to play 40+ games in their immediate and previous season. Next, we can find the relationship between age and (Shots – Projected Shots/60)…
The linear equation can then be used to adjust Projected Shots/60 considering the age of the player in question. The same age-adjustment method can be easily applied to the rest of the projected measures.
The following table compares the predictive ability of Marcel projections to single-season data (40+ GP in season n-1, 40+ GP in season n)…
Here are your projected Forward leaders in each 5v5 category for 2015-16…
…and here are your projected leaders among defensemen
Big thanks to War-On-Ice.com for the data used to calculate these projections
The other day I listened to an audio file featuring Panthers head coach Gerrard Gallant and assistant coach Mike Kelly. They were asked about the emergence of modern statistics in hockey and their consideration for that data. What ensued was a candid discussion that made for an extremely frustrating listen. Here are some direct quotes…
“When we get the sheet that’s got the guys who are the best offensive players, the best defensive players, the best faceoff guys blah blah blah – about five different things… we knew it all before we got the sheet”
“If you do your work and you’ve been around the league, you’ll know the players around the league.”
“One of our players was our top analytics player and I couldn’t stand watching him on the ice. He was our top analytics player and I didn’t like the way he played one bit. So some of it is really good, but some of it you can’t get fooled by it”
That last quote (which seems to be referring to ex-Panther Sean Bergenheim) is the most troubling. It’s a perfect example of searching for and interpreting information in a way that confirms one’s existing beliefs. You may or may not recognize this an example of confirmation bias – a treacherous cognitive bias that can lead us into making poor decisions by ignoring valid, yet contrary evidence. Confirmation bias is just one of the numerous cognitive biases that can steer us away from rationality.
I wonder if Gallant and Kelly’s tones would change with respect to modern statistics if they were to gain a better understanding of how our brains process large amounts of information. Because that’s what we do when we base our analysis of hockey on observations alone – we attempt to process a very large number of events. We like to think that our beliefs are derived from the objective analysis of our accurate and expansive query of observations. The prevalence of confirmation bias proves that we aren’t as objective as we think we are and reading further will demonstrate how troubles also arise when sifting through our expansive memory banks.
In addition to biases we have heuristics. A heuristic is a mental shortcut which helps speed up decision making. Without heuristics we would spend most of our time overanalyzing problems before making a decision, leaving us with very little time to react. The issue is that sometimes these shortcuts can lead us into making irrational decisions – that’s when they take their place alongside confirmation bias as cognitive biases.
We feel safer driving in a car than flying in a plane. Why? Because we tend to overestimate the likelihood of events that stand out in our minds – the catastrophic event of a plane crash fits that description.
This is an example of availability heuristic – the tendency to overestimate the likelihood of events with greater “availability” in memory, which can be influenced by how recent those memories are or how emotionally charged they may be.
According to the National Highway Traffic Safety Administration, there are approximately 1.27 deaths per 100 million miles travelled by car in the United States. That number is virtually 0 per 100 million miles travelled by U.S. air carrier (a statistic compiled by the National Transportation Safety Board). Based on these relevant statistics, it’s apparent that flying is the far safer mode of transportation.
The funny thing is that even after reading those statistics, the next time you board a plane you will still likely feel a higher sense of anxiety than the next time you start your car and shift into drive. That’s just how powerful availability heuristic is.
When we evaluate the play of individuals in hockey, we sometimes fall victim to availability heuristic. There’s a reason why NHL coaches feel safer deploying defensemen like Roman Polak when defending a lead than say, Jake Gardiner. Jake Gardiner has been known to give the puck away, pivot in a lazy fashion, lose his man and occasionally the result has been a goal against. Many would conclude that he is prone to mental lapses in the defensive zone and it seems safe to assume that the Leafs give up more chances against with him on the ice than a “safer” alternative like Roman Polak. Scoring chance data (via War-On-Ice.com) from this past season suggests that the opposite is true – Jake Gardiner SCA/60: 28.5 Roman Polak SCA/60: 31.7
In hockey, mistakes that directly lead to quality chances against are more available in memory than the small errors like needlessly chipping the puck into the neutral zone, passive zone entry defense etc which are also linked, albeit less directly, to conceding scoring chances against. Gardiner’s errors are more dramatic in our memory than Polak’s in the same way a plane crash stands out in our minds more than a car crash.
Availability heuristic is actually the phenomenon Tyler Dellow was describing in this quote from his well-known “Big Mistake” piece…
“If someone asked me what I think the biggest failing of the eyeball test is, I’d respond that it’s the emphasis on the big mistake. There are gigabytes of information contained in a hockey game. So much information that I think it’s difficult for anyone to take it in and organize it rationally. The way that our brains deal with that is by focusing on the big mistake.”
Clearly there are some major issues with evaluating play by relying solely on observations. Studies have shown that even improper linear models (models developed by estimating the weights of predictive variables) are better predictors than human judgement. It’s apparent that implementing valid and reliable data as part of the decision making process can help us overcome some of the biases and heuristics that actively cause us to deviate away from logical interpretation and accurate judgement. The ignorance and arrogance portrayed by Florida’s coaching staff can only hamper their team’s chances of success in an environment as competitive and unforgiving as the National Hockey League. I’ll leave you with this…
“We can be blind to the obvious, and we are also blind to our blindness.” – Daniel Kahneman, Thinking Fast and Slow
Data courtesy of Puckalytics.com – encompasses all F/D seasons in which they play 400+ 5v5 minutes (2007-08 to 2014-15).
A common goal in modern hockey statistics is to find repeating patterns that best predict future results. In other words, you want two things: repeatability and predictive ability. Understanding the importance of these two concepts has led to the popularization of shot attempt based measures like Corsi and Fenwick. Consequently, on-ice percentages have often times been deemed useless at the player level due to their lack of repeatability. Therein lies the inefficiency. Just because a metric is susceptible to high levels of variance doesn’t necessarily warrant its complete dismissal. In the case of shooting percentage and save percentage, it comes down to regressing and extracting whatever useful information we can. The question is, how far do we regress? The point of this piece is to answer that exact question and show how we can regress percentages to markedly improve on the predictive ability of Corsi at the player level.
REGRESSING ON-ICE CORSI SHOOTING PERCENTAGE (FORWARDS)
The idea is to use a method that regresses on-ice Corsi shooting percentage dependant on sample size. To do so we need to determine the number of shot attempts (at league average CSh%) that need to be added to a player’s existing sample in order to produce a regressed CSh% that, when multiplied with Corsi For / 60 (CF60), best predicts future Goals For / 60 (GF60). In this case, “Future GF60” will be a player’s GF60 in his subsequent season.
The chart below shows how the predictive ability of regressed Goals For / 60 (rGF60 = regressed CSh% * CF60) changes as varying amounts of shot attempts (at league average CSh%) are added to a forward’s sample…
The vertex at (1100, 0.5030) shows where the predictive value of regressed GF60 peaks. This means that in order to best predict future goals, we need to add 1100 Corsi For and 46.75 Goals For (1100 CF * league average conversion rate of 4.25% = 46.75 GF) to a forward’s existing sample.
The following graph demonstrates how far a forward’s CSh% is regressed to the mean when adding 1100 CF and 46.75 GF…
Notice how a forward’s actual on-ice Corsi shooting % carries more weight as his sample size increases.
How rGF60 (regressed CSh% * CF60) compares to CF60 and GF60 when predicting future goals…
REGRESSING ON-ICE CORSI SHOOTING PERCENTAGE (DEFENSEMEN)
The same method can also be used to regress on-ice Corsi shooting percentage for defensemen…
In this case, the predictive ability of rGF60 peaks at the point (2900, 0.2480). So in other words, when trying to best predict future GF60, you need to add 2900 Corsi For and 123.25 Goals For (2900 CF * league average conversion rate of 4.25% = 123.25 GF) to a defenseman’s existing sample.
The following chart demonstrates how adding 2900 CF and 123.25 GF regresses a defenseman’s observable CSh% to the mean (vs. adding 1100 CF and 46.75 GF for forwards)…
How rGF60 (regressed CSh% * CF60) compares to CF60 and GF60 when predicting future GF60 for defensemen…
REGRESSING ON-ICE CORSI SAVE PERCENTAGE (FORWARDS + DEFENSEMEN)
Performing the same test on on-ice CSv% does result in a slight bump in predictive ability for both forwards and defensemen. However, persistence in goaltending is likely responsible for that small increase. Because of this, on-ice Corsi Sv%s are regressed 100% to the mean for both forwards and defensemen.
CALCULATING rGF% (FORWARDS + DEFENSEMEN)
*Click image to enlarge
TESTING THE PREDICTIVE ABILITY OF GF%, CF% & rGF%
The predictive ability of rGF% is a marked improvement over CF% for forwards – the same cannot be said for defensemen. This makes sense given the fact that on-ice shooting percentages are generally more repeatable for forwards.
Constructing a roster full of talent is necessary for success in the NHL as it undoubtedly is in all team-based professional sports. Evidently, it is also important that a coach deploys his artillery of talent in an optimal fashion. The magnitude of a player’s impact is largely a product of opportunity. In hockey, opportunity comes in a few different forms – the most prominent one being ice time.
Now consider what we know about useful possession. Teams with better shot-attempt differentials are more likely to win a larger percentage of future games than those with poor shot-attempt differentials. Why? Because they are still capable of winning when the percentages don’t swing in their favor. It is a coach’s job to deploy his roster in a way that optimizes his team’s Corsi differential and consequently improves their chances of success. In order to do so, better possession players need to be the beneficiaries of additional even strength ice time.
Below are a couple of charts that show the relationship between even strength ice time per game and Usage Adjusted Corsi% for forwards and defensemen sine 2007-08 (500+ 5v5 TOI)…
It seems as though NHL coaches have historically done a really poor job allocating ice time to defensemen who tilt the ice in the right direction. This is staggering considering that defensemen have virtually no bearing over on-ice percentages and therefore their possession rates are particularly illustrative of their effectiveness. The issue likely stems from a few misconceptions surrounding which attributes make a defenseman effective. There are obviously kinks in the evaluation process and the point of this piece to bring to light a few of these fatal fallacies.
DEFENSE IS NOT LIMITED TO THE DEFENSIVE ZONE
In hockey, defense is used to describe anything that contributes to a team’s ability to reduce the rate at which goals are scored against. The best way to reduce picking pucks out your own net is to suppress the opposition’s rate of chances and attempts for. Jen Lute Costella (@RegressedPDO on Twitter) gave an outstanding presentation about shot suppression at the Pittsburgh Analytics Workshop back in November. Her point was basically that anything that increases the amount of time between your opponent’s opportunities technically qualifies as shot suppression. That includes…
– Zone coverage
– Zone exits
– Preventing zone entries
– Entering the opposing team’s defensive zone with or without control
– Sustaining offensive zone pressure
– Fore-checking (less applicable to defensemen)
As you can see, defense happens all over the ice. Too often we see NHL coaches lean on big physical blue liners like Brooks Orpik, Andrew MacDonald, Rob Scuderi etc who lack the ability to move the puck and sustain offensive zone pressure – the archetypal shut-down defenseman. Regardless of whether or not these types of defensemen are effective in their own zone, they often spend far too much time defending and as a result, rack up attempts/chances against at a very high rate.
PHYSICALITY/GRIT IS NOT A PROXY FOR DEFENSIVE EFFICACY
This one ties also into the idea that defense is limited to the defensive zone. Coaches often attribute physicality to being reliable defensively but as noted earlier, defensive play goes far behind pinning an opposing forward along the wall and clearing the crease. To prove it we can look at hit totals for defensemen and test the correlation with goals against per 60.
Since defensemen who spend more time in their own zone tend to hit more frequently, we will adjust hits for possession by dividing hits by Corsi against
A correlation coefficient of 0.20664 suggests that there is a link, albeit a weak one, between hits per Corsi against and goals against per 60. This means that physicality does play a role in reducing a defenseman’s goals against (which really isn’t surprising). The issue in hockey is that the relationship between physicality and defensive ability is perceived as being much stronger and that leads to sub-par defensemen being over-utilized. The great thing about Corsi is that it is a macro-statistic and therefore a gritty defender’s ability to reduce attempts against using his size is already captured within shot attempt measures. This dilemma is very similar to the over-emphasis of a Center’s ability to win face-offs: causal factor with a marginal influence on Corsi differential.
PRODUCTION IS MORESO A PRODUCT OF OPPORTUNITY THAN IT IS SKILL
The objective in hockey is to out-score your opposition and thus it makes sense that production is such a highly sought after commodity in the NHL. The hockey analytics movement has changed the way we measure point production by shifting from units per game to units per 60 minutes. This way we can even-handedly compare players who receive differing amounts of ice time per game. Just like most measures in hockey, point scoring is also subject to random noise from season to season as on-ice shooting percentages and individual point percentages flux uncontrollably.
The following charts tests the repeatability of 5v5 Points/60 from season-to-season since 2007-08 (500+ 5v5 TOI)…
As you can see, even strength production is much more repeatable for forwards than it is defensemen. That shouldn’t be surprising given the fact that high scoring forwards do have the ability to sustain high on-ice shooting percentages to a certain degree whereas high scoring defensemen do not. Here is another look at how 5v5 Points/60 regresses to the mean from season to season for forward and defensemen (each bucket contains 150 skaters)…
Regardless of his rate of production in year 1, a defensemen should be expected to finish within 0.60-0.85 points per / 60 at even strength in his subsequent season. That isn’t a large spread. The disparity we see between defensemen who consistently finish with a large sum of points year over year compared to those who don’t is largely the result of opportunity in the form of ice time.
The reality is that the National Hockey League is just starting to scrape the surface of analytics and their application to the decision-making process. A variety of inefficiencies still persist waiting for teams to either exploit or fall victim to and that definitely includes the way the market assesses defensemen. As time progresses along with conceptual understandings, the line between the terms offense and defense will become blurred and thus the evaluation of blue liners will drastically imrpove. For now we wait and pull our hair out as the Dion Phaneufs and Marc Staals of the world sign off on massive contracts and continue to be excessively deployed and relied on to “shutdown” opposing teams’ top lines.