Sunday, August 19, 2007
The Failures of the Goals Saved Method
In my look at sabermetrics and hockey this summer, I have looked at the goals saved method to assess goalies. This method was developed by the hockey outsider (Peter Albert) in an attempt to have one number to evaluate goalies. The idea is to calculate the number of goals that were saved by a particular goalie (when compared to average in a given season) if that goalie had played for the average NHL team. This method is very successful in picking the best goalie in a given season but has problems when comparing between different seasons. This is because the comparison to the average goalie in a given season has no meaningful carry-over from one season to the next. If (for example) expansion occurs, there are more weaker goalies let into the NHL who play on weak teams that allow lots of high quality shots who will adjust the average goalie value downward, and thus make the top flight goalies look better when compared to average with the same performance that would not have been as far above average in a smaller NHL.
We can look at the ten best seasons ever posted by a goaltender according to the goals saved method and we get this list:
There is little reason to believe these are the top 10 seasons that a goaltender has ever had in the NHL (at least since the 1951/52 season - before which sufficient stats do not exist). It is very suspicious that the four "best" seasons ever all occurred between 1973/74 and 1975/76. This is likely because this is the point when the level of the "average" goalie was the lowest. The NHL was undergoing rapid expansion in the 70's and the World Hockey Association was as well. There were more goalies employed in professional hockey than ever before and some were low quality goalies on poor teams who pulled the average level of NHL goalies down - thus making the elite goalies of the time appear better than average by a larger than reasonable amount. After this period, the number of WHA teams began to decline and the NHL didn't expand again until it added the remaining WHA teams in 1979(thus removing the worst goalies from the talent pool) and no more seasons at the top of the goals saved charts were recorded.
A further problem occurs because shot quality data is not taken into account (it does not exist before 2002/03). Thus teams with top defences that allow lower quality shots will appear to have better goalies. It is no surprise that Ken Dryden, playing behind the Montreal defence that included Larry Robinson, Serge Savard and Guy Lapointe, is listed as the best goals saved season ever. He was a very good goalie playing behind a very good defence in a season where the average level of goaltending was depressed due to expansion. All the circumstances were right to overrate this season by this method. Did anyone at the time suggest that was the best season ever for a goaltender? If so, why didn't anyone suggest Ken Dryden for the Hart Trophy? He had been a runner up in the past, but not that season.
After the four years from the mid-70's come two Dominik Hasek seasons in the early 90's that happened in a rapidly expanding NHL. The league had added 5 teams over three seasons and this depressed the average level of goaltending. Hasek's two MVP seasons came later when the NHL was in between expansions and thus do not rank as highly on this list.
Next up is Pete Peeters in 1982/83. He is the first goalie ranked on this list that is not playing in a rapidly expanding era. Does that make his 1982/83 season the best ever by a goalie? He was playing behind one of the best defences in hockey. One that included Ray Bourque, Brad Park, Mike O'Connell and Mike Milbury and a forward unit that included the previous season's Selke Trophy winner in Steve Kasper. This was a great defence that likely allowed only low quality shots.
Next up on the list is Patrick Roy, playing at the beginning of the same expansion era that helped Hasek get his name on this list.
At the end of this list are two Jacques Plante seasons. These are the only seasons from the original six era on this list - where the quality of opposition in the NHL was at its highest - due to that small number of players and teams in the league. His 1961/62 season saw Plante win the Hart Trophy. This is the only season in the top 10 goals saved seasons where the goalie in question was seen as a serious MVP candidate. Although, it is reasonable to question these seasons as well because the Montreal team Plante played behind was a great team with a great defence that included such greats as Doug Harvey, Tom Johnson, Jean-Guy Talbot and JC Tremblay in those years.
It seems that to finish in the top 10 seasons in terms of goals saved, it is necessary to either play in an expansion era or behind a great defence. In order to be at the top of the list it is necessary to do both. While these ten seasons are all good seasons, they seem more like a random list of ten good seasons by goalies then a ranked list of the ten best ever.
The goals saved method is definitely on the right track. It fails with this list of best seasons ever because it does not take into account shot quality. This problem is one that cannot easily be fixed since shot quality data does not exist for most of the seasons in NHL history. It also fails when comparing one season to another because each goalie is referenced to the average in his given season and this average is affected by the quality of goaltending leaguewide (most notably by expansion). In order to compare different seasons, it is necessary to remove a "floating average" that varies from year to year. I am not sure this is possible. Goaltending has changed significantly over the years. For example, a top saves percentage from the 80's (Pete Peeters makes number seven on this list with a .904 saves percentage), would not appear very impressive when compared to today when a goalie can have a saves percentage better than .920 and not make this list. I am not sure how to solve this problem, but it is necessary to compare goalies to a standard that is not so dependant upon the overall goalie quality in a given NHL season (since that varies significantly over time).
When comparing goalies in a single season, the goals saved method is very good. It is not perfect and sometimes over or underrates certain goalies, but it is a good marker to compare goaltenders. When comparing over several seasons, it is far less accurate.
We can look at the ten best seasons ever posted by a goaltender according to the goals saved method and we get this list:
| Rank | Goalie | Team | Year | Goals Saved |
|---|---|---|---|---|
| 1 | Ken Dryden | Montreal | 1975/76 | 73.7 |
| 2 | Bernie Parent | Philadelphia | 1973/74 | 69.5 |
| 3 | Rogie Vachon | Los Angeles | 1974/75 | 63.8 |
| 4 | Bernie Parent | Philadelphia | 1974/75 | 62.1 |
| 5 | Dominik Hasek | Buffalo | 1994/95 | 61.8 |
| 6 | Dominik Hasek | Buffalo | 1993/94 | 58.2 |
| 7 | Pete Peeters | Boston | 1982/83 | 55.5 |
| 8 | Patrick Roy | Montreal | 1991/92 | 54.2 |
| 9 | Jacques Plante | Montreal | 1958/59 | 53.8 |
| 10 | Jacques Plante | Montreal | 1961/62 | 53.1 |
There is little reason to believe these are the top 10 seasons that a goaltender has ever had in the NHL (at least since the 1951/52 season - before which sufficient stats do not exist). It is very suspicious that the four "best" seasons ever all occurred between 1973/74 and 1975/76. This is likely because this is the point when the level of the "average" goalie was the lowest. The NHL was undergoing rapid expansion in the 70's and the World Hockey Association was as well. There were more goalies employed in professional hockey than ever before and some were low quality goalies on poor teams who pulled the average level of NHL goalies down - thus making the elite goalies of the time appear better than average by a larger than reasonable amount. After this period, the number of WHA teams began to decline and the NHL didn't expand again until it added the remaining WHA teams in 1979(thus removing the worst goalies from the talent pool) and no more seasons at the top of the goals saved charts were recorded.
A further problem occurs because shot quality data is not taken into account (it does not exist before 2002/03). Thus teams with top defences that allow lower quality shots will appear to have better goalies. It is no surprise that Ken Dryden, playing behind the Montreal defence that included Larry Robinson, Serge Savard and Guy Lapointe, is listed as the best goals saved season ever. He was a very good goalie playing behind a very good defence in a season where the average level of goaltending was depressed due to expansion. All the circumstances were right to overrate this season by this method. Did anyone at the time suggest that was the best season ever for a goaltender? If so, why didn't anyone suggest Ken Dryden for the Hart Trophy? He had been a runner up in the past, but not that season.
After the four years from the mid-70's come two Dominik Hasek seasons in the early 90's that happened in a rapidly expanding NHL. The league had added 5 teams over three seasons and this depressed the average level of goaltending. Hasek's two MVP seasons came later when the NHL was in between expansions and thus do not rank as highly on this list.
Next up is Pete Peeters in 1982/83. He is the first goalie ranked on this list that is not playing in a rapidly expanding era. Does that make his 1982/83 season the best ever by a goalie? He was playing behind one of the best defences in hockey. One that included Ray Bourque, Brad Park, Mike O'Connell and Mike Milbury and a forward unit that included the previous season's Selke Trophy winner in Steve Kasper. This was a great defence that likely allowed only low quality shots.
Next up on the list is Patrick Roy, playing at the beginning of the same expansion era that helped Hasek get his name on this list.
At the end of this list are two Jacques Plante seasons. These are the only seasons from the original six era on this list - where the quality of opposition in the NHL was at its highest - due to that small number of players and teams in the league. His 1961/62 season saw Plante win the Hart Trophy. This is the only season in the top 10 goals saved seasons where the goalie in question was seen as a serious MVP candidate. Although, it is reasonable to question these seasons as well because the Montreal team Plante played behind was a great team with a great defence that included such greats as Doug Harvey, Tom Johnson, Jean-Guy Talbot and JC Tremblay in those years.
It seems that to finish in the top 10 seasons in terms of goals saved, it is necessary to either play in an expansion era or behind a great defence. In order to be at the top of the list it is necessary to do both. While these ten seasons are all good seasons, they seem more like a random list of ten good seasons by goalies then a ranked list of the ten best ever.
The goals saved method is definitely on the right track. It fails with this list of best seasons ever because it does not take into account shot quality. This problem is one that cannot easily be fixed since shot quality data does not exist for most of the seasons in NHL history. It also fails when comparing one season to another because each goalie is referenced to the average in his given season and this average is affected by the quality of goaltending leaguewide (most notably by expansion). In order to compare different seasons, it is necessary to remove a "floating average" that varies from year to year. I am not sure this is possible. Goaltending has changed significantly over the years. For example, a top saves percentage from the 80's (Pete Peeters makes number seven on this list with a .904 saves percentage), would not appear very impressive when compared to today when a goalie can have a saves percentage better than .920 and not make this list. I am not sure how to solve this problem, but it is necessary to compare goalies to a standard that is not so dependant upon the overall goalie quality in a given NHL season (since that varies significantly over time).
When comparing goalies in a single season, the goals saved method is very good. It is not perfect and sometimes over or underrates certain goalies, but it is a good marker to compare goaltenders. When comparing over several seasons, it is far less accurate.
Comments:
<< Home
Could you change your "line in the sand" by decade? By that I mean maybe you could take the 1967-1979 era of expansion and WHA and have a number reflective of that era, and then the 1995-2004 number can reflect the climate of the game at that time.
Good suggestion. I agree that I should change the average or threshold save percentage for different eras. I'm not sure how to do that yet.
One possibility is to look at the z-score (http://en.wikipedia.org/wiki/Standard_score). This might be a better solution because a goalie who earns a high level of GS in a low-quality league will have a lower z-score (and vice versa).
One possibility is to look at the z-score (http://en.wikipedia.org/wiki/Standard_score). This might be a better solution because a goalie who earns a high level of GS in a low-quality league will have a lower z-score (and vice versa).
# posted by 
: August 19, 2007 6:03 PM
: August 19, 2007 6:03 PM
Interesting post. Since goalie statistical analysis is what I do almost exclusively on my blog, I had a couple of comments.
First, would it be possible to use replacement level as the threshold, rather than the league average? This might give a better measure of value because there is some value provided by a goalie that plays lots of minutes at a league average level. Also, replacement level won't vary as much with expansion and talent dilution. Because of the relatively few number of goalie positions available in the NHL and the large number competing for those spots, there will always be a large group of mediocre goalies available and ready to step into an NHL job if given the chance. If expansion causes a number of minor-league goalies to enter the league, that will drag down the league average but the replacement level should still be similar.
There have been some attempts to pin down an estimate of replacement value, it's not as easy as figuring out the average, but I think it would be worth it in terms of getting a more accurate estimation of goalie value.
Secondly, every basic goalie stat is strongly team dependent, so there needs to be some adjustment for teams. As you point out, though, this is difficult since we don't have shot quality info. My proposed solution, discussed on my blog, is to look at the results of other goaltenders on the team to try to factor in shot quality. It won't be perfectly precise, but it should work based on the rationale that goalies who outperform their teammates are valuable while goalies who do not are easily replaced, regardless of where the league average is.
First, would it be possible to use replacement level as the threshold, rather than the league average? This might give a better measure of value because there is some value provided by a goalie that plays lots of minutes at a league average level. Also, replacement level won't vary as much with expansion and talent dilution. Because of the relatively few number of goalie positions available in the NHL and the large number competing for those spots, there will always be a large group of mediocre goalies available and ready to step into an NHL job if given the chance. If expansion causes a number of minor-league goalies to enter the league, that will drag down the league average but the replacement level should still be similar.
There have been some attempts to pin down an estimate of replacement value, it's not as easy as figuring out the average, but I think it would be worth it in terms of getting a more accurate estimation of goalie value.
Secondly, every basic goalie stat is strongly team dependent, so there needs to be some adjustment for teams. As you point out, though, this is difficult since we don't have shot quality info. My proposed solution, discussed on my blog, is to look at the results of other goaltenders on the team to try to factor in shot quality. It won't be perfectly precise, but it should work based on the rationale that goalies who outperform their teammates are valuable while goalies who do not are easily replaced, regardless of where the league average is.
michael kors outlet, christian louboutin outlet, oakley sunglasses, tiffany and co, nike air max, longchamp outlet, ugg boots, jordan shoes, gucci outlet, uggs on sale, polo ralph lauren outlet, ugg boots, burberry outlet online, nike air max, polo ralph lauren outlet, ray ban sunglasses, ray ban sunglasses, louboutin shoes, michael kors outlet, tiffany jewelry, michael kors, ugg boots, burberry, chanel handbags, oakley sunglasses, louis vuitton outlet, louboutin, prada handbags, longchamp, louboutin outlet, louis vuitton, replica watches, louis vuitton outlet, oakley sunglasses, oakley sunglasses, ugg boots, nike free, michael kors outlet, prada outlet, kate spade outlet, louis vuitton, nike outlet, michael kors outlet, cheap oakley sunglasses, replica watches, ray ban sunglasses
nike air max, north face, true religion jeans, longchamp pas cher, ralph lauren pas cher, mulberry, vans pas cher, hollister, michael kors, timberland, sac longchamp, michael kors, oakley pas cher, hermes, air max, ray ban pas cher, coach factory outlet, hogan, burberry, ralph lauren uk, true religion jeans, air jordan pas cher, michael kors, abercrombie and fitch, nike roshe, lacoste pas cher, north face, air force, tn pas cher, nike air max, nike air max, nike blazer, kate spade handbags, louboutin pas cher, nike free run uk, coach purses, michael kors, lululemon, nike roshe run, converse pas cher, coach outlet, hollister pas cher, true religion jeans, new balance pas cher, coach outlet, vanessa bruno, true religion outlet, ray ban uk, sac guess, nike free
timberland boots, insanity workout, beats by dre, p90x workout, ghd, converse, louboutin, baseball bats, nike trainers, north face outlet, asics running shoes, lululemon, celine handbags, jimmy choo shoes, soccer jerseys, mont blanc, oakley, north face outlet, vans, soccer shoes, nike huarache, giuseppe zanotti, chi flat iron, mcm handbags, nfl jerseys, herve leger, ralph lauren, longchamp, babyliss, valentino shoes, hollister, nike air max, mac cosmetics, abercrombie and fitch, vans shoes, bottega veneta, reebok shoes, iphone 6 cases, hollister, hollister, nike roshe, gucci, new balance, wedding dresses, ray ban, ferragamo shoes, birkin bag, instyler, nike air max, converse outlet
toms shoes, ugg boots uk, moncler, supra shoes, pandora jewelry, swarovski, doudoune canada goose, bottes ugg, karen millen, coach outlet, barbour jackets, moncler outlet, juicy couture outlet, canada goose, louis vuitton, marc jacobs, moncler, doke gabbana outlet, canada goose outlet, pandora charms, michael kors outlet online, louis vuitton, juicy couture outlet, barbour, canada goose, canada goose outlet, ugg pas cher, canada goose, lancel, wedding dresses, hollister, louis vuitton, links of london, moncler, thomas sabo, moncler, moncler, sac louis vuitton pas cher, canada goose uk, moncler, louis vuitton, michael kors handbags, canada goose, ugg,ugg australia,ugg italia, replica watches, pandora charms, michael kors outlet, swarovski crystal, pandora jewelry, ugg,uggs,uggs canada, montre pas cher, moncler
Post a Comment
<< Home
