Tuesday, July 22, 2008

Worst 20 On/Off Ice Adjusted +/- Ratings

Yesterday, I posted the top 20 on/off ice adjusted +/- ratings. This is an attempt to look at the sabermetrics and hockey problem of rating an individual player's defensive ability. I discuss the method here and credit Gabriel Desjardins of behind the net for its development.

I cannot argue that this method directly determines the best defensive player in the NHL. It will select players who played well in their roles, where their team played better with them on the ice than with them off the ice. These players will be offensive and defensive stars (most are solid two-way players).

In the comments Mathman criticizes it, claiming it tends to select good players on bad teams. I do not buy this criticism completely. In order to normalize +/- ratings to compare between teams, some assumptions must be made. The assumption made here is that the NHL is a parity-filled league. No team is really bad and no team is really good. The difference between the performance of a player while he is on the ice and and his team's performance while he is off the ice on his team would be roughly constant if a player was used in the same role on a different team. Given the weak link between a player's +/- and his ability (as he is often on the ice for goals he is not directly involved in), I think this is a reasonable assumption. I think it is no more problematic than using +/- as a starting point for this study. Obviously, other normalizations are possible and will give different results, but I do not think these differences are huge (at least in most cases).

The fact that 15 of the 20 players on the top 20 list come from playoff teams is strong evidence that it is not tiled toward bad teams. In fact they are under-represented if this list was a random list of teams. It shows that good teams tend to have good players - which agrees with common sense.

The specific example Mathman points out is Shawn Horcoff of the Edmonton Oilers, who makes the list in a large part because his team was awful when he was not on the ice. Horcoff is the player (who played 50 or more games last year) who has the worst off ice +/- rating. The fact he does not come from a team which can be realistically called the worst in the NHL (Edmonton finished 19th out of 30 teams last year) helps to show the model of a parity-filled NHL is a good one. It is probably not surprising that the player with the worst off ice +/- has a good adjusted rating in a parity filled league. The worst off ice +/- ratings are not all from one or two bad teams. They are spread out throughout several teams in the league. The player who's team played the worst without him on the ice is likely a valuable player to that team and likely to be one of the better players in the league. Of course, there are many questions that can be asked upon looking at the list. How would Shawn Horcoff have done on a better team? How would Milan Michalek have done if he didn't play with Joe Thornton? How would David Perron have done if he was played in a larger role in St Louis? These types of questions cannot be answered easily. They show one of the fundamental flaws in hockey sabermeterics. It is likely impossible to remove a player's contribution from his environment. This is a limiting factor in any kind of sabermetric argument.

Along with the top 20 players we can also look at the 20 worst players. These are players who struggled last season. They did not do well in their roles. In most cases, they neither scored well, nor played defence well. These are players who will likely see reduced roles in the upcoming season (if they don't wind up out of NHL jobs).

Top 20 On/Off Ice Adjusted +/- Ratings 2007/08
RankPlayerTeamOn Ice +/-Off Ice +/-Adjusted +/-
1Craig AdamsChi-1.96+0.74-2.70
2Dallas DrakeDet-1.30+1.16-2.46
3Niclas WallinCar-1.32+1.09-2.41
4Marcel GocSJ-1.66+0.71-2.37
5Radek BonkNas-1.61+0.63-2.24
6Eric GodardCal-1.56+0.59-2.15
7Adam BurishChi-1.48+0.45-1.93
8Mark SmithCal-1.19+0.71-1.90
9Ryan HollwegNYR-1.26+0.59-1.85
10Colton OrrNYR-1.36+0.48-1.84
11Trevor LetowskiCar-1.18+0.57-1.75
12Alexander SeminWas-0.96+0.79-1.75
13Vitaly VishnevskiNJ-1.03+0.67-1.70
14Jerred SmithsonNas-1.20+0.48-1.68
15Kirk MaltbyDet-0.82+0.79-1.61
16Kris DraperDet-0.30+1.27-1.57
17Jarret StollEdm-1.58-0.04-1.54
18Maxim AfinogenovBuf-1.04+0.50-1.54
19Mike FisherOtt-0.80+0.73-1.53
20Jed OrtmeyerNas-1.07+0.40-1.47

A quick look at this list shows that Mathman's complaint about the top 20 list is better suited for the bottom 20 list. This list is primarily filled with players on playoff teams. Those six who did not make playoffs were on some of the better teams that missed the playoffs (two Chicago Blackhawks, two Carolina Hurricanes, one Edmonton Oiler and one Buffalo Sabre). Every player on this list played for a team that had a positive +/- rating when he was off ice except for Jarret Stoll (who was barely a minus at -0.04). This somewhat confirms the talent structure in the NHL. There are a few best players. Most come from the best teams in the NHL. That is why those teams are the best. At the bottom of the NHL talent pool are many players who are roughly interchangeable with one another. Those who are used in tough checking roles on good teams who are not particularly good players will be on the bottom of this list. That is almost the entire list. There are a couple players who are nominally scorers, who did not score particularly well who also make the list. This list identifies some of the players who most struggled in their roles in 2007/08 and thus may see reduced roles in the future.

Craig Adams ( who I had listed as the worst regular in the NHL at one point this season) does the worst on this list. He is followed by the retiring Dallas Drake and a rare defenceman in Niclas Wallin. Marcel Goc and Radek Bonk are next. Bonk is the first player who is nominally a scorer (in that he is not used in a checking role). Eric Godard, Adam Burish and Mark Smith are next. They are followed by New York Rangers Ryan Hollweg and Colton Orr ( who I picked as the worst regular player in the NHL last season). Next up is Trevor Letowski and then Alexander Semin. Semin is probably the best offensive talent on this list, but he fell to a disappointing 42 points last year and has never been considered a defensive talent, even when he is scoring. Vitaly Vishnevski is the second defenceman on the list (joining Niclas Wallin). He is followed by Detroit's Kirk Maltby and Kris Draper. These players were once valuable checkers (Draper won the 2004 Selke Trophy), but time is passing them by. Jarret Stoll is next, followed by Maxim Afinogenov (who is nominally a scorer, but his 28 points were a failure). Mike Fisher and Jed Ortmeyer round out the list.

This list is a group of players who did not have good 2007/08 seasons. Most of them played on pretty good teams, but were used unsuccessfully in checking roles against top competition. These are players who probably should have been used in reduced roles, given their effectiveness last season, but they may not be the worst players in the NHL. This +/- normalization method makes it hard to fall this far if you play on a bad team. There are other ways to look at the problem to try to find such players. I would imagine that the worst players are roughly evenly spread throughout the NHL. Each team's weakest couple players are roughly interchangeable. It is the top players that differentiate good from weaker teams. This list is too heavily slanted toward players on good teams, but I think the worst players in the NHL are likely found uniformly on both good and bad teams. There is nobody on this list that I would suggest was a good defensive player in 2007/08.

Adjusting +/- for on and off ice ratings is a valuable tool to understand the NHL. Like any number it is imperfect and those imperfections must be kept in mind. That said, the top players in their adjusted +/- rating were some of the best players in the role and the worst +/- rating belong to players who struggled in their roles. Their identification is valuable to planning the future of your team.

I should first point out that on/off +/- does not evaluate defensvie ability because it clearly considers GF into the equation which is an offensive ability. If you wanted to look at defense alone, On/Off GA is a better stat.

Second, I agree with Mathman that it still is a flawed stat. The stat does not work well for teams with a broad spectrum of talent level or teams that don't do much line juggling. Detroit is one such team. With Datsyuk, Zetterberg, Holmstrom, Lidstrom and Rafalski they can put out an all star top 5. The Red Wing players who don't play much with those 5 will not do well in the the on/off +/- ice ratings not necessarily because they are bad players, but because they are nowhere near as good as those 5. Not surprisingly there are 3 Red Wing players in the bottom 20. In Draper's case it was mostly influenced by the off ice +/-, not the on ice +/-. This is because he rarely got to play with is star teammates. According to my stats he played 822:45 in even strength ice time but only played with Lidstrom 216:19, Rafalski 188:31, Datsyuk 30:38, Zetterberg 22:07, and Holmstrom 21:49.

I don't want to toot my own horn but to reliably evaluate defensive (or offensive) ability you need to do far more work in isolating that ability. I have attempted to do this with my player ratings (which you can find at stats.hockeyanalysis.com). I looked at how each player performed playing with each of his team mates compared with when he wasn't playing with that team mate. Furthermore, I compared how an opponent played against him and compared it to how he played when not playing against him. Only when you do this do you start to get a more accurate picture of a players ability.

Now, I still don't think my ratings are perfect but I do feel they are better than on/off +/-.

Let's look at the Red Wings and the players noted above. Below are their offensive, defensive and overall ratings.

Lidstrom, 1.42, 1.36, 1.39
Datsyuk, 1.55, 1.29, 1.43
Rafalski, 1.27, 1.26, 1.26
Zetterberg, 1.47, 1.13, 1.31
Draper, 0.69, 1.23, 0.98

Now Draper is hardly a bad player (1.00 is an average player) but compared to his superstar team mates he looks pretty ordinary. Unfortunately he gets penalized big time in the on/off ice ratings because he doesn't get the opportunity to play with those superstar team mates.

Something else that on/off +/- doesn't take into account that my ratings do is special teams play. Kris Draper is Detroit's top penalty killing forward in ice time and Detroit had a pretty good PK unit. If Draper was such a bad defensive player as your stat seems to indicate would Detroit use him as their #1 PK forward and if so would Detroit's PK unit be one of the best in the league?

To finalize let me say that on/off +/- is probably a better stat than straight +/- but it still has significant flaws.
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?