Tuesday, August 12, 2008

On/Off Ice Adjusted Goals For Average

In order to look at the biases of the sabermetric problem of adjusting +/- ratings using the on/off ice method, I thought I would apply it to a simpler problem. That of goals for average (GFA) (namely how many goals are scored per 60 minutes a player plays in 5 on 5 situations). As we have seen, the goals for average leaders are among the best offensive players in hockey and the worst GFA players are rarely significant NHL players. By subtracting the off ice GFA from a player's on ice one we should get a better look at the offence a player brings independent of his team (if there is value to on/off ice adjusted +/- ratings). The potential problem with this line of logic comes from the assumption that both offence and defence benefit from the on/off ice adjustment even though offence is far less team dependant than defence. It is possible that there are few team effects to remove from individual offensive numbers when compared to defensive numbers and thus defensive numbers are the only place where the adjustment makes sense. However, as +/- rating are made up of both offensive and defensive numbers, looking only at the offensive numbers is a simplified system to see that the adjustment provides meaningful results. Let's see what the results look like.

Here are the top ten on/off ice adjusted GFAs among players who played 50 or more games in 2007/08:

Here are the ten best goals for averages (among players who played 50 games last season):

Top 10 Individual On/Off Ice Adjusted Goals For Averages 2007/08
RankPlayerTeamGFARank Unadjusted
1Dany HeatleyOtt+2.471
2Sidney CrosbyPit+2.295
3Jason SpezzaOtt+2.222
4Ryan GetzlafAna+2.2019
5Pavel DatsyukDet+2.144
6Paul StastnyCol+2.113
7Alexander OvechkinWas+2.048
8Corey PerryAna+1.9832
9Jarome IginlaCal+1.8713
10Evgeni MalkinPit+1.8410


We are subtracting the goal scoring rate when a player is off ice from the player is on the ice. This is intended to see the effect the player has independent of his team. Does it? There are not many significant changes on the list by subtracting off ice goals for. The biggest change is the addition of two Anaheim players in Ryan Getzlaf and Corey Perry. Their climb is significant. I think this is because Anaheim has a low scoring dedicated checking line and Getzlaf and Perry benefit from their comparison to players who rarely score. The unusual Anaheim checking line makes simple sabermetric calculations more difficult.

If we ignore Anaheim players, does this list look improved as a result of the on/ice adjustment? Gone from the previous top 10 are Henrik Zetterberg (who is now 11th), Jason Arnott (who is 12th) and Derek Roy (who is 18th). In are Jarome Iginla and the Anaheim pair of Getzlaf and Perry. Neglecting the Anaheim pair and thus adding Zetterberg and Arnott to the list, only one player makes this list that did not before (Iginla) and only one player is dropped (Derek Roy). That is a small change, but one that makes sense to any reasonable hockey fan. As for changes within the top 10, the biggest rise is Sidney Crosby who moves from 5th up to 2nd. Again, this move is one that seems reasonable. Its clear Crosby is a very important offensive player and an offense ranking system that sees that is a good one. The biggest drop who remains in the top 10 (neglecting Getzlaf and Perry) is Jason Arnott. That seems reasonable. He is not expected to be as big a part of his team's offence by most fans as his top 10 rank would show. The biggest drop among top players is Derek Roy. Roy is another player who ranks surprisingly high in GFA and (like Arnott) does it playing on a balanced team that does not overly rely on one offensive line. As a result, the off ice GFA is strong enough to drop him on the list (this is despite the fact both Roy and Arnott do not play on as strong teams as many of the players ranked ahead of them). I would argue that from looking at the top 10 on/off ice adjusted GFAs, this list is closer to a list of the best offensive players in the game than the unadjusted GFA list, as long as we take into account the complication of Anaheim's checking line. There is no huge change between either list, so this is not exactly a strong convincing argument, but it is suggestive of being on the right track.

Now let's look at the 10 worst on/off ice adjusted GFAs. Here they are:

Worst 10 Individual On/Off Ice Adjusted Goals For Averages 2007/08
RankPlayerTeamGFA Rank Unadjusted
1Ryan HollwegNYR-2.141
2Craig AdamsChi-2.0214
3Jarred SmithsonNas-2.0210
4Dallas DrakeDet-1.957
5Mark SmithCal-1.928
6Colton OrrNYR-1.912
7Jed OrtmeyerNas-1.8724
8Eric GodardCal-1.8111
9Jeremy ReichBos-1.693
10Sami KapanenPhi-1.6513


The bottom list is more shuffled than the top is, which is expected because there is more differentiation between the best players in the NHL than the worst (in general the worst players in the league are roughly interchangeable with one another). The biggest surprise is that nobody from the Anaheim checking line makes this list, especially given Travis Moen having the 4th worst GFA unadjusted (he is 11th so he barely misses this list). Players on teams that are higher scoring than Anaheim tend to rise on this list (to worse rankings) and players on lower scoring teams end to have rankings improve. That is the point of adjusted rankings. Team effects are being taken into account. Four players make this list that do not make the unadjusted list. They are Craig Adams, Jed Ortmeyer, Eric Godard and Sami Kapanen. All play on relatively high scoring teams. Other players on higher scoring teams such as Detroit (Dallas Drake) and Calgary (Mark Smith) find themselves moving up the futility list. Players on lower scoring teams tend to improve their rankings (getting less bad). Examples of this are Colton Orr, rising from 2nd to 6th and Rodney Pelly who was 5th and is now 14th. I would argue that if we are looking for offensive talent (or lack of it) these adjusted lists are better than the unadjusted lists.

In order to test the effect of the on/off ice adjustment to +/- that I have looked at in several posts this summer, I decided to look at a simpler system. Goals For Averages (GFA) is this simpler system. Defensive contributions to +/- are removed. Since defence is harder to measure than offence this makes a simpler system. For the most part, lists with the on/off ice adjustment are closer to what I might expect as a ranking of top (or bottom) offensive players in the league. There is one problem introduced by this method. The problem occurs in Anaheim. Anaheim is a low scoring team. They have a low scoring checking line made up of Travis Moen, Rob Niedermayer and Sami Pahlsson who get significant ice time. This appears to give a large boost to Anaheim's scorers due to a low off ice GFA. This pushes Ryan Getzlaf and Corey Perry further up an adjusted list than they otherwise should be. Clearly, this adjustment is not perfect but it does appear to improve these offensive numbers by a moderate amount.

Comments:
Clearly, this adjustment is not perfect but it does appear to improve these offensive numbers by a moderate amount.

Why does the adjustment appear to improve the offensive numbers? Because you believe Getzlaf, Perry and Iginla to be better offensvie players than Zetterberg, Arnott and Roy?

You have a system in which you looked at the top 10 players and found that (at least) 20% of them are likely to be unfairly boosted while the remaining 80% don't really change much from the stat you think is in need of adjustment. How do you come to the conclusion that the adjustment "does appear to improve these offensive numbers by a moderate amount" from that?
 
David

Perhaps the answer to your question is in the blogpost. I recommend you read it before asking me to restate the ideas in a comment.

Nevertheless, the adjustment does things that make sense. It boosts Sidney Crosby and Jarome Iginla while dropping Derek Roy and Jason Arnott. It removes the bias where players on low scoring teams are preferentially chosen as the worst GFA players inserting more teams into the race.

It is by no means a 100% ironclad defence of the on/off ice adjustment, but the changes it makes seem to make sense for the most part. The exception is among the Anaheim forwards, which can be explained by an already noticed anomaly. If we figure out how to properly treat the Anaheim checking line (which is another issue entirely), the adjusted results look very good.
 
It boosts Sidney Crosby and Jarome Iginla while dropping Derek Roy and Jason Arnott.

The point I was trying to make is you are just using your personal opinion and/or common perception (which may or may not be accurate) on a small sample size to evaluate a system which clearly has flaws. That is not a very solid basis for anything.
 
David

I think the point you are trying to make is a philosophical one that goes well beyond the scope of this blog.

You basically asked how do I know that my perception of reality (or hockey in this case) is accurate or how do I know that the common perception of reality is accurate.

I know because the results are largely repeatable. It is independent of observer. If somebody watches a Pittsburgh game they will come away with the impression that Sidney Crosby is a good offensive player. It doesn't matter who watches. It is a logically consistent reality. It really comes down to inductive reasoning.

I adjusted the entire season of 5 on 5 data for the whole league (small sample size? how many seasons make a sufficient sample size in your opinion?) and got results which are not significantly changed but those changes make sense and fit with what I already know about hockey. So it seems like it was a reasonable thing to do.
 
By small sample size I mean that in the argument you presented to us you just looked at the top 10 and bottom 10 players.

Anyone with half a brain can tell you that Crosby, Ovechkin, Datsyuk, Spezza, etc. are among the top offensive players in the NHL. Whether Crosby is first or fifth really isn't what is interesting. But I am also confident in saying that 54 point Corey Perry isn't a better offensive player than Malkin, or Zetterberg, or Iginla, or any number of other players.

What is interesting (and more difficult to answer) are questions like, how good of an offensive player is Ryan Malone or who is the better defenseman, Ron Hainsey or Jeff Finger. If a significant portion of your sample size (that you presented to us) is showing a significant over adjustment, how am I to know that Jeff Finger isn't over adjusted or under adjusted. Unlike the top players in the NHL, I don't have a feel for how good or mediocre or bad Jeff Finger is. Nor do I watch enough Avalanche games to get an idea of who he plays with and whether his rating might be biased and by how much.

If you really want to test your system then the proper way to do so is to compare the same player over multiple years. Most players ratings should be more or less the same from year to year. If a large number of players are seeing a significant swing in their 'rating' from one year to the next, then it is an unreliable system. If the year to year correlation is fairly good then you may have something.
 
David

You major fallacy is that this is my system. It isn't. Credit belongs to Gabriel Desjardins of behindthenet.ca. This is a system which I think is good. It is logical and gives results that tend to make sense. It is by no means a final answer. I am trying to show this as I have gone along. I have pointed out flaws as well as strengths.

On the behindthenet website you can see numbers for this season and last, although there are some weird things in last year's stats (for example it lists Peter Forsberg as having played 40 games with Philadelphia - which he did but he also played 17 games with Nashville).

Once again THIS IS NOT MY SYSTEM
 
Nevertheless, this is a well established website. Its results are regularly linked to by websites like James Mirtle, LoweTide, Hockey Analytics and The Hockey News.
 
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?