Thursday, July 24, 2008

My Sabermetric Philosophy

Since summer is underway and hockey news is scarce I have embarked upon a series of posts on sabermetrics and hockey. So far this summer, I have been looking at +/- ratings and what information can be gathered from them, with an eventual goal of trying to rate individual defensive performances. When presenting simple models to interpret +/- ratings I have received comments here and here that point me toward more complex linear regression type models to try to determine these things.

I would argue that these models are not significantly better than those that I am presenting. They are more mathematically complex and may be satisfying to those who enjoy "exact solutions" (in as much as anything that comes from flawed hockey statistics can be exact), but it is hard to draw any more information from them than can be gathered from the more simple models I am presenting. Any model should be as simple as necessary to produce the results we are looking for (but no simpler).

One must keep in mind that the input parameters (the statistics) are flawed. Most events (for example goals) are only weakly correlated with all the players on the ice. When most goals are scored, some players on the ice were not directly involved. There will be a huge amount of statistical noise inherent in any numbers which can create significant error in even the most "exact" calculations.

Further, a complex linear regression model obscures the process with which we analyze the statistics. It becomes a "black box" into which we input our statistics and the computer crunches numbers to give us a solution. The question of how valid that solution is and where things may be distorted is not so easy to answer as in the case of a simpler model. At the same time, a good simpler model will give essentially the same results. It is more instructive and time saving to stick to the simpler models as long as they are reliable.

As examples of that philosophy, here are some facts about the NHL last season that the on/off ice adjusted +/- ratings have shown which I believe are reliable conclusions (despite not being widely held among NHL fans). With a much more complex linear regression model you would get the same conclusions out with less intuition about why you obtained that result.

Conclusion 1: Johnny Oduya is a very good defenceman who is a key to the success of the New Jersey Devils.

Conclusion 2: David Perron and Michel Ouellet are two players who performed very well last season in protected roles against weak competition and should have their roles increased.

Conclusion 3: Paul Stastny, Ryan Getzlaf, Jason Arnott and Shawn Horcoff all had strong all star level performances and are much more valuable to their respective teams than most fans realize.

Conclusion 4: Kris Draper and Kirk Maltby are no longer as valuable players as most fans believe. Their even strength performance could be replaced by the Detroit Red Wings without too much trouble (and likely with a cost savings).

Conclusion 5: Niclas Wallin had an awful season and was probably the worst defender in the NHL last season. He definitely does not belong in the NHL any longer (except possibly in a bit role).

These are not statements that are clear to most fans and I am sure that if you (for example) posted them in the appropriate place on hfboards could get people to call you crazy for making them, but they are clearly born out by sabermetric analysis. These conclusions (and others) come from relatively simple analysis and I don't think much more is gained (and some transparency is lost) by using a more complex model that requires some computation time to solve.

Comments:
These are not statements that are clear to most fans and I am sure that if you (for example) posted them in the appropriate place on hfboards could get people to call you crazy for making them, but they are clearly born out by sabermetric analysis.

Are there? They are born out by your analysis, but as I and others have pointed out, your analysis is flawed. All your analysis concluded is how a player rates as compared to other players on their own team and one could argue that it doesn't even do a good job at that because not all players play in the same situations, with the same linemates and against the same opponents.




One must keep in mind that the input parameters (the statistics) are flawed. Most events (for example goals) are only weakly correlated with all the players on the ice. When most goals are scored, some players on the ice were not directly involved. There will be a huge amount of statistical noise inherent in any numbers which can create significant error in even the most "exact" calculations.

I agree, but even with all that noise you can find meaningful patterns and with enough data the results will trend towards the truth. Maybe Player A gets unlucky and has a goal scored against just as he steps on the ice, but over the course of a season the few 'unfortunate' situations averaged out.

Your model and my model agree a fair amount but where your model falls apart is at the extremes. Very good teams like Detroit or mediocre teams like the Oilers. Why should Draper get penalized simple because the coach does not put him on the same line as Datsyuk and Zetterberg as he does with Holmstrom? Why should Draper get compared with Zetterberg and Datsyuk while a similar player in Mike Sillinger gets compared with Mike Comrie and Bill Guerin? If we put Horcoff on the Red Wings and have him play on the second line behind Zetterberg, Datsyuk, and Holmstrom would his rating be near as good. Probably not. Not because he is any worse, but because the players he would be compared to (i.e. his teammates) are much better. That amounts to a flawed system.
 
David

I knew you would complain about the conclusion about Kris Draper and Kirk Maltby - even though it is a valid one.

Of course there are some flaws in the analysis. There are flaws in any analysis (and that includes yours).

I suspect your flaw is an inability to remove team effects from the players and Draper and Maltby are rated higher than they should be because Detroit tends to outscore their opponents except when Draper and Maltby (and a couple other players) are on the ice. I have to say I suspect that is the problem because your system lacks the transparency to check. I cannot find step by step instructions to reproduce your numbers anywhere on your site. I am sure if they did exist, it would be far more complex and far harder to see its inherent flaws.

If we look at these numbers it is clear that Draper and Maltby do not play against the toughest opposition that the Red Wings face, yet they still allow more goals than they face on a strong team.

I think it is really interesting that other teams like Pittsburgh (who have superstars in Malkin and Crosby to compare to - players who many rank above Zetterberg and Datsyuk in ability) do not find their third or fourth line checkers have nearly as bad adjusted +/- ratings as Draper or Maltby do. I think it interesting that the other teams that had top regular season +/- ratings (Ottawa, Anaheim...) also do not have players who rate as poorly as Draper and Maltby do. That says something about Draper and Maltby. They are no longer as good as you think they are.

I think that is a safe conclusion. Detroit could replace Maltby and Draper's even strength performance without huge trouble. That said, that doesn't seem to be something they are doing. I suspect that it is because they think it is worth keeping them around. It is worth it because they are seen as a classy organization where people (ie Marian Hossa) want to sign. They look after their players wo have given their heart and soul to the franchise. They dont turf them out once they stop being as useful as they once were (see for example Toronto as an example of why that doesn't work). Its the same reason they brought back Darren McCartzy this year, even though he had essentially no value whatsoever. A player knows if he is loyal to the Red Wings they will be loyal to him. That is worthwhile for them. They can stand overpaying Draper and Maltby after their value has significantly decayed if it lands them Marian Hossa and it gives them a chance to resign Henrik Zetterberg for next year at below open market value.
 
There's something to be said for "fan favorites" as well. The thinking could be that as long as the Wings are winning, they can afford to have fan favorites stay on the roster, even if they are somewhat overpaid and somewhat underperforming.

Butts land in the seats with their Draper and Maltby jerseys, they keep winning. It's the Circle of Life.
 
I think it is really interesting that other teams like Pittsburgh (who have superstars in Malkin and Crosby to compare to - players who many rank above Zetterberg and Datsyuk in ability) do not find their third or fourth line checkers have nearly as bad adjusted +/- ratings as Draper or Maltby do.

Maybe it is because Crosby and Malkin are not good defensive players, or for that matter offensive players.

Crosby: GAON/60:2.51, GAOFF/60: 2.01

Malkin: GAON/60: 2.71, GAOFF/60:1.87

Those two give up far more goals when they are on the ice than their teammates do. Unlike Datsyuk and Zetterberg, they aren't great defensive players. Datsyuk and Zetterberg give up about as many goals as their teammates do. In fact the comparison isn't even close. Not only do Datsyuk and Zetterberg score at a higher rate than the Penguin duo, they give up far fewer goals too. You just cannot make that comparison.

The same can be said for Heatley, Spezza and especialy Alfredsson.

As for Anaheim, they have several players rated below -1.00 including Moen, Pahlsson, Beauchemin, and Scott Niedermayer but they weren't quite bad enough to make your top 20 list. Anaheim also only had one player in your top 20 list where as Detroit had 3 so the imbalance is not so great.

Ottawa is an interesting case in that Daniel Alfredsson is a negative player where as Heatley and Spezza are large positive players. Care to explain how Alfredsson, with the third highest +/- on the team at +15 and the the 9th leading scorer in the league despite missing 12 games is a negative player?
 
David

That seems like you want me to defend a bunch of things that I am NOT stating as conclusions instead of discussing Kirk Maltby and Kris Draper.

Yes Pittsburgh, Anaheim and Ottawa have players who are not so good at even strength. None are as bad as Draper or Maltby come out (and there may be other circumstances we are not talking about yet in some cases). That says something about Draper and Maltby not being as strong at even strength. You cannot accept that.

And you actually make the claim that Crosby and Malkin are not good defensive players, or for that matter offensive players. which is ridiculous.

You will argue anything crazy to try to support your point even if its patently false.
 
Yes Pittsburgh, Anaheim and Ottawa have players who are not so good at even strength. None are as bad as Draper or Maltby come out (and there may be other circumstances we are not talking about yet in some cases). That says something about Draper and Maltby not being as strong at even strength. You cannot accept that.

I do not for one second believe that Draper is one of the worst 20 players in the NHL. The problem with your stat is you are comparing Draper to his team mates which include a bunch of superstars (i.e. when Draper is not on the ice, Datsyuk, Zetterberg, Holmstrom, etc. are). Of course when you compare Draper's production to theirs he won't look very good.

The problem is, you are not comparing Mike Sillinger's production to Detroit's superstars either. You are comparing his production to Comrie, Satan, Guerin, etc. Sillinger's comparables are much weaker than Draper's so it is an unfair comparison.

And you actually make the claim that Crosby and Malkin are not good defensive players, or for that matter offensive players. which is ridiculous.

What I meant to say is in comparison with Datsyuk and Zetterberg.

Malkin: GFON/60 = 3.74
Crosby: 3.92
Datsyuk: 4.04
Zetterberg: 3.77

Crosby/Malkin trail Datyuk/Zetterberg slightly in offense and significantly in defense. Offensively the Pittsbrugh duo trail Spezza, Heatley, Forsberg and Sastny while Malkin also trails Arnott, Ovechkin, and Derek Roy.


Maybe I am missing something but when you compare a player to his team mates, if one players team mates are much better than anothers (as Draper's are vs. Sillinger's), then you have a hard time comparing those players.
 
Once again I am not claiming or concluding Kris Draper is one of the 20 worst players in the NHL. If you would like to argue, argue with what is claimed.

I think the source of your ridiculous claim that Evgeni Malkin and Sidney Crosby are not ver good offensive or defensive players is the same source of your unwillingness to accept that Draper and Maltby have relatively easily replaceable contributions at even strength. Its an inability to decouple team effects from that of an individual player.
 
Once again I am not claiming or concluding Kris Draper is one of the 20 worst players in the NHL. If you would like to argue, argue with what is claimed.

I guess I misinterpret what you meant to imply when you

1. Show a rating system which has Draper in the bottom 20

2. Defend that rating system as valid.

3. Make the claim that Draper's even strength performance can easily be replaced, a conclusion I can only assume you arrived at based on Draper being int he bottom 20.

4. Fail to recognize the rating systems flaws as a possible reason why Draper shows up so poorly.

I think the source of your ridiculous claim that Evgeni Malkin and Sidney Crosby are not ver good offensive or defensive players is the same source of your unwillingness to accept that Draper and Maltby have relatively easily replaceable contributions at even strength.

My source is the Behind the net website which shows that Malkin has a 2.71 GAON/60 which is probably in the bottom third of the league and Crosby at 2.51 isn't a whole lot better. Compared to Datsyuk and Zetterberg and many other top players that isn't very good. I explained what I meant by the offensive claim which you failed to recognize.
 
I guess I misinterpret what you meant to imply when you

1. Show a rating system which has Draper in the bottom 20

2. Defend that rating system as valid.


Obviously you do misinterpret things. I show you one stat. If you think that is the same as a definitive ranking of the players in the NHL you entirely lack reading comprehension skills.

David Perron comes number one in the league. Do you think I am claiming David Perron is the best player in the NHL? If you think that, its a far bigger strawman argument for you to make than this Kris Draper one. You should be spending your time arguing against David Perron and not Kris Draper, because Perron is clearly further from the best player in the NHL than Draper is from 16th worst position player who played 50 or more games last year. The fact you are not arguing that implies to me that you do understand the situation, or you did before you started arguing. Once David Johnson starts arguing all rationality goes out the window.

It should be obvious that I do not think David Perron is the best player in the NHL. It should be equally obvious that I am not saying Kris Draper is one of the 20 worst players in the NHL for coming in the bottom 20 in a given stat.

That said, the adjusted +/- stat is a good one. Like every number it has flaws, but it makes a clear enough case that Kris Draper's even strength performance in 2007/08 was not that great. That performance could likely be replaced by the Detroit Red Wings with a cost savings.
 
I could have picked on Perron or any number of other players to express my point, but Draper was one of the players you pointed out so is one that I focused my attention on. He is probably also one of the players that would be most negatively affected by the inherent shortcomings in the statistic.

I guess my beef is that the stat is flawed and yet you used the stat to draw conclusions about certain players like Draper. When a stat/rating system is flawed/biased, and you draw conclusions from that rating system (i.e. that Draper is no longer that valuable and could be easily replaced) without recognizing the flaws/bias in the rating system that may incorrectly or unfairly lead to that conclusion, you are not being thorough in your analysis and in essence are misleading your readers.
 
I have been very openly showing the flaws in the system. I discussed them alongside the method and its results. I am also showing that you can draw useful conclusions from it despite any flaws (this is true of all statistics - they have flaws and can lead to useful conclusions).

The conclusion that Kris Draper and Kirk Maltby's even strength contribution to the Detroit Red Wings is something the Wings could replace at considerable cost savings is a good one that is born out by the analysis. It is not one that is readily accepted by all hockey fans (you clearly prove that point). That shows the value in this analysis. It can demonstrate some facts about the NHL that are not clear without the analysis.

What this doesn't show is that Kris Draper is one of the 20 worst players in the NHL or that David Perron is the best player in the NHL. Both of those conclusions are clearly false. That is the level of conclusion you are arguing against. That is not the level of conclusion I am making. That is why you want to disagree with me. You are not allowing yourself to see my argument. You are arguing against some ridiculous stuff I am not saying.
 
The conclusion that Kris Draper and Kirk Maltby's even strength contribution to the Detroit Red Wings is something the Wings could replace at considerable cost savings is a good one that is born out by the analysis.

But is it? To really show that it is, you need to show that the negative impact towards Draper's rating due to the fact he plays on a near all-star team is not that large and that his rating more or less reflects Drapers production level. To me you have not proven that. It is my claim that if Draper played for the Islanders his rating would be more than respectable.

In fact, let's do a simple example.

Draper on Detroit:
GFON/60: 1.51
GAON/60: 1.81
ON +/-: -0.30

Let's assume Draper played for the NY Islanders. By moving him to the Islanders, let's assume both his offensive and defensive numbers get worse, significantly in the case of GA/60.

Draper on Islanders:
GFON/60: 1.40
GAON/60: 2.60
ON +/-: -1.20
Of Islander forwards had the Islanders highest +/-OFF/60 so lets use Sillinger's off ice numbers as Draper's off ice numbers on the Islanders.

Draper on Islanders:
GFOFF/60: 2.15
GAOFF/60: 2.29
+-OFF/60: -0.14

Those numbers would give Draper a rating of rating of -1.06.

Wow. Even though I penalized Draper fairly significantly on the move to the Islanders, and gave him a near worst case scenario in terms of off ice numbers, his rating jumped from -1.58 to -1.06.

Now -1.06 isn't great either but it is a drastic improvement and an example of how the team one plays for can dramatically affect the statistic.

BTW, the theoretical GF/60 I gave Draper would put him dead last on the Islanders by a measurable margin and the theoretical GA/60 would put him as the Islanders 5th worst forward behind Comrie, Sillinger, Hunter and Park. The point I am trying to make is even if we assume a worst case scenario Draper would see a significant improvement. That should give you an indication of how much he is penalized just because he plays on the Red Wings.
 
You are not making much of a point. You made up some numbers and showed that even with them, Draper isnt all that good with this stat. Which would verify the idea that the Red Wings could replace Draper's even strength contribution.

I also argue the point that Detroit is a "near all star team". There is too much parity in the league to allow any such team to form. Sure Detroit won the cup, somebody had to and they were the best team. But they are not a near all star team. There is no such thing in the NHL. There are several teams with a few all star players. Detroit is merely one.

The fact that Draper and not (for example) Daniel Cleary or Tomas Kopecky show this poor +/- performance when both play against roughly the same calibre of opposition (give or take) and is no all star himself is my point. There are examples of non-all star Red Wings who play regularly and have far better numbers than Draper. That should have you questioning Draper.
 
One flaw of the on/off ice adjusted */- is that the worst players in the ranking tend to come from the top and middle of the NHL teams and not the bottom ones. The off ice performance of the bad teams is bad. It is hard to be bad enough on ice to get you adjusted +/- into the 20 worst.

So, if your point is that you could hide Kris Draper on a team like the New York Islanders and he would not be found among the worst 20 players despite his not playing any better than he did in Detroit this year, I would accept it. That doesn't make his even strength performance this year any better.
 
The fact that Draper and not (for example) Daniel Cleary or Tomas Kopecky show this poor +/- performance when both play against roughly the same calibre of opposition (give or take) and is no all star himself is my point.

What if Maltby was the problem. What if Maltby was the bad defensive player and not Draper. What if Maltby is pulling down Drapers numbers. Maltby played 412:59 with Draper, 177:16 with Cleary and 97:16 with Kopecky.

Or maybe it is Dallas Drake who played 252:27 with Draper, 147:34 with Kopecky, and 89:14 with Cleary.

And that is the problem with your stat. You have not done a good job isolating Draper from Maltby or Drake or accounting for the fact he doesn't play at all with Detroit's top players. If Maltby and Drake suck, Draper unfortunately gets dragged down with them only made worse by the fact he plays on a team with several all-stars, of which he hardly gets to play with.
 
Its true Drake is worse than Maltby who is worse than Draper.

But none of them provide anything to the Red Wings at even strength that could not be replaced without a big effort. Which once again is the point I am arguing.
 
You two are too much.

Personally, I think there's a fair amount of "so what?" to on-ice/off-ice stats myself. I really don't know what value I put on knowing how well Getzlaf's +/- rate compares against Pahlsson's -- hopefully it's better, but I don't know if the ratio tells me very much, especially as a comparative statistic to other players in the league.

I'd much rather take a look at Getzlaf's rates compared to Selanne's rates from the year before -- at least there should be more similarities in terms of quality of icetime and quality of teammates (Bertuzzi notwithstanding). I haven't run the numbers myself, but I suspect that the "important" Getzlaf was still a downgrade defensively from what the Ducks had on the top line their cup season.

Overall, I think you guys are generally on the right track -- context of results matters, as do per-minute rates. Still, I think the on-ice/off-ice thing tries to pull too much into one easy set of numbers (how one player does, how his linemates do, how his opposition does, how his non-linemates do, how their opposition does), and by the time it's easily presentable, it loses a lot of its intended meaning.

No matter the numbers, Carlyle and I will still call Pahlsson Anaheim's defensive stud, and he'll continue to face off against the opposition's best scorers. His results are best compared against other such forwards (Pandolfo and friends), but frankly, I don't know what value there is in comparing them against Getzlaf & Perry.
 
But none of them provide anything to the Red Wings at even strength that could not be replaced without a big effort. Which once again is the point I am arguing.

Which is what I am disagreeing with, particularly for Draper because you have not proven to me that Draper is the problem and not the other two. Is Draper replaceable? Sure, but there are a lot of players in the NHL far worse than Draper.
 
David

From what I can tell after a long discussion, you agree with me while simultaneously claiming I am wrong.
 
I still think the on ice off ice stat is misleading and not all that useful and I still don't think Draper is as bad as I think you think he is so I don't think we agree completely.
 
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?