Well, we already knew this, but let me expand. The other day Buster Olney made the case for using a statistic known as POP (Productive Out Percentage) to measure the success of teams making “productive outs.” Basically, POP is the proportion of outs made by teams that advanced runners with a few exceptions. The reason this statistic is meaningful, is that it captures how teams maximize the use of their outs. A hitter who advances a runner in making an out is better than a hitter striking out. While both situations have the same cost, the productive out produced something of value. Olney uses his new toy to poke the “Moneyball” crowd who frown upon bunting and other strategies to make productive outs. He describes the Moneyball style of play as “never bunt, don’t take chances on the bases, sit back and let your hitters hack away and do the work regardless of the game situation, regardless of the identity of the opposing pitcher.” To Olney, the problem with this philosophy is that this style of baseball sacrifices productive outs for unproductive outs, and the net difference in output favors the traditional “smallball” strategy. Olney quotes Detroit hitting coach Bruce Fields to make his point. “That’s how games are won and lost — productive outs, advancing baserunners and getting guys in from third with less than two out.”
As a fan I prefer productive to unproductive outs, but that is not saying much. Of course, all things being equal, productive outs are better than unproductive outs. But the problem is that rarely are all things equal in baseball. When there is a runner on first the opportunity cost of bunting him over is not just a strikeout, it is the entire forgone at-bat that could have resulted in a strikeout, walk, hit, hr, etc. The Moneyballers argue that the forgone good outcomes are more valuable than the foregone bad outcomes.
But, Olney is not convinced by this logic. I’ve never seen him respond so I’ll assume he remains unconvinced by the result that is largely a folk theorem. So I thought I would do my best to test the impact Olney’s POP and some simple sabermetric-friendly statistics (OBP and SLG) on runs per game. Unfortunately, I can’t find the official POP statistic anywhere, but sacrifice bunts and sacrifice flies are both available. Most, but not all, sacrifices are classified productive outs so I think this will be sufficient. I suspect that the correlation between the number of sacrifices and POP is strong. My model is:
Runs/Game = f( OBP, SLG, Sacrifices/PA).
I estimate this model on a sample of all teams from 1961-2003 using an OLS regression with random effects. Additionally, I test a smaller sample of the post-1997 (30 team) era in case the game has changed from the past. It could be that Sac/PA has only recently become important. I ran several diagnostic tests to look for the common problems and the results I report are, in my mind, the best. But, even small changes to the estimation procedure do very little to the results. Here are the regression results. Coefficient estimate in bold are statistically significant at the 1% level or better.
Surprise! OBP and SLG are not only statistically significant and Sac/PA is not (not even at the 10% level), but the magnitude of OBP and SLG impacts on runs per game are much larger. To the right of each estimate I provide the elasticities at the average of the variables. For example, the elasticity of 1.83 means that a 1% increase in OBP is associated with a 1.83% increase in runs per game at the average level of OBP and runs per game. For both samples OBP and SLG are very relevant while Sac/PA has almost no meaningful quantitative impact on runs per game. In the 30 team era, a higher Sac/PA is actually associated with lower run production. But one more thing, the regression technique I use holds OBP and SLG constant for changes in Sac/PA. Maybe Olney is finding some relationship (which he is not sharing) between Sac/PA and runs per game that is correlated with OBP and SLG. It is possible that teams with high OBP and SLG also have more sacrifices. While making such an assessment from such a correlation would be mistaken, it might explain why Olney goes wrong. Below, I present a scatter plot of all Sac/PA and runs per game with a fitted simple regression line.
This does not seem to help. If anything, teams with greater sacrifices are associated with scoring fewer runs per game. I’m not sure why Olney is so confident in his metric.
So what are we left with? If Olney would provide the POP stats I would gladly directly test its effect on run production, but until he does this is the best I can do. Larry Mahnken at THT has tested the “correct” POP statistic on the past two post-seasons, and he too finds results contrary to Olney. Futility Infielder and Talking Baseball (among a host of other sites) have demonstrated the pointlessness of this statistic. The online baseball community has done its part to refute the theoretical and empirical support provided by Olney. It is time for Olney to put up or shut up as to the validity of this stat. I don’t want to hear the phrase “productive out” again on ESPN until Olney shows us something tangible.