Archive for Steroids
“The ones that have come out and admitted it, and are proven guilty, [their numbers] should not count. I’ve been cheated out of the game,” Oswalt continued.
As a Ranger, Rodriguez was 3-for-5 vs. Oswalt with two doubles, one home run, three RBIs and two walks. Last year, as a Yankee, Rodriguez was hitless in two at-bats against Oswalt.
“The few times we played them, when he got hits, it could have cost me a game,” Oswalt said. “It could have cost me money in my contract. He cheated me out of the game and I take it personally, because I’ve never done [PEDs], haven’t done it, and they’re cheating me out of the game.”
— Former MLBPA head Marvin Miller notes that ignorant coverage of performance-enhancing drugs’ benefits may actually encourage use.
“A kid who would love to be a professional athlete reads the sports pages or watches ESPN and is told over and over again, ‘These are performance-enhancing drugs. They will make you a Barry Bonds or an A-Rod or a Roger Clemens.’ The media, without evidence, keep telling young people all over the country, ‘All you have to do to be a famous athlete with lots of money is take steroids.’ The media are the greatest merchants of encouraging this that I’ve ever seen.”
Miller is a bit over the top here, because I think that anabolic steroids likely do enhance performance and players should want some testing system to prevent their use; however, when it comes to growth hormone, he’s dead on. The media has completely botched the coverage of this issue, which is one of the reasons why I think that growth hormone should be legalized.
The point is that players have a strong incentive to gain an edge on each other. This road will inevitably lead many of them to seek out illicit solutions in an area where the experts are the guys who sell the stuff. And when they investigate further, they find prominent sports reporters declaring that HGH is just as effective as steroids. Do you think players are going to search through the scientific literature on PubMed? Heck, if I didn’t share an office suite with exercise physiologists, I probably wouldn’t know any better.
At the end of the day, players are going to take a long hard look at the list of prohibited substances. The fact that these drugs are banned will be sufficient to convince most players that the performance-enhancing benefits are real.
First, I suggest a system of fines and bonus. This is a Pigouvian tax and subsidy system that taxes players in accordance with the external costs that users impose on non-users—users may feel the personal benefits of a higher salary outweigh the health risks—and then transfers the financial gains to non-users who earn relatively less due to the fact that they chose to remain clean. This has the deterrence effect similar to suspensions; however, the substantial fine revenue gives players who feel they are in a use-or-lose situation an incentive not to use and to identify new cheating methods.
Second, I propose handing over all monitoring and testing to the players. It is the players who suffer the most from steroids. They are in an arms races where steroids make no individual relatively better than any other player—hence, there is no financial gain—yet, users end up suffering health consequences. This resembles a prisoner’s dilemma game.
Following yesterday’s post, I received an e-mail from an interested reporter asking me if it was possible to compare Alex Rodriguez‘s performance versus expected performance during his admitted steroid seasons.
Something about the wording of his e-mail stirred me to think about a better way to look at this than what I had previously done. The procedure is simple, I use Baseball-Reference’s “Neutralize Stats” tool to convert A-Rod’s home-run rate (HR/AB) performance to a consistent baseline. Then I use my estimates of aging from a study to be published in Journal of Sports Sciences (working paper version) to examine Rodriguez’s neutral career aging trajectory.
The table below lists A-Rod’s career performance neutralized for park and era effects—these are not his actual numbers, which are polluted by ballpark and era effects—as well as a correction for natural aging. The “Aging (v. Peak)” column reports the percent difference from his projected peak HR/AB. I base the aging progression towards his peak using the mean of his age 23 and 24 performances to project his peak HR/AB performance of 9.18% at age 30. It’s interesting to note that A-Rod’s peak HR/AB performance occurs at age 31 at 9.37%.
Year Age Neutral Aging Pred. Neutral Pred. Neutral HR HR/AB (v.Peak)HR/AB HR HR - Pred. HR 1994 18 0.00% -67.63% 2.97% 1995 19 3.18% -56.73% 3.97% 1996 20 5.63% -46.79% 4.89% 1997 21 3.94% -37.80% 5.71% 1998 22 5.98% -29.77% 6.45% 1999 23 7.91% -22.70% 7.10% 2000 24 7.41% -16.58% 7.66% 2001 25 8.12% -11.43% 8.13% 51 51.07 -0.07 2002 26 8.91% -7.23% 8.52% 55 52.56 2.44 2003 27 7.53% -3.98% 8.82% 45 52.72 -7.72 2004 28 5.98% -1.70% 9.03% Gain 01-03 -5.35 2005 29 8.28% -0.37% 9.15% Gain 01-02 2.37 2006 30 6.00% 0.00% 9.18% 2007 31 9.37% -0.59% 9.13% 2008 32 6.84% -2.13% 8.99%
The reporter also noted that A-Rod claims to have quit taking steroids after a 2003 spring training injury; therefore, we might not want to include 2003 as a steroid season. Thus, I include his 2001–2003 and 2001–2002 total gains in home runs. As the table indicates, 2003—the season in which we know he tested positive—he hit nearly 8 fewer home runs than expected. He hit almost exactly as many home runs as expected in 2001 and 2.44 more than expected in 2002.
So, what were A-Rod’s steroids worth? 2.37 home runs over two seasons, or a little over one home run a season. At least, that is the estimate based on the method I laid out above; however, it’s probably best to say that there was no observed effect. It is possible that the steroids did give Rodriguez a boost, and this may have helped him through an injury or some other factor that my estimate does not account for. It’s also likely that he hit more home runs than expected through random chance. Given the general swings in the play of the game, it is very difficult to separate true performance changes from random swings in performance. The deviation here isn’t large enough to say much.
The important finding is that the statistical record doesn’t reveal an obvious spike in home-run performance by Alex Rodriguez during the time when he admits to using performance-enhancing drugs.
Addendum: The reporter mentioned above is Carl Bialik, who now has a post up on the subject at The Numbers Guy.
Further Addendum: For those who have asked why posted this brief analysis, see Thomas Boswell’s latest.
Rodriguez may have taken performance-enhancing drugs for only three years — never before, never after.
For one thing, his statistics, as we’ll show, indicate that he may be coming clean. He averaged 33 percent more homers in his dirty Texas years — from 2001 to 2003 — than in the other 10 full seasons of his career. That’s a huge leap, similar to the numbers that first incriminated Barry Bonds in many baseball minds….
In his three years in Texas, from 2001 to 2003, he averaged 52 homers vs. 39.2 everywhere else. The jump was even bigger when compared to his previous five superstar years in Seattle, when he averaged 36.8 homers.
After hitting 42, 42 and 41 homers in his last three years in Seattle, he hit 52, then 57 in his first two years in Texas. Granted, the Ballpark is a launching pad. Rodriguez slugged .666 there in three years vs. .576 on the road. But that’s still worth only a few extra homers a year, not 12 or 15.
We now know why the MLBPA didn’t destroy the 2003 drug tests.
“In mid-November 2003, the 2003 survey test results were tabulated and finalized. The MLBPA first received results on Tuesday, November 11. Those results were finalized on Thursday, November 13, and the players were advised by a memo dated Friday, November 14. Promptly thereafter, the first steps were taken to begin the process of destruction of the testing materials and records, as contemplated by the Basic Agreement. On November 19, however, we learned that the government had issued a subpoena. Upon learning this, we concluded, of course, that it would be improper to proceed with the destruction of the materials. The fact that such a subpoena issued in November 2003 has been part of the public record for more than two years. See, U.S. v. CDT, 473 F3d at 920 (2006), and 513 F3d at 1090 (2008) (both opinions have now been vacated). Other subpoenas followed, including one for all test results.
“Over the next several months we attempted to negotiate a resolution of the matter with the United States Attorneys Office for the Northern District of California. During that time we pledged to the government attorneys that the materials would not be destroyed. When the government attorneys refused to withdraw its subpoena for all 2003 test results, we decided to ask a judge to determine to what the government was entitled. See, 473 F3d at 944, and 513 F3d at 1118. On the same day we were filing our papers with the court, the government attorneys obtained a search warrant and they began seizing materials the following day. Pursuant to that search warrant which named only 10 individuals, the government seized records for every baseball player tested under our program, in addition to many records related to testing in other sports,and even records for other (non-sport) business entities.
Also note the jab to the media: “The fact that such a subpoena issued in November 2003 has been part of the public record for more than two years.” And the Gene Orza “tipping” allegation appears to be the previously-documented 2004 meetings with players to address the seizures. Again, this was covered over a year ago.
Aside: The MLBPA’s director of communications is named Greg Bouris. How many guys with a similar last name are involved in this thing? Jeff Borris (Barry Bonds’s agent), Scott Boras (Alex Rodriguez’s agent), and now Greg.
I don’t have time for much commentary, but in case you are wondering…
2001–2003 are his admitted steroid years. They are also the years in which he played in homer-friendly Rangers ballpark.
If you look hard enough, you’ll probably think you see something, but there doesn’t appear to be much here. Yes, his homers went up when the AL’s HR rate went down, but then take a look at 2007 and 2008. Also, he was also still experiencing aging improvements in a helpful environment.
UPDATE: More here.
In case you missed, Sports Illustrated broke the story this weekend that Alex Rodriguez’s 2003 “anonymous” drug test revealed use of an anabolic steroid. Here are some of my thoughts on the issue.
— This is an absolute embarrassment to the US government. Here we have a private organization implementing a program to fix a problem that government officials wanted fixed. Players did not have to agree to random testing, and without the 2003 anonymous testing we might have a very different MLB drug policy today. The samples ultimately got used for something other than their intended purpose, and people wonder why players are were to reluctant to agree to testing in the first place? President Obama is right to shut down Gitmo for violating civil rights. He should shut down the BALCO case as well. The proper role of government isn’t to satisfy our curiosity about doping in sports. This has what this case is about.
— Why didn’t the union destroy the tests? My guess is that the assumed the government would not be able to get their hands on the list—why would they think that the government might want this?—and the union wanted to have the information for its own purposes. I have argued that it’s in the player’s interest to test each other, owners shouldn’t care. I suspect that union leaders could find a lot of uses for these tests. For example, if a player fails a test now and claims it must be a false positive, the union could benefit from knowing if the player had used before when it plans its defense
— It’s way to early to think about Hall of Fame implications. We have no idea about how we’re going to think about steroid use in the future.
— Releasing the remainder of the 104 names isn’t fair. If one name leaked off a list of homosexual players who wished to remain in the closet, would it be fair to out the remaining names? What happened to A-Rod was unfair, but that doesn’t mean that releasing the names of other players is now morally acceptable. If players don’t want their names trickling out, then individual players on the list can reveal this on their own.
— What’s A-Rod’s strategy going to be? Many have suggested that he “come clean” and apologize like Andy Pettitte did. The Pettitte woe-is-me act won’t work for A-Rod. We didn’t forgive Pettitte because we thought he was sorry. He wasn’t sorry, because he changed his story of “used once” to “used twice” when new details came up. Pettitte got a pass because he’s liked, not for being honest. I think the strategy is going to be sue everybody. Whoever leaked this information did so illegally, and the only way to find out who did it is to take the whole thing down. Maybe he’ll cop to some use, I don’t know; but, I think this is going to get very ugly.
Addendum: King Kaufman demonstrates why the Andy Pettitte defense won’t work for A-Rod.
…Alex Rodriguez is Alex Rodriguez. He can’t scratch his ear without somebody declaring that he symbolizes all that’s wrong, evil and distorted with baseball, sports, America, humanity and the universe. I mean, did you see him scratch his ear?
Update: And now we can test if the Pettitte strategy works for A-Rod.
“When I arrived in Texas in 2001, I felt an enormous amount of pressure. I needed to perform, and perform at a high level every day,” Rodriguez told ESPN’s Peter Gammons in an interview in Miami Beach, Fla. “Back then, [baseball] was a different culture. It was very loose. I was young, I was stupid, I was naïve. I wanted to prove to everyone I was worth being one of the greatest players of all time.
“I did take a banned substance. For that, I’m very sorry and deeply regretful.”
“The best evidence for a power pitchers’ potency is strikeouts. Clemens’s strikeout rate relative to the league declined as he aged. If he was getting some artificial help, wouldn’t we have expected him to have improved in this area? The one thing that jumps out at you when you look at the numbers for Clemens’ last several seasons is not striking out batters but preventing walks. That’s the part of his game least likely to have been affected by PEDs. I think this lends support to the idea that Clemens was able to maintain effectiveness as he got older because he simply got smarter and tougher.”
If you would like to see my analysis of Roger Clemens’s career, you can read my initial post on the subject here. And, this post contains links to several statistical analyses of Clemens’s career. In summary, my view is this: if Clemens used performance-enhancing drugs, they didn’t work. Contrary to popular opinion, the statistical record does not support Brian McNamee’s allegations.
This will be short and simple.
For the third year in a row, Mark McGwire did not receive sufficient votes from the Baseball Writers Association of America (BBWAA) to be elected to the Hall of Fame. The explanation is simple: many writers feel that his performance was aided by performance-enhancing drugs. There are certainly several sources for accusations, but they have some credibility problems. Others point to his continued excellent performance into his thirties and his bulging biceps. McGwire fits the profile of a steroid user, and I wouldn’t be surprised if he admitted to using them. McGwire hasn’t helped his case by refusing to testify under oath before Congress; however, that is the advice that every lawyer would give to his client in these circumstances.
I don’t want to pick a fight regarding whether or not he used steroids. I don’t really care. My argument is simple. Let’s assume McGwire used hard-core anabolic steroids every day of his baseball career. He didn’t violate a single baseball rule. Mark McGwire played his last baseball game in 2001. It wasn’t until 2004 when anabolic steroids became a punishable offense despite the fact that serious doping regulations had been instituted in nearly all other sports. McGwire shouldn’t be excluded any more than any other player who drank amphetamine-laced coffee prior to its ban. He’s being barred from the Hall of Fame for doing something that people wish was against the rules but wasn’t.
I guess my mind hasn’t changed since last year.
A story in today’s NYT discusses the potential of Viagra as a performance-enhancing drug.
Viagra, or sildenafil citrate, was devised to treat pulmonary hypertension, or high blood pressure in arteries of the lungs. The drug works by suppressing an enzyme that controls blood flow, allowing the vessels to relax and widen. The same mechanism facilitates blood flow into the penis of impotent men. In the case of athletes, increased cardiac output and more efficient transport of oxygenated fuel to the muscles can enhance endurance.
“Basically, it allows you to compete with a sea level, or near-sea level, aerobic capacity at altitude,” Kenneth W. Rundell, the director of the Human Performance Laboratory at Marywood, said of Viagra.
Some experts are more skeptical. Anthony Butch, the director of the Olympic drug-testing lab at U.C.L.A., said it would be “extremely difficult, if not impossible” to prove that Viagra provided a competitive edge, given that the differences in performance would be slight and that athletes would probably take it in combination with other drugs. Scientists have the same uncertainty about the performance-enhancing effects of human growth hormone, though it is banned. But some athletes do not need proof — only a belief — that a drug works before using it, Dr. Butch said.
During the previous summer, there was some discussion of Viagra as an ergogenic aid. At that time, I asked some exercise physiology colleagues about the potential effects and here is what they said.
I asked two of my exercise physiologist colleagues about the potential performance-enhancing effects of Viagra. Both felt that it is unlikely that the drug would help. The reason being that the problem isn’t getting blood flow to the muscle but the muscles using the blood flow. One was a little more believing of the theoretical improvement, but felt it would have no practical effect. The other felt there was almost no way it could work. In sum, it’s probably a placebo effect that leads athletes to use Viagra. This effect could be exacerbated by positive sexual experiences.
Three hours into a conference held Monday by Major League Baseball on human growth hormone, the real question of the day emerged when officials from the commissioner’s office and the players union wondered aloud about how effective the current blood test for human growth hormone was if no one had tested positive.
In the wake of last December’s Mitchell report, Commissioner Bud Selig said he would bring together leading experts in the field of performance-enhancing drugs to discuss the barriers of testing for human growth hormone.
HGH is not a performance-enhancing drug. Why is MLB doing this? The same reason I have to attend diversity training: to give the appearance of solving a problem that the public cares about. And in the case of growth hormone, public opinion is at odds with the scientific consensus.
No one would believe MLB if it stood up and stated what exercise physiologists have long known: that there are no ergogenic effects from using HGH. The response would be, “MLB is refusing to fight drugs!” They can’t win that battle any more than throwing a ball around circle to discuss racial feelings is going to cure Klansmen of racism. So, we live this bizarre fiction that HGH does work and that it is worth stopping, despite the fact that it runs counter to findings of scientific studies.
I guess I can’t blame MLB. It’s the cheapest way to fight a public relations problem—that’s all this is. And the sad part is that HGH’s prohibition signals to potential users that it works, and the drug has many bad side effects. If anything, the war on growth hormone will do more harm than good. As I have suggested before, the best solution is to legalize it.
My response to Brian J. Schmotzer, Jeff Switchenko, and Patrick D. Kilgo’s reply to my criticism of their study follows. I would like to thank the authors for offering their response; however, I do not think their explanations succeed in validating their study.
First let me address a few minor issues about which I will not go into significant depth. I have no problem with mixed effects, it just isn’t the model I would have used. In fact, in my initial critique I stated, “I suspect that it ought to get the job done”. Aging also is not a big issue, but I am appreciative that the authors took the steps to re-estimate their model according to my previous analysis. It appears that aging adjustments do not make much difference.
I see that our disagreements boil down to two points on which I will focus my remarks: coding and the statistical significance of the estimates.
On coding, the authors main argument is that coding is difficult in some cases, and that their coding choices “are not coding errors but simply differences of opinion.” We certainly do share a difference of opinion on this matter—after all, all disagreements come down to opinion—but, the authors claim that the coding that I previously believed resulted from human error was the result of conscious action. Thus, in this sense, their designations are deliberate choices, not coding errors. I believe that several of the designations discussed reveal a pattern of inconsistent assignment of steroid use that does not follow from their stated methodology. There are two main player examples at issue: David Segui and Barry Bonds.
In terms of Segui, the authors state the following.
He clearly is accused in the Mitchell Report of being a steroid user. There is no question about it. However, our task was to designate seasons of abuse – simply recognizing a player as an accused abuser was not good enough. Because of our strict criteria, we did not denote any of Segui’s years as steroid seasons. (We note here an actual mistake in our manuscript. We said 1995 and 2004 were denoted as steroid seasons when we actually denoted them as HGH seasons.)
First, the authors did make an error. 1995 and 2004 were not steroid seasons. This explains my confusion when I stated “I can find no explanation for the authors’ chosen steroid designations of 1994–1995 and 2004–2005.” Thus, the error was not in the coding, but in the mis-reporting of the coding. But it creates new confusion: why were 1995 (especially) and 2004 coded as HGH seasons? I can find no supporting documentation for these designations in the Mitchell Report. The authors did not provide an explanation in their response.
Second, the authors then include a list of quotes of accusations in the Mitchell Report to demonstrate ambiguities in the accusations. I appreciate the authors’ difficulty, and do not have a problem with employing a conservative standard. I stated in my initial critique, “This is certainly a defensible method—though, I would like to have seen results with “a more liberal reading” as well—however, I lack confidence that the authors employed their designation properly.” Upon further review, I still find their coding choices to be strange, in that the coding does not appear to form a consistent pattern.
The Mitchell Report includes four legible checks from Segui to Kirk Radomski from 2002–2004. The authors state:
the checks could have been written for HGH. In terms of the checks, we viewed this as weaker evidence since many unnamed players presumably wrote checks to Radomski for innocent items.
On page 151 of the Mitchell Report, Radomski states that he sold steroids to Segui, even going so far as to say that the growth hormone he was getting came from a doctor in Florida. The checks themselves are the most damning evidence in the entire report. This is physical evidence, backed with testimony from both parties that Segui’s relationship with Radomski was one of steroid seller and steroid purchaser. I believe it is unreasonable to assume that these checks could be for “innocent items.” If this is not considered “Other ancillary items including the source of the allegations and whether there is a paper trail of evidence”, then I don’t know what standard the authors have set, especially when viewed with flimsiness with which Segui and Bonds were identified as users at other times in their careers.
The authors state on page 3 of their paper Segui’s “paper trail of evidence with Radomski begins in 2004 and lasts through 2005.” Where does the 2004 date come from? David Segui didn’t even play baseball in 2005. Where did the 1994 and 1995 designations (see above) come from? I understand the difficulty in the decision to designate other years as steroid (or HGH) years, which is why I stated, “At the minimum, 2002 and 2003 should be listed as dirty.” I was acknowledging by their own standards he should have been declared a user at these times.
The case of Segui becomes more problematic when compared to Bonds’s case. If the physical evidence regarding Segui doesn’t meet the standard of “Other ancillary items including the source of the allegations and whether there is a paper trail of evidence”, then Bonds should have been listed as clean for the entire sample.
First, you suggest that 2001 should be labeled a steroid season. But this is based on the BALCO evidence and the Game of Shadows (or more precisely, the Mitchell Reports references to those sources). While these sources may be reliable, they are irrelevant for the present discussion because we used the seasons denoted by the Mitchell Report as our sole data source. This is obviously suboptimal (as we acknowledge in our paper) but based on this methodology the 2001 season should not be labeled a steroid season. To bring in other sources would be to slide down that slippery slope headfirst with no chance for objectivity to survive. Second, you suggest that 2004 should not be labeled a steroid season because the BALCO mess occurred in 2003. However, the Mitchell Report states that Anderson was removed from the clubhouse in 2004 but continued to work with Bonds after that. Further, the Giants asked Bonds to have no contact with Anderson early in 2005. Although we cannot confirm that Bonds did stop dealing with Anderson at that time, our conservativeness suggested that 2005 should not be a steroid season. But it seems that it is still reasonable to call 2004 a steroid season under that scenario (pages 126-127). This, too, is debatable.
First, I don’t necessarily think 2001 should be coded as a steroid season. As readers of this website know, I have been hesitant to condemn Bonds. He did first visit BALCO in 2001, but it is the start of the steroid designations in 2004 that make no sense to me. And the designation does not fit with with the excessive conservativeness for designating Segui and Bonds’s prior use.
— Four checks and testimony from the parties involved that these facilitated the purchase of steroids does not constitute a paper trail for Segui.
— Leaked grand jury testimony (which has since been released to the public) in which Bonds admits taking substances in 2002 and 2003 that prosecutors identified as performance-enhancing drugs (Bonds did not believe “The Cream” or “The Clear” were steroids) is not sufficient evidence of use. The Mitchell Report relies much on the investigation of Bonds, just as it relies heavily on the government investigation of Kirk Radomski.
— The Giants dismissing Anderson from the clubhouse in 2004 at a time when Jeff Novisky is monitoring Bonds’s every move and MLB has instituted testing is considered to be evidence of use.
In their rebuttal, the authors conclude “our decisions were not made lightly, and we hope you can see the merit of our methodology even if you don’t agree with it.” I see merits to a conservative “paper trail” methodology, but in practice the designations do not make sense. The implementation is seriously flawed. If I arranged the above choices on a spectrum of conservative and liberal designations, the one that gets coded dirty is the most liberal of the three. The coding of steroid use is not consistent, and I do not see how these designations can be defended as reasonable choices.
As you can see, the denoting of steroid seasons in some cases is a complicated task. We have made every effort to be conservative in our designations and to base them (to the extent possible) on strict evidence from the Mitchell Report. Luckily, the majority of cases were straightforward and are highly unlikely to contain any mistakes of note. Unluckily, a few of the cases were more difficult. We hope the above explanations illuminate more fully our rationale on those more ambiguous adjudications.
I disagree. In fact, when I re-read portions of the Mitchell Report after reading their paper, I found identifying specific years of use for most players listed in the report to be a difficult task. Segui and Bonds actually seem to be two of the simpler cases, and there does not appear to be a consistent standard for coding their potential use of steroids. And here is the real problem with the coding: where I can see what is being done, I observe it is being done incorrectly. This leads me to believe that coding is being done incorrectly with other players as well. I have no faith in the analysis, especially considering this study’s findings are at odds with the findings of other studies of Mitchell Report players by Cole and Stigler and The Milwaukee Sentinel (which interpreted its own findings incorrectly).
The authors also take the step of re-estimating the effect of designations on performance while adjusting the designations according to my suggestions. The new estimates are superior to the old estimates; however, this doesn’t allay my fears that coding errors may exist elsewhere. In addition, the authors do not report standard errors for determining statistical significance for their new estimates; therefore, it is impossible to know if “dirty” players improved their performances relative to “clean” players. This leads to the next point of contention: statistical significance.
I want to address the response to the robustness of the the results.
First, you suggest that the results are “fragile”. Quite the contrary. Under a variety of analysis assumptions, the steroid effect was always positive and nontrivial.
I did not dispute that the reported coefficient estimates are consistent across many specifications. I state the results are fragile, because when Bonds is removed, “dirty” players do not perform better than “clean” players at a statistically significant level. The fragility of the results comes from the fact that when Bonds is removed from the sample, the coefficient estimates of steroids use are no longer statistically different from zero. For readers not familiar with statistics, this means that the impact for those steroid-designated seasons is not meaningfully different from non-steroid seasons, given the typical deviations in performance for all players in this sample.
Second, you note that when Bonds is excluded from the model, the steroid effect is “not statistically significant”. It is understandable to infer that we did not focus our discussion on statistical significance because the p-values for some models were large. However, this is not the case. (We could have easily omitted p-values from our paper or omitted models with large p-values from our paper if we felt we had something to hide.) In fact, we presented p-values because it is traditional to do so for statistical models. But in reality, they are largely (one could argue, completely) irrelevant for this study. This is a census, not a sample. There is no sampling variability. The effect we observed in our models is the true effect by definition.
This is a curious response. The authors reported one significant p-value estimate (Model 1) in the paper’s abstract as evidence of a performance effect. “The effect of steroid use was an additional 0.58 ADJRC27, an increase in production of 12.6% (p=0.0108).” At one time, the authors believed the p-value to be relevant, and they were correct in their belief.
Now, they suggest that the p-values are not relevant because “This is a census, not a sample. There is no sampling variability. The effect we observed in our models is the true effect by definition.”
First, this is a sample that includes 1336 players had 50 plate appearances in a season from 1995–2007. Second, I am confused as to why the authors think this would be relevant if it was true. The p-values of the steroid coefficient reveal the likelihood that the two cohorts (because this is a binary variable) performed differently from one another. That is why the statistical programs the authors employed reported p-values along with the coefficient estimates. When Bonds is excluded from the sample, the p-values are greater than 0.05 in five of the six models that exclude Bonds (5,6,7, 11, and 12; 9 is the exception). The variance of the performances (which the p-value measures) is key. The standard errors indicate that though the estimate of change in performance in steroid seasons is positive, it is not outside the normal variation in player performance. The fact that the authors did not provide p-values in the revised tables leads me to believe that the new estimates are not statistically significant, either.
In summary, though the authors have offered responses to my criticisms, their rebuttal falls short of rectifying the problems that I previously identified. The coding choices are not consistent with the stated methodology, which makes it reasonable to assume that other coding problems exist. Also, estimates do not indicate that players coded as steroid users perform better than non-steroid users at statistically significant level. I thank the authors for their reply, but I do not agree their conclusions.