According to a new study published in Journal of Quantitative Analysis in Sports (JQAS)—Did Steroid Use Enhance the Performance of the Mitchell Batters? The Effect of Alleged Performance Enhancing Drug Use on Offensive Performance from 1995 to 2007 by Brian J. Schmotzer, Jeff Switchenko, and Patrick D. Kilgo—the answer is yes.
Conclusions: This analysis suggests a significant and substantial performance advantage for players who used steroids during the study period. It is estimated that offensive production increased approximately 12% in steroid users versus non-users. This analysis represents the first attempt to quantify the overall effect of PED abuse on offensive performance in baseball.
This study intrigued me, but the results are highly suspect because of obvious factual errors that should have been caught by a referee. I think the errors in the paper need to be addressed before we accept the results.
Though the empirical method employed—a mixed effects estimation—isn’t one that I would choose, I suspect that it ought to get the job done. Also, I don’t like the method used to account for aging, and I believe the method might bias the results (see my latest study on aging in baseball in Journal of Sports Sciences). I’m going to overlook these things, for now. What concerns me about the paper is the designation of players as users and non-users.
According to the paper, the authors identified performance-enhancing drug use in the following manner.
all steroid or HGH use was inferred strictly from exact years listed in the Mitchell Report corresponding to the seasons which may have been artificially aided by PEDs. Every attempt was made to use conservative measurements for alleged steroid use and non-use by players. Conservative, in this sense, means that even when the Mitchell Report listed anecdotal evidence or implied guilt by association, these seasons were not labeled as PED seasons unless there was a definitive statement in the report (specific dates and paper trails of evidence). A more liberal reading of the report (not considered herein) would have led to many more player seasons labeled as PED seasons. (p. 3)
This is certainly a defensible method—though, I would like to have seen results with “a more liberal reading” as well—however, I lack confidence that the authors employed their designation properly. What shakes my confidence? The paper highlights instances where judgment calls were made, and in a few cases those judgments are incorrect.
Mark McGwire – McGwire is named in several places in the Mitchell Report for his use of androstenedione in 1998 (the only season designated for him as a PED season). Androstenedione is thought to be a pre-cursor to steroids but not actually a steroid. However, it is a banned substance at present and thus his 1998 season was classified as a steroid season for the purposes of this analysis (p. 3).
I only have a minor problem with this designation—and it is by no means a damning error—but I think it is relevant. Andro is not an anabolic steroid, and if McGwire’s hitting success was fueled by PED use, andro was not the culprit. The literature on the ergogenic effects of andro indicates that it is not an effective aid. If the authors had reviewed the literature on andro as they did for growth hormone, they would not have designated this as a steroid season. And, it just so happens that 1998 happened to be an outstanding season for McGwire—the second highest RC/27 (the metric used to measure output) of his career. If he did use PEDs, it is likely that he used other PEDs for many years, not andro during this single season. I understand that identifying the exact dates of use is difficult, but this is an odd choice.
Barry Bonds – The Mitchell Report pinpoints Bonds’ steroid use to the 2003-04 seasons. It also hints that the use could have extended back as far as the 2000 season, but only based on his associations rather than stating an allegation definitively. Therefore, only the 2003 and 2004 seasons were classified as steroid seasons for the purposes of this analysis (p. 3).
This designation is what first alerted me that there might be problems with the study. To someone who has closely followed the Bonds case, it is obvious that these dates do not correspond with alleged doping by Bonds. Even those who have casually followed Bonds’s PED allegations ought to wonder why Bonds’s record-breaking 2001 is considered “clean” in this analysis.
BALCO was raided in September 2003, and Bonds testified to a grand jury about his involvement in December 2003. If he used in 2004, he had to have had another supplier, and he was boldly using drugs while under heavy suspicion and facing random drug tests. The Mitchell Report does not pinpoint any PED use in 2004; however, it does document his first visit to BALCO in 2001. Game of Shadows, the main source of the Mitchell Report’s discussion of Bonds, alleges steroid use beginning after the 1998 season. 2004 happens to be Bonds’s best season in terms of RC/27, and it’s classified as “dirty”.
I’m going to come back to this in a moment, because the inclusion of Bonds in the sample is important to the authors’ claim that Mitchell Report hitters demonstrated improved performance.
David Segui – Segui admitted to using steroids in the 1994-95 seasons; however, his paper trail of evidence with Radomski begins in 2004 and lasts through 2005. Therefore his 1996-2003 seasons are not classified as steroid seasons (p. 3).
This is not accurate. The paper trail to Rodomski (and PED use) in the Mitchell Report begins prior than 2004. And, this isn’t an easy mistake to make. Segui may be the most-mentioned player in the the Report. Here are the mentions of Segui’s ties to PEDs in the Mitchell Report.
“In 1999 or 2000, Chuck Hawke, an attendant working in the visiting clubhouse in Kansas City, found syringes and vials that were hidden in an Oakley sunglasses bag when he was unpacking luggage for David Segui” (p. 110).
“According to Radomski, Deca-Durabolin was Segui’s steroid of choice in the 1990s because it was safe, did not expire for three to four years, and was thought to help alleviate joint pain. Deca-Durabolin, however, stays in the body for up to a year or more and therefore is easily detectable in tests. Radomski said that Segui paid for the steroids by check although Radomski never asked him to pay for them. Radomski produced six checks drawn on David Segui’s checking account that were deposited into Radomski’s checking account….Radomski said he engaged in more than twelve transactions with Segui and dealt with Segui more than any other player. Toward the end of his career, Segui told Radomski that he had a growth hormone deficiency and was getting human growth hormone from a doctor in Florida (p. 151). Note that Segui purchased anabolic steroids, not growth hormone, from Radomski.
“McNamee first learned about Kirk Radomski through David Segui during the 2000 season” (p. 170).
“Kirk Radomski recalled meeting McNamee through David Segui. Radomski confirmed that he supplied McNamee with human growth hormone and anabolic steroids from 2000 to 2004” (p. 174).
“Radomski believed that Santangelo was referred to him by David Segui when both played for the Expos between 1995 and 1997” (p. 182).
“According to Radomski, he was introduced to Lansing by David Segui while Segui and Lansing played together with the Expos….Radomski produced two $1,000 money orders from Lansing, retrieved from his bank, made payable to Radomski; both were dated February 5, 2002” (pp. 196-197).
“Hairston was referred to Radomski by David Segui, his teammate on the Orioles from 2002 to 2004. Radomski said that he sold human growth hormone to Hairston on two or three occasions during 2003 and 2004” (p. 207).
The Appendix includes three personal checks from Segui to Radomski with dates prior to 2004: two checks from 2002 and a single check from 2003 (p. D-18).
David Segui did not play baseball in 2005; although, it may be that this date was not included in the sample and simply refers to irrelevant evidence from that year.
There appears to be strong evidence that Segui was an active user of steroids continuously during his major-league career. I found these examples using a simple search of “Segui” in a PDF file of the Report. I can find no explanation for the authors’ chosen steroid designations of 1994–1995 and 2004–2005. At the minimum, 2002 and 2003 should be listed as dirty. And if there was some concern about whether or not he was using, he should have been excluded from the sample.
If the authors made these types of errors, then I have no confidence for the designations used for the other players in the sample. This is especially important considering that the results appear to be fragile.
What if we assume these are the only mistakes in the study, and that the Segui coding errors are unbiased and average out? According to the authors’ reported results, the performance effect may disappear, though they do not stress this in the paper. When Bonds is excluded from the model and the performances are centered around player averages (as they should be) the estimates of the correlation between Mitchell Report inclusion and performance are not statistically significant. See Table 1 (Models 6 and 7) and Table 2 (Model 12) on page 11. This means that players cited in the Mitchell Report did not perform in a way that is outside the normal deviation in performance from players not cited in the report.
In summary, this study contains serious errors. I would like to stress that I do not think that any of the mis-coding was intentional; I work with data, and I understand how easy it is to make mistakes. However, the authors and referees should have caught these errors, especially considering that the results are quite different from those reported by Jonathan Cole (sociologist, Columbia) and Stephen Stigler (statistician, University of Chicago) in The New York Times (a study cited by the authors).
For the 48 batters we studied, the average change in home runs per year “before” and “after” was a decrease of 0.246. The average batting average decreased by 0.004. The average slugging percentage increased by 0.019 — only a marginal difference. So while some batters increased their totals, an equal number had falloffs. Most showed no consistent improvement, several showed variable performance and some may have extended the years they played at a high level, although that is a difficult question to answer.
It is possible that the coding is defensible in a way that is not obvious to me. I hope that the editor of the JQAS will re-review the paper and ask the authors to justify their designation or redo the study with correct coding. My goal is to find truth, and I do not wish to embarrass anyone. But in the meantime, if you notice someone citing this study, please be sure to note its problems.
UPDATE: The authors respond.