Did Steroid Use Enhance the Performance of Mitchell Report Hitters?

According to a new study published in Journal of Quantitative Analysis in Sports (JQAS)—Did Steroid Use Enhance the Performance of the Mitchell Batters? The Effect of Alleged Performance Enhancing Drug Use on Offensive Performance from 1995 to 2007 by Brian J. Schmotzer, Jeff Switchenko, and Patrick D. Kilgo—the answer is yes.

Conclusions: This analysis suggests a significant and substantial performance advantage for players who used steroids during the study period. It is estimated that offensive production increased approximately 12% in steroid users versus non-users. This analysis represents the first attempt to quantify the overall effect of PED abuse on offensive performance in baseball.

This study intrigued me, but the results are highly suspect because of obvious factual errors that should have been caught by a referee. I think the errors in the paper need to be addressed before we accept the results.

Though the empirical method employed—a mixed effects estimation—isn’t one that I would choose, I suspect that it ought to get the job done. Also, I don’t like the method used to account for aging, and I believe the method might bias the results (see my latest study on aging in baseball in Journal of Sports Sciences). I’m going to overlook these things, for now. What concerns me about the paper is the designation of players as users and non-users.

According to the paper, the authors identified performance-enhancing drug use in the following manner.

all steroid or HGH use was inferred strictly from exact years listed in the Mitchell Report corresponding to the seasons which may have been artificially aided by PEDs. Every attempt was made to use conservative measurements for alleged steroid use and non-use by players. Conservative, in this sense, means that even when the Mitchell Report listed anecdotal evidence or implied guilt by association, these seasons were not labeled as PED seasons unless there was a definitive statement in the report (specific dates and paper trails of evidence). A more liberal reading of the report (not considered herein) would have led to many more player seasons labeled as PED seasons. (p. 3)

This is certainly a defensible method—though, I would like to have seen results with “a more liberal reading” as well—however, I lack confidence that the authors employed their designation properly. What shakes my confidence? The paper highlights instances where judgment calls were made, and in a few cases those judgments are incorrect.

Case 1:

Mark McGwire – McGwire is named in several places in the Mitchell Report for his use of androstenedione in 1998 (the only season designated for him as a PED season). Androstenedione is thought to be a pre-cursor to steroids but not actually a steroid. However, it is a banned substance at present and thus his 1998 season was classified as a steroid season for the purposes of this analysis (p. 3).

I only have a minor problem with this designation—and it is by no means a damning error—but I think it is relevant. Andro is not an anabolic steroid, and if McGwire’s hitting success was fueled by PED use, andro was not the culprit. The literature on the ergogenic effects of andro indicates that it is not an effective aid. If the authors had reviewed the literature on andro as they did for growth hormone, they would not have designated this as a steroid season. And, it just so happens that 1998 happened to be an outstanding season for McGwire—the second highest RC/27 (the metric used to measure output) of his career. If he did use PEDs, it is likely that he used other PEDs for many years, not andro during this single season. I understand that identifying the exact dates of use is difficult, but this is an odd choice.

Case 2:

Barry Bonds – The Mitchell Report pinpoints Bonds’ steroid use to the 2003-04 seasons. It also hints that the use could have extended back as far as the 2000 season, but only based on his associations rather than stating an allegation definitively. Therefore, only the 2003 and 2004 seasons were classified as steroid seasons for the purposes of this analysis (p. 3).

This designation is what first alerted me that there might be problems with the study. To someone who has closely followed the Bonds case, it is obvious that these dates do not correspond with alleged doping by Bonds. Even those who have casually followed Bonds’s PED allegations ought to wonder why Bonds’s record-breaking 2001 is considered “clean” in this analysis.

BALCO was raided in September 2003, and Bonds testified to a grand jury about his involvement in December 2003. If he used in 2004, he had to have had another supplier, and he was boldly using drugs while under heavy suspicion and facing random drug tests. The Mitchell Report does not pinpoint any PED use in 2004; however, it does document his first visit to BALCO in 2001. Game of Shadows, the main source of the Mitchell Report’s discussion of Bonds, alleges steroid use beginning after the 1998 season. 2004 happens to be Bonds’s best season in terms of RC/27, and it’s classified as “dirty”.

I’m going to come back to this in a moment, because the inclusion of Bonds in the sample is important to the authors’ claim that Mitchell Report hitters demonstrated improved performance.

Case 3:

David Segui – Segui admitted to using steroids in the 1994-95 seasons; however, his paper trail of evidence with Radomski begins in 2004 and lasts through 2005. Therefore his 1996-2003 seasons are not classified as steroid seasons (p. 3).

This is not accurate. The paper trail to Rodomski (and PED use) in the Mitchell Report begins prior than 2004. And, this isn’t an easy mistake to make. Segui may be the most-mentioned player in the the Report. Here are the mentions of Segui’s ties to PEDs in the Mitchell Report.

“In 1999 or 2000, Chuck Hawke, an attendant working in the visiting clubhouse in Kansas City, found syringes and vials that were hidden in an Oakley sunglasses bag when he was unpacking luggage for David Segui” (p. 110).

“According to Radomski, Deca-Durabolin was Segui’s steroid of choice in the 1990s because it was safe, did not expire for three to four years, and was thought to help alleviate joint pain. Deca-Durabolin, however, stays in the body for up to a year or more and therefore is easily detectable in tests. Radomski said that Segui paid for the steroids by check although Radomski never asked him to pay for them. Radomski produced six checks drawn on David Segui’s checking account that were deposited into Radomski’s checking account….Radomski said he engaged in more than twelve transactions with Segui and dealt with Segui more than any other player. Toward the end of his career, Segui told Radomski that he had a growth hormone deficiency and was getting human growth hormone from a doctor in Florida (p. 151). Note that Segui purchased anabolic steroids, not growth hormone, from Radomski.

“McNamee first learned about Kirk Radomski through David Segui during the 2000 season” (p. 170).

“Kirk Radomski recalled meeting McNamee through David Segui. Radomski confirmed that he supplied McNamee with human growth hormone and anabolic steroids from 2000 to 2004” (p. 174).

“Radomski believed that Santangelo was referred to him by David Segui when both played for the Expos between 1995 and 1997” (p. 182).

“According to Radomski, he was introduced to Lansing by David Segui while Segui and Lansing played together with the Expos….Radomski produced two $1,000 money orders from Lansing, retrieved from his bank, made payable to Radomski; both were dated February 5, 2002” (pp. 196-197).

“Hairston was referred to Radomski by David Segui, his teammate on the Orioles from 2002 to 2004. Radomski said that he sold human growth hormone to Hairston on two or three occasions during 2003 and 2004” (p. 207).

The Appendix includes three personal checks from Segui to Radomski with dates prior to 2004: two checks from 2002 and a single check from 2003 (p. D-18).

David Segui did not play baseball in 2005; although, it may be that this date was not included in the sample and simply refers to irrelevant evidence from that year.

There appears to be strong evidence that Segui was an active user of steroids continuously during his major-league career. I found these examples using a simple search of “Segui” in a PDF file of the Report. I can find no explanation for the authors’ chosen steroid designations of 1994–1995 and 2004–2005. At the minimum, 2002 and 2003 should be listed as dirty. And if there was some concern about whether or not he was using, he should have been excluded from the sample.

If the authors made these types of errors, then I have no confidence for the designations used for the other players in the sample. This is especially important considering that the results appear to be fragile.

What if we assume these are the only mistakes in the study, and that the Segui coding errors are unbiased and average out? According to the authors’ reported results, the performance effect may disappear, though they do not stress this in the paper. When Bonds is excluded from the model and the performances are centered around player averages (as they should be) the estimates of the correlation between Mitchell Report inclusion and performance are not statistically significant. See Table 1 (Models 6 and 7) and Table 2 (Model 12) on page 11. This means that players cited in the Mitchell Report did not perform in a way that is outside the normal deviation in performance from players not cited in the report.

In summary, this study contains serious errors. I would like to stress that I do not think that any of the mis-coding was intentional; I work with data, and I understand how easy it is to make mistakes. However, the authors and referees should have caught these errors, especially considering that the results are quite different from those reported by Jonathan Cole (sociologist, Columbia) and Stephen Stigler (statistician, University of Chicago) in The New York Times (a study cited by the authors).

For the 48 batters we studied, the average change in home runs per year “before” and “after” was a decrease of 0.246. The average batting average decreased by 0.004. The average slugging percentage increased by 0.019 — only a marginal difference. So while some batters increased their totals, an equal number had falloffs. Most showed no consistent improvement, several showed variable performance and some may have extended the years they played at a high level, although that is a difficult question to answer.

It is possible that the coding is defensible in a way that is not obvious to me. I hope that the editor of the JQAS will re-review the paper and ask the authors to justify their designation or redo the study with correct coding. My goal is to find truth, and I do not wish to embarrass anyone. But in the meantime, if you notice someone citing this study, please be sure to note its problems.

UPDATE: The authors respond.

7 Responses “Did Steroid Use Enhance the Performance of Mitchell Report Hitters?”

  1. Brandon H says:

    With little if any evidence of who actually DID and DID NOT do PED’s over the last 5, 10, 15, 20+ years the author is simply hypothesizing that those who took them only took them when the the Mitchell report concluded.

    For example. Could I watch a game in which Mike Cameron hit 4 home runs, then walk away asserting that Mike Cameron hits 4 home runs a game?

    Comparing ‘known’ users to those who did not use is simply hypothetical. There is not any conclusive evidence to argue one way or another. Do we know that steroids helped Alex Sanchez get to the majors, did they help him stay in the majors, or did they prohibit him from becoming a better hitter? Will we ever know?

    Honestly, we will not. Even when we know which players used and did not use, the evidence will simply become hypothetical. How would Alex Sanchez perform in a non-steroid era?

  2. Ken Houghton says:

    So McGwire’s best season and Bonds’s best season are those where they appear to have been clean, at least judging by reality (as opposed to the whitewash that is the Mitchell Report).

    I believe it was an umpire talking about Mike Scott (during his resurgence with the Astros, pitching against Gary Carter and the Mets) who once observed, “They challenged his first pitch, which was clean.  Therefore, he had demonstrated what he could do with a ball, and I had no reason for suspicion for the rest of the game.”

    When Clemens is admitted and Bonds rejected, can we just rename the building in Cooperstown as “Not Just the Ball Must Be White”?

  3. justafan72 says:

    Geez, ANOTHER HGH-STEROIDS article? How about some analysis of whether or not the Braves should fire TP as the batting coach? Can we measure a batting coach’s impact?

    Or, how about a preview of some of free agents and their numbers? Something other than this dead horse..please, pretty please?

  4. Patrick says:

    Look, there are many possible ways this study could have been done. But there is no evidence that the quibbles you are pointing out would have SYSTEMATICALLY biased the results.

    It seems like JC is just bitter that HE is not the one getting the attention and credit for doing this study

    Now, I have a sneaking suspicion JC is going to censor this comment. And if so, then fine, it’s your blog, do what you like: But get over it already.

  5. JC says:

    Look, there are many possible ways this study could have been done. But there is no evidence that the quibbles you are pointing out would have SYSTEMATICALLY biased the results.

    1. My main “quibble” is that the paper suffers from obvious coding errors. In academic journals, such errors normally result in a paper being retracted from publication unless the authors demonstrate that the errors are not major.
    2. Actually, there is evidence that the errors bias the results, and the evidence is provided by the authors. When the mis-coded Bonds is excluded from the sample, steroid use is no longer associated with improved performance.

    Now, I have a sneaking suspicion JC is going to censor this comment. And if so, then fine, it’s your blog, do what you like: But get over it already.

    Patrick, if that is his real name, has been submitting comments for some time under the names of real people (baseball players, GMs, etc.) . I would have kindly asked him to stop, but he always submits a fake e-mail. I was hoping a ban might cause him to contact me so that I could tell him in private that I would like him to stop posting under other people’s names.

  6. eman says:

    this study is a little flawed because we will never officially know who did and did not take steroids or other drugs. obviously there was an advantage for steroid users, but i have found that some players [like David Bell] used steroids and it negatively affected their production.

  7. Marc Schneider says:

    JC never claimed that this is a Braves blog.  If you want to talk about the Braves, go to Braves Journal.  As for Patrick, the old saying seems to apply that it’s better to remain silent and let people think you are an idiot rather than speaking and removing all doubt.