Stat Abuse

I’ve had a few people e-mail me about a mini-study in the Milwaukee Journal-Sentinel that examines changes in performance by players named in the Mitchell Report. The authors interpret the results as evidence that the performance-enhancing drugs mentioned in the report “boosted” player performances.

This one is a real head-scratcher, because as I read the study it seems indicate the drugs had no effect on performance. This is without even taking into account other methodological problems with the study. Luckily, my buddy Frank Stephenson (economics prof. at Berry College) properly criticizes the study and crunches the numbers at Division of Labour.

The Mitchell Report named 86 players so finding 46 (a mere 54%) that improved might well nothing more than random chance. Indeed, 46 is less than one standard deviation (4.6) away from 43 for a binomial distribution with n=86 and p=.5. Could it be that the Freakonomics guys have been, ahem, fooled by randomness?

2 Responses “Stat Abuse”

  1. Cliff says:

    The first obvious issue to put in any such study is age. That is, we know for hitters when (by age) declines normally happen and when increases normally happen. The hitters in such a study should be compared to the relatively well known age curve. If the first time users are routinely 30 to 35 and then 54% of them have increases, that would be big. Normally, 65% would experience at least some decline.

  2. tangotiger says:

    Read Phil’s post:
    http://sabermetricresearch.blogspot.com/2007/12/milwaukee-newspaper-article-on-steroids.html

    The 50% improvements is based on anyone who improved in *either* of 2 years (TT, TF, FT). That means the 50% decline are those that decline in each of the two years (FF).