The response to the Wolfers-Price study finding evidence of racial bias in the calls of NBA referees has been big. This paper is everywhere. Much of the criticism is based off a misunderstanding of what the authors did: “But most of the players in the leagues are black” or “The tops foulers in the league are white.” The authors use statistical methods to control for these and similar problems. I won’t defend the study, but even though there are some potential holes in the paper, I’ve yet to see a truly damning criticism.
The main legitimate criticism of the study is that the authors use game-level data (looking at total fouls within a game) as opposed to more granular foul-by-foul data. From the box scores, the authors cannot determine which referee called which foul. This is true, but it doesn’t condemn the study. Just because a study is imperfect doesn’t mean it’s wrong or that it should be ignored. Heck, the best study would involve implanting neural sensors on referee brains to interpret subconscious feelings of racial prejudice. The denial from imperfection fallacy is one of the most annoying critiques that researchers have to put up with.
The granular data is not available to the public, so we have to choose to live in a world where he accept to deny the inference of this study…except that the data could be made available by the NBA. The main argument coming from the NBA is that they have done a study with granular data, and that they do not find the same result.
“The biggest flaw is it’s wrong,” Joel Litvin, NBA president of league and basketball operations, said. “They didn’t have sufficient data to do the study they purported to do. They were trying to prove that individual referees, black vs. white, white vs. black, showed bias. But you can’t look at a box score and know which referee called which foul.”
But this is not the point. Wolfers admits that the data the NBA has is superior.
“Of course it would be better if the data recorded who blew the whistle,” he said. “Even in the absence of that, and this is not an interpretation, it’s a fact, when an individual player plays, he has fewer fouls when the majority of the referees are white than the same player on a night when the majority of the referees are black.”
The problem is that the NBA won’t share its data. And no wonder, the league is in a great position. The suits can claim the high-ground that their study is superior, yet not bear the burden of exposing the study to further scrutiny. I once had a researcher do this to me. I’ve been waiting two years for this person to send me the study that “refutes” my findings. Needless to say, it’s infuriating enough just to have the information to be withheld. But the frustration increases when people buy into this trick.
David Stern has not handled this properly. He has every right to defend his league, and his agitation is understandable. However, Wolfers and Smith aren’t out to paint the NBA as a bunch of racists. They are just using sports data to examine theories of human behavior. What Stern should do is go to the president of the American Economics Association (Thomas Sargent) to select a group of independent qualified researchers. Supply them with the NBA’s data with the stipulation that the economists not reveal anything more than their findings. This removes the privacy concerns. After all, the NBA was willing to share its data with the private firm it hired to do the study. The data stays secret and we get an independent evaluation of the conflict.
Right now we have one study that may or may not have problems that we can see, and one that we cannot. In the end, the paper will go through the standard refereeing process, and it will almost certainly be published. Following referee suggestions the authors may even determine that there is no evidence of racism. Unfortunately, the refereeing process is long, and it will be 1-3 years before this paper is published. And even then, scholars may still disagree over the findings. If I have to defer to a study that exists or one that doesn’t, it’s an easy decision. I think it is in the NBA’s best interest to investigate this further, because the issue isn’t going away.