Racist Referees Again

The response to the Wolfers-Price study finding evidence of racial bias in the calls of NBA referees has been big. This paper is everywhere. Much of the criticism is based off a misunderstanding of what the authors did: “But most of the players in the leagues are black” or “The tops foulers in the league are white.” The authors use statistical methods to control for these and similar problems. I won’t defend the study, but even though there are some potential holes in the paper, I’ve yet to see a truly damning criticism.

The main legitimate criticism of the study is that the authors use game-level data (looking at total fouls within a game) as opposed to more granular foul-by-foul data. From the box scores, the authors cannot determine which referee called which foul. This is true, but it doesn’t condemn the study. Just because a study is imperfect doesn’t mean it’s wrong or that it should be ignored. Heck, the best study would involve implanting neural sensors on referee brains to interpret subconscious feelings of racial prejudice. The denial from imperfection fallacy is one of the most annoying critiques that researchers have to put up with.

The granular data is not available to the public, so we have to choose to live in a world where he accept to deny the inference of this study…except that the data could be made available by the NBA. The main argument coming from the NBA is that they have done a study with granular data, and that they do not find the same result.

“The biggest flaw is it’s wrong,” Joel Litvin, NBA president of league and basketball operations, said. “They didn’t have sufficient data to do the study they purported to do. They were trying to prove that individual referees, black vs. white, white vs. black, showed bias. But you can’t look at a box score and know which referee called which foul.”

But this is not the point. Wolfers admits that the data the NBA has is superior.

“Of course it would be better if the data recorded who blew the whistle,” he said. “Even in the absence of that, and this is not an interpretation, it’s a fact, when an individual player plays, he has fewer fouls when the majority of the referees are white than the same player on a night when the majority of the referees are black.”

The problem is that the NBA won’t share its data. And no wonder, the league is in a great position. The suits can claim the high-ground that their study is superior, yet not bear the burden of exposing the study to further scrutiny. I once had a researcher do this to me. I’ve been waiting two years for this person to send me the study that “refutes” my findings. Needless to say, it’s infuriating enough just to have the information to be withheld. But the frustration increases when people buy into this trick.

David Stern has not handled this properly. He has every right to defend his league, and his agitation is understandable. However, Wolfers and Smith aren’t out to paint the NBA as a bunch of racists. They are just using sports data to examine theories of human behavior. What Stern should do is go to the president of the American Economics Association (Thomas Sargent) to select a group of independent qualified researchers. Supply them with the NBA’s data with the stipulation that the economists not reveal anything more than their findings. This removes the privacy concerns. After all, the NBA was willing to share its data with the private firm it hired to do the study. The data stays secret and we get an independent evaluation of the conflict.

Right now we have one study that may or may not have problems that we can see, and one that we cannot. In the end, the paper will go through the standard refereeing process, and it will almost certainly be published. Following referee suggestions the authors may even determine that there is no evidence of racism. Unfortunately, the refereeing process is long, and it will be 1-3 years before this paper is published. And even then, scholars may still disagree over the findings. If I have to defer to a study that exists or one that doesn’t, it’s an easy decision. I think it is in the NBA’s best interest to investigate this further, because the issue isn’t going away.

9 Responses “Racist Referees Again”

  1. I agree that looking at refereeing crews rather than individuals could be erroneous.

    However, I would also like to point out that the paper looks at technical fouls. In my opinion technical fouls would be much more prone to emotion and bias than personal fouls. The coefficient on blackplayer*%white referees is not statistically significant in this case.

    Also, why is the dependent variable fouls per 48 minutes instead of fouls per game? This seems to suggest that the dependent variable not give equal weight to those who come in to the game for only a few minutes and commit a foul. This shows up in the data as well:

    Mean Fouls per game:

    Black players — 2.5
    White players — 2.5

    Mean fouls per 48 minutes:

    Black players — 4.3
    White players — 4.97

  2. Kyle S says:

    First, the NBA’s lawyers probably don’t allow Stern to share that data with anyone for legal reasons. Once it’s in the wild, there’s no getting it back.

    Second, they absolutely shouldn’t turn over the data unless they know what the result is going to be and are comfortable with that being aired in public. What if the independent researchers come back saying referees X, Y, and Z are racists? This is something that the NBA should find out for themselves but not necessarily in this setting – they can conduct the analysis during the offseason and quietly not renew those referees’ contracts.

    Finally, the complaint about the NBA not sharing its data seems like so much whining to me. The authors could probably find game film of all those games and log all the foul calls themselves. Sure it would be much easier to use the NBA’s database, but that data could easily be recreated with some effort.

  3. tangotiger says:

    The biggest showing was in the all-white versus all-black crews (see Table 3). Therefore, you already know the race of the whistle-blower, if you limit the study to these groups. The age of the referee, his quality (did he make the post-season), and simply the small number of referees to begin with, might all have an impact as well. As written elsewhere, maybe it’s Euro-whites and american-white issue, not black/white. What we can be sure of is that it’s something, though not what is being reported.

  4. Rick Wilcox says:

    Kyle:
    A few rebuttals. First off the bat, we have the issue of asymmetric information being utilized to its fullest by the NBA, claiming “our data proves that you’re wrong” while simultaneously saying “we won’t let you see that data”. It’s a schoolyard game, not an honest response – especially when you take into consideration Litvin’s accusal that the researchers were “trying to prove that individual referees, black vs. white, white vs. black, showed bias”. While you say lamenting the lack of sufficiently granular data is “whining”, I’d say Litvin’s comment is closer to the target there.

    Second, and this one’s really short: If the granular data can be re-created via public information (game footage), then the granular data is already public information, and thus your argumentum ad djinn is seemingly moot. Especially in light of Dr. Bradbury’s suggestion:

    Supply them with the NBA’s data with the stipulation that the economists not reveal anything more than their findings. This removes the privacy concerns. After all, the NBA was willing to share its data with the private firm it hired to do the study. The data stays secret and we get an independent evaluation of the conflict.

  5. SkepSys says:

    I think the NBA is making a mistake by not releasing the database for study. Their attempts to discredit the research and issue positive talking points won’t remove the image of racism, which is not intended by the authors. It just seems petty and alienating to the fans. They should be more open with the community that researches their product and has the fans’ ear. The NBA has never been open to the research community though, not even the counting of blocks and steals until their monopoly was threatened by the ABA.

    Creating a database of referee calls from scratch is now very difficult or impossible. It would require: obtaining copies of the 1230 NBA games per year, presumably from the same source that is withholding the database; sitting through the games while tabulating (about 22 16-hour days, if the tabulating is immediate); and guess on the calls made offscreen; and then clean the errors. Not possible, but surely not a legal issue either, as the data is public, if not condensed. Although, the red tape of a corporate legal department can take a long time just to make that evaluation.

  6. SkepSys says:

    “about 22 16-hour days”
    Sorry, that should say – ’22 weeks of 16-hour days.’
    Assuming 2 hours per game, on average.

  7. Rachel Powell says:

    I think this whole racist referees issue is dumb. Since when do people watch a game and automatically assume a ref’s call is recist? I think it’s just an excuse to start lawsuits. I think all referees do their best especially at the professional level. I seriously doubt many, if ANY referees seriously base their foul calling on race. It’s just not ethical. They’re at the professional level, being paid a lot of money. Why would any of these referees do anything to jeopardize that? I just don’t agree with this study on racist referees. I think it’s a waste of time. Like the publisher said, no one knows which referee actually blew the whistle, so how can this study be correct?

  8. Rick Wilcox says:

    Rachel:
    I think this whole racist referees issue is dumb. Since when do people watch a game and automatically assume a ref’s call is racist? I think it’s just an excuse to start lawsuits. I think all referees do their best especially at the professional level. I seriously doubt many, if ANY referees seriously base their foul calling on race. It’s just not ethical. They’re at the professional level, being paid a lot of money. Why would any of these referees do anything to jeopardize that?
    It seems to me that you may have missed the point of the study. I highly doubt that the study is an attempt to show that referees are necessarily “racist”, or that they “base their foul calling on race” – the study’s trying to show that it’s possible even for someone who’s “do[ing] their best[...]at the professional level” to have some degree of prejudice, and that’s it’s quite possible to allow those prejudices to influence one’s judgment. One’s salary and outward professionalism does not preclude one from being influenced by non-professional, even subconscious, thought processes.

    just don’t agree with this study on racist referees. I think it’s a waste of time.
    And I think the study could give a valuable insight into the ability of prejudices and personal emotion to cloud professional judgment; referees aren’t machines, nor are they all-seeing. Compound that with the fact that technical fouls, which can include greater subjective judgment than regular fouls, were included in the study and we get a better picture of the study’s thrust – when the decision of whether or not to call a foul is not a purely binary “this happened or it didn’t” situation, the study attempts to investigate the question of whether or not the referee’s own mentality could have influenced their call.


    No one has said that “no one knows which referee actually blew the whistle”. In fact, the NBA seems to indicate that they have sufficiently granular data that they do, indeed, know who blew the whistle on each and every call of the 150,000 plays in their own study. They’re just not releasing that data outside of their own study.

  9. Jake says:

    Rachel,

    I think what this study is trying to pick up is not a conscious level of racial hatred. I think the word racist needs to be replaced by subconscious racial preference. Similar to that of the Implicit Association Test (IAT) study discussed by Malcolm Gladwell in Blink. It’s not stupid to study these issues. The more we can attempt to (ethically) learn about human behavior the better.