This morning, I ran across an article by Alan Barra in the WSJ that reminded me of a blog post that I have been meaning to write for several years. Barra discusses the ability of players to perform in “clutch” moments. In closing, Barra cites Bill James as an agnostic regarding clutch ability, citing the last line of his article Underestimating the Fog, “Let’s not be too sure that we haven’t been missing something important.”
James’s article caused a bit of a stir when it was first published. Here was James arguing that several common notions among sabermetricians—including that clutch ability is a myth—were not necessarily so. As James metaphorically stated the problem, clutch hitting lay in a fog beyond a sentry, on the lookout for approaching forces. In a thick fog, the enemy may be invisible despite existing in strong numbers. The fog that obscures the view of the guard is similar to the fog that shrouds the randomness in baseball that makes it difficult for us to identify ability from chance. While we have methods for disentangling luck from ability, there exists the possibility that clutch ability is real and we just haven’t found a way to see through the fog properly. Therefore, we shouldn’t be so quick to believe an idea to be true, even when the bulk of the evidence we have indicate that it is true. Maybe the truth is just lost in the fog.
No one can deny this. Of course it is possible that clutch ability exists and we just haven’t found a way to measure it properly. But we dismiss lots of other possible events as likely outcomes everyday with good reason. And the tradeoff of acting with too little evidence must be balanced against not acting with sufficient evidence. It’s a dilemma familiar to all scientists. This is explained with the distinction between type I and type II errors.
Let’s begin with the null hypothesis that player performance in clutch situations is identical to performance in non-clutch situations. A type I error occurs when we reject a correct null hypothesis. Studies of clutch hitting find that performance differences in these situations are small and often not statistically meaningful. The null stands and clutch-hitting skill is seen as a myth. A type II error occurs from not rejecting an incorrect null hypothesis. When James advocates agnosticism towards clutch-hitting as a skill, it is because that despite the studies showing little evidence of clutch-hitting he wants to avoid committing type II error. The problem is, this choice between type I and II errors isn’t free. By raising the decision criterion to avoid type II error, you necessarily increase the chance of committing type I error.
Identifying clutch hitting is practical problem that requires a decision involving real costs. Should a team factor in clutch ability when choosing between free agents. Should it matter for the manager choosing among pinch hitters? Should a historically big-game pitcher start the playoff series over your regular season ace? Based on the available evidence, if I had to decide between Jeter or A-Rod it’s not even close: Alex Rodriguez is a far superior player to Derek Jeter, and that’s what is relevant. And in cases were the players’ performances are more similar, I wouldn’t consider clutch performance for even a moment. If clutch ability exists, it would show up in bunches using the empirical methods already employed by researchers seeking to study the question.
In my view, the fog is a distraction: something to bring up to keep the argument going. But arguing takes time, which is valuable. Let’s stop it with the fog, already. Of course it’s possible that something exists that just hasn’t been discovered yet (e.g.the Loch Ness Monster, Sasquatch, ergogenic effects of HGH); but the evidence we have says these things don’t exist, and hanging hopes on the possible isn’t a very persuasive argument.