Be Careful When You Criticize Experts

Yesterday, baseball and political analyst Nate Silver accused Harvard economist Greg Mankiw of academic intellectual dishonesty. It’s not a charge that ought to be thrown around lightly.

The type of tax cut that Romer and Romer think falls into this category is what they call an “exogenous” tax cut — one designed not to counter business cycles, but rather a “spontaneous” tax cut under relatively healthy economic circumstances.

This is very much not the type of tax cut that we are contemplating right now. Instead, what is being contemplated is a countercyclical action in an unhealthy economy designed to return the economy to normal growth. Romer and Romer are not all that keen on this type of tax cut; in fact, they argue that such “countercyclical fiscal policy is not achieving its intended purpose,” and that “policymakers’ efforts to adjust taxes to offset anticipated changes in private
economic activity have been largely unsuccessful”…

The thing is that’s really irksome is that Mankiw should know a lot better. This is not some random blogger at Townhall trying to parse a difficult economics paper and overlooking an important point of context — this is one of the premier economists in the world. He knows very well what the Romer and Romer paper says — and he’s made a deliberate choice to misrepresent it.

In poker terms, this is what we’d call a “tell”. Mankiw doesn’t have anything. He’s bluffing. Out of ideas. Taking one for the team, and touting the party line for shits and giggles. Except, this isn’t exactly fun and games, and Mankiw should leave the discussions to people who are serious about getting our economy moving again.

Nate is a smart guy—smart enough that he ought to know when he’s in over his head—but he misses the boat on this one, as Mankiw is “kind enough” to point out.

I usually don’t respond to blogosphere commentary on my work because, after all, time is scarce. But this critique by Nate Silver is noteworthy because the error it makes is so fundamental. It offers a teachable moment….

This argument raises the question: Why did the Romers focus on exogenous policy changes? The reason is that these are the only changes that can be used to reliability identify the effects of tax policy. If a tax change is made in response to some event, call it X, that influences the economy, it is hard to disentangle the effects of the tax change from the the direct effects of X. The Romers focus on exogenous tax changes for the same reason doctors conduct randomized drugs trials–not because they are interested in randomization as a prescriptive tool, but because randomization solves a statistical identification problem.

Imagine if a clinical doctor reasoned the same way as Silver did. He would say, “All the evidence on the effects of this drug are from randomized drug tests. In my practice, I never randomize treatment of my patients. Therefore, I can safely ignore the results from the randomized experiments.”

That is, of course, fallacious. We need the randomized experiments to inform us about the effects of medical interventions, even though interventions in practice are rarely randomized. Similarly, we need to consider the effects of exogenous tax changes, even though many actual tax changes are not at all exogenous.

The point isn’t that Mankiw “should know a lot better”; he does know better. That’s why he’s a professor at Harvard and one of the most respected economists in the world. It’s always possible that an expert is wrong—they make mistakes all the time—but, when preparing a critique of an expert you’re first thought should be “what am I missing?” not “what an idiot!” If you think an expert should know better, he/she probably does. Start from this premise that you are wrong, and reexamine the disagreement. If you still think the expert is wrong, proceed with caution in presenting your criticism. E-mail a friend with knowledge in the area, or even send an e-mail to the expert. Jumping ahead to calling someone a partisan hack is a less forgiving approach, especially if the mistake is on your end.

The “error” that Mankiw supposedly makes is so fundamentally correct that it does make for a teachable moment (in more than one respect)—the kind that will stay in Mankiw’s class lecture notes for years.


Addendum:
For all the people out there criticizing Mankiw for dissing the Romers, I just learned something interesting from Mankiw’s recent NPR interview.

— Mankiw was a friend of Christina’s in graduate school.
— Mankiw was best man at the wedding of David and Christina.
— David Romer was best man at Mankiw’s wedding.

If he’s doing anything that the Romers find offensive or inappropriate, I think he’s going to hear about it and it would cost him some close friends.

16 Responses “Be Careful When You Criticize Experts”

  1. Jesse says:

    Mankiw surely knows more about economics than Nate Silver (who is probably more concerned with drawing blog traffic than anything else).  But why do you think Mankiw is so clearly right on this point?

    “Randomized experiments” (or supposedly close approximations thereof) have advantages *and* disadvantages.  They are not a scientific magic bullet.

    Dani Rodrik had a post on his blog some time back about a randomized experiment indicating the benefits of distributing free bed nets in western Kenya.  His initial reaction was — “Jeffrey Sachs is right, giving out bed nets for free is the right policy.”  But critics immediately pointed out that he was jumping to conclusions.  Western Kenya is a particular place with particular social, economic, and political conditions; the particular program studied was implemented in a particular way (targeting pregnant women, for example).  The experiment in itself provides zero evidence about how the results would generalize if the same program was applied elsewhere.

    Mankiw mocks this position by saying “Well I guess we should never trust medical trials then, haw haw” but it’s a legitimate issue.  Interpreting the results of medical trials is a non-trivial task.

    Anyway, it’s too bad the author of that tax cut study is probably locked away somewhere and won’t have a chance to give her opinion on the matter to Obama.

  2. JC says:

    My point isn’t that Mankiw is right on fiscal policy here (though, I do happen to agree with him).  Silver accuses Mankiw of misusing a study with certain characteristics to serve his political preferences. Contrary to Silver’s assertion,  these characteristics did not make the study irrelevant but were employed to solve a serious problem inherent in this type of empirical research.

  3. Jesse says:

    I wasn’t actually talking about the fiscal policy, but about how to interpret the Romer/Romer study.

    Let’s use the example of the bed nets in Kenya (Rodrik’s post is linked under my name).

    Mankiw might say, “Look, here’s a randomized experiment showing that giving the nets away for free works.”
    Silver says, “This study only talks about a particular program in a particular place, and I have reason to think it won’t generalize.”
    Mankiw replies, “Randomized experiments are done for a very good reason.  They avoid problems that plague most other approaches.  We use randomized experiments in medicine.”

    This exchange doesn’t vindicate Mankiw, in my view.  Even though Mankiw is an expert, I don’t see how Silver is *wrong*.

    Now if you want to talk about Silver’s tone (as opposed to his substance) that’s another thing.

  4. Millsy says:

    Any suggestions on where to draw the line between ‘expert’ and ‘pompous blogger’? (Obviously, there’s no thin line in this case…but often times there can be).  I would argue that taking care in evaluating any claim in a public forum is a good idea.  I think we forget sometimes that posting things online is not just like debating with your buddies.  Anyone can see it, and it’s an easy way to look stupid…especially if your job depends on it.

    (side note–this is NOT a poke at JC’s blog)

  5. Ryan says:

    It seems to my that Silver is right, or at least not wrong, about the problems of going directly from the study to policy, but he is wrong in calling Mankiw a hack. The exogenous/endogenous problem that Silver identified is not trivial, but it is unavoidable in the Romer study.

    Mankiw is right to point out why the study made proper methodological choices, but should perhaps have used more caution in interpreting them. In sum, Silver had a point but it was lost when he started name-calling.

  6. tw says:

    Be careful when you criticize people much smarter than you, as you might find you’re just flat-out wrong.

    Also, Silver accused Mankiw of intellectual dishonesty, not academic.  There is a very, very large difference between the two.

  7. JC says:

    Thanks tw. Fixed. Being the first day of classes, I’m in syllabus-explanation mode, and academic dishonesty is one of my big topics. :-)

  8. Jason W says:

    Jesse and Ryan,
    Silver is “wrong” because he essentially calls Mankiw an idiot when he’s using the best guess from the appropriate methodology.  Using your bed net example, if one study finds a significant benefit in one area, it is true that it might not benefit another area.  However, to say that we should not try bed nets where malaria is a problem, just because it might not have similar results, seems foolish.  Furthermore, to call people intellectually dishonest when they are a proponent of bed nets is very bad form.  Silver is not wrong to say that Mankiw could be wrong, but Silver is wrong to suggest that Mankiw “should know a lot better”.

    Also, when you say randomized experiments have disadvantages, I think you mean they are not perfect.  I don’t think there are too many advantages of a non-randomized experiment over a randomized experiment.

  9. ADC says:

    JC, I’m wondering what your thoughts are regarding Nate’s response to the Mankiw response. I don’t claim to be an expert, but Nate’s explanation is fairly convincing to me. It seems like he has a valid criticism.

  10. JC says:

    This is a difficult issue to dissect, which is why I think Nate is having trouble understanding what is going on in the paper. Using exogenous taxes to evaluate their impact is like using a DIPS ERA to analyze pitchers. Yes, most taxes and runs allowed are generated in other contexts; however, their impact is clouded by other factors. The exogenous portion of the analysis is an asset, not a detriment to the analysis, as Nate is claiming. 

    I guess that because Nate is not familiar with econometric issues in public economics he’s not understanding what the Romers are intending to do. Mankiw is familiar with the econometric issues, and he appears to be reporting the results of the paper correctly.

    Here is what Tyler Cowen had to say about the paper in July 2007.

    Their work is of the very highest quality, and not to be confused with many of the more dubious claims made about taxation and investment.  In particular they make a point of isolating exogenous changes in the tax code.

    Here is the abstract of the paper (something that has strangely been missing from the discussion).

    This paper investigates the impact of changes in the level of taxation on economic activity. We use the narrative record — presidential speeches, executive-branch documents, and Congressional reports — to identify the size, timing, and principal motivation for all major postwar tax policy actions. This narrative analysis allows us to separate revenue changes resulting from legislation from changes occurring for other reasons. It also allows us to further separate legislated changes into those taken for reasons related to prospective economic conditions, such as countercyclical actions and tax changes tied to changes in government spending, and those taken for more exogenous reasons, such as to reduce an inherited budget deficit or to promote long-run growth. We then examine the behavior of output following these more exogenous legislated changes. The resulting estimates indicate that tax increases are highly contractionary. The effects are strongly significant, highly robust, and much larger than those obtained using broader measures of tax changes. The large effect stems in considerable part from a powerful negative effect of tax increases on investment. We also find that legislated tax increases designed to reduce a persistent budget deficit appear to have much smaller output costs than other tax increases.

    Also, you can read the NBER Digest for a more thorough description.

    Here is what Mankiw wrote in the NY Times:

    MIGHT TAX CUTS BE MORE POTENT? Textbook Keynesian theory says that tax cuts are less potent than spending increases for stimulating an economy. When the government spends a dollar, the dollar is spent. When the government gives a household a dollar back in taxes, the dollar might be saved, which does not add to aggregate demand.

    The evidence, however, is hard to square with the theory. A recent study by Christina D. Romer and David H. Romer, then economists at the University of California, Berkeley, finds that a dollar of tax cuts raises the G.D.P. by about $3. According to the Romers, the multiplier for tax cuts is more than twice what Professor Ramey finds for spending increases.

    Why this is so remains a puzzle. One can easily conjecture about what the textbook theory leaves out, but it will take more research to sort things out. And whether these results based on historical data apply to our current extraordinary circumstances is open to debate.

    Christina Romer, incidentally, has been chosen as the chairwoman of the Council of Economic Advisers in the new administration. Perhaps this fact helps explain why, according to recent reports, tax cuts will be a larger piece of the Obama recovery plan than was previously expected.

    (emphasis added)

    I’m not seeing anything that rises to intellectual dishonesty.

  11. Jesse says:

    However, to say that we should not try bed nets where malaria is a problem, just because it might not have similar results, seems foolish.

    Just to clarify, the debate about bed nets is usually whether they should be given away for free, or whether they should be sold at a discount.  The question is, which method of distribution will maximize their use and save the most lives?  From my understanding, this is a contentious issue and there are strong opinions, based on a variety of sources, on both sides.  Is the experimental finding in Kenya a silver bullet that puts to rest the debate, or even shifts it strongly from one side to another?  I don’t think so (and perhaps more importantly neither does Rodrik, who after all is a expert and therefore not to be reckoned with).

    As for Silver’s tone, well it looks fairly obvious to me that he has a website to run and taking a shot at Mankiw was probably good for advertising revenue.  On the other hand, see expert Brad DeLong’s comment on Mankiw under my name.
    “It is somewhat puzzling that Mankiw appears to believe that the Romers do think that tax multipliers are larger than spending multipliers, as they do not, and this is something that he could have very easily checked.” — this is polite but carries the same implication as what Silver was saying, doesn’t it?

  12. JC says:

    Jesse,

    It’s good for advertising revenue to call Mankiw intellectually dishonest? Is that some sort of justification? To excuse him on these grounds would be despicable. I hope you are not arguing this.

    As for Brad DeLong, he’s not being polite, he’s arguing relevant points which Mankiw would agree are up for debate. DeLong also understands why the Romers did what they did, unlike Nate. The argument over multipliers hasn’t been settled by DeLong. Economists have been arguing over multipliers since before I was born.  DeLong and Mankiw can debate this as fellow economists and former colleagues (I believe they were both at Harvard at the same time).

    Here are some previous posts by Mankiw regarding the Romer paper.

    Spending and Tax Multipliers

    Obama’s Multipliers

    That is, Team Obama assumes that tax changes are less than half as potent in influencing the economy as the new CEA Chair estimated them to be in her own research.

    Of course, it is prudent to take any research, including that of the very careful, very sensible Romers, with a grain or two of salt. The same can be said of the mainstream models on which Team Obama is relying.

    Finally, I find it odd that Mankiw would purposely misrepresent the work of a coauthor.

  13. Jason W says:

    Jesse,
    Thanks for the clarification, but just to make sure I’m clear…

    If the best research says bed nets should be given away for free, fine.  If they should be sold at a discount, fine.  I don’t know.  My point is that in both cases we should try to use the best research available.  Your point above that it is not a “magic bullet” is right, the best research could be wrong.  However, we have to make policy.  So, we should use the best research available.

    Also, I’m not a macroeconomist, so I don’t know if Mankiw is right.  DeLong’s comment may be right for all I know.  However, I think Silver’s explanation is confusing at best and wrong at worst (the latter I think).  His point is that this is not an exogenous tax cut, so we shouldn’t use Romer and Romer’s results.  Mankiw’s point, which is correct, is that we can only estimate the effects of exogenous tax cuts.  This doesn’t mean exogenous and endogenous tax cuts have different effects, it’s just that during endogenous tax cuts a lot of stuff is going on so we can’t estimate it.  So even though this tax cut is endogenous, we estimate it with exogenous tax cuts.  It seems to me that Silver is criticizing Mankiw for using (or citing) the correct methodology.

  14. Mitch says:

    I am not capable of contributing to the discussion on an intellectual level since I do not have the background that permits my understanding of the highly technical nature of the issue at hand, but I did want to pass along a further discussion of the issue.  Here is Mark Thoma discussing DeLong’s response.

    http://economistsview.typepad.com/economistsview/2009/01/whos-the-honest-broker.html

  15. Jesse says:

    JC – I’m not intending to *defend* Silver’s tone, just to explain my (half-baked) theory on what motivated him.  (This is not a particularly good analogy, but I thought it was like wondering why Ann Coulter uses such nasty language and simple-minded arguments in her book — cha-ching.  I could be very wrong.)

    Jason W — I guess my point is that while it sounds nice to “use the best available research,” people can (and do) disagree on what the best available research is.  A more scientific experimental design is good in many ways, but if the scientific improvements don’t address the question of generalizability, then it seems silly to say that this research demands priority when you have to generalize the results to apply them.

    True, this may be a somewhat esoteric  point to make on a baseball stats blog (the next time I see a standard error estimate on a baseball projection will be the first).

  16. true999 says:

    In the interest of “intellectual honesty”, I think it relevant to disclose that Manikow is the same guy who said, in 2004 that outsourcing jobs to other countries is a good thing for the USA and went on to predict the addition of millions of new jobs that would be added to the economy.

    I’d be real careful praising this guy.

    Actually, I’d like to take him on tour of Ohio and watch him “teach” the folks who have watched their jobs go overseas and towns dry up.

    Manikow does not belong on a pedestal.