Michael Giberson
Al Roth quotes from an article in the Atlantic discussing the powerful incentives to publish badly done, probably false medical research dressed up as success. In a sense the problem is the same as with other academic “publish or perish” reward systems except the incentives in medical research can be much, much higher. The article centers on the work of medical-research researcher Dr. John Ioannidis, who believes part of the problem is that we expect researchers to always find success and continually be right:
“We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough. But as long as careers remain contingent on producing a stream of research that’s dressed up to seem more right than it is, scientists will keep delivering exactly that.
“Science is a noble endeavor, but it’s also a low-yield endeavor,” he says. “I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”
(Emphasis added by Roth)
I wonder if we could say the same thing about public policy. Are innovations in government policy also a “low-yield endeavor”? Should we be very comfortable with the fact that only a very small percentage of policy research is ever likely to lead to major improvements in social and economic outcomes and quality of life?
Should we reward politicians and government bureaus for producing a large stream of innovating, possibly-outcome-improving policies, or only for policy innovations that turn out to be outcome improving? The answer must depend on how difficult it is to judge the quality of “innovating, possibly-outcome-improving policies” and how difficult it is to measure whether a policy innovation was outcome improving.
What’s special about public-policy conclusions is that we tend to vote on them (by voting for the politicians who support or oppose them) long before there’s any reasonable way to judge their rightness (effectiveness in improving outcomes), not to mention the difficulty of getting much of the public to agree upon any reasonable means to evaluate rightness. There’s zero mileage to be had politically in being right, only in looking and feeling right to the public. Individually as a matter of principle we can ask ourselves whether we want to support only policies that have high chances of being right by our own measures, and not those that have more modest chances of being right. But there aren’t many of us who will take the trouble!