Michael Giberson
Al Roth quotes from an article in the Atlantic discussing the powerful incentives to publish badly done, probably false medical research dressed up as success. In a sense the problem is the same as with other academic “publish or perish” reward systems except the incentives in medical research can be much, much higher. The article centers on the work of medical-research researcher Dr. John Ioannidis, who believes part of the problem is that we expect researchers to always find success and continually be right:
“We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough. But as long as careers remain contingent on producing a stream of research that’s dressed up to seem more right than it is, scientists will keep delivering exactly that.
“Science is a noble endeavor, but it’s also a low-yield endeavor,” he says. “I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”
(Emphasis added by Roth)
I wonder if we could say the same thing about public policy. Are innovations in government policy also a “low-yield endeavor”? Should we be very comfortable with the fact that only a very small percentage of policy research is ever likely to lead to major improvements in social and economic outcomes and quality of life?
Should we reward politicians and government bureaus for producing a large stream of innovating, possibly-outcome-improving policies, or only for policy innovations that turn out to be outcome improving? The answer must depend on how difficult it is to judge the quality of “innovating, possibly-outcome-improving policies” and how difficult it is to measure whether a policy innovation was outcome improving.