Category Archives: Science

A call for controlled experimentation in California’s energy efficiency programs

Posted on by

Michael Giberson

UC-Berkeley economist Catherine Wolfram has an op-ed in the Sacramento Bee advocating the state use controlled experimentation to discover with energy efficiency programs work best. As she explains, retailers are increasingly using experimentation and advanced data analysis to discover how to increase sales. Surely, she suggests, when planning to spend nearly half a billion tax dollars annually on energy efficiency California ought to devote a bit of effort into separating programs that sound good and work well from those that merely sound good.

[HT to Elizabeth M. Bailey at Energy Economics Exchange.]

Doing what seems like it should work: Experiments, tests, and social progress

Posted on by

Michael Giberson

My title is a little grand, at least the “and social progress,” but maybe it will be justified in some later, more carefully worked out version of the ideas clashing about in my head. As this is a blog, I’m sharing the more immediate, less carefully worked out version. ;-)

I’ve been reading Redirect: The Surprising New Science of Psychological Change. My wife brought it home from the library and then recommended it to me. (Thanks!) The book makes some surprisingly strong claims for personal improvements from what the author calls “story editing,” a bundle of techniques that subtly (or sometimes not so subtly) get people to revise their self narratives. (More from the Scientific American blog, more from Monitor on Psychology.)

Counterpart to that focus of the book is an emphasis on testing social and psychological interventions to discover what actually seems work. Author Timothy Wilson details numerous self-help and social change projects, some of which capture millions or even billions of dollars in public support, which seem like they should work but when subjected to careful evaluation show no evidence of success. In fact, some very expensive programs actually seem to worsen the problem that the program was designed to fix: programs to fight teenage smoking that lead to higher smoking rates, programs to discourage teenage pregnancy that lead to higher pregnancy rates, efforts to discourage littering – or cheating – on campus that have the opposite effects. Wilson advocates a strong preference for testing social interventions with randomized control experiments when possible (and ethical). When randomized control tests are not possible, then other attempts at measurement and replication are important even though difficult to do well.

Whether or not “story editing” is key to successful personal and social change – Wilson makes a strong case, but he could be cherry picking his evidence and I’m sure he has his professional critics – the emphasis on experimentation and testing interventions is an important one.

Lynne’s posts last week on experimentation in social contexts are related: Economic experimentation, economic growth, and regulation and Experimentation, Jim Manzi, and regulation/deregulation. I’m most of the way through Russ Robert’s EconTalk interview with Jim Manzi that Lynne mentioned in second-listed link (recommended); Manzi makes related arguments in favor of well-designed experiments where possible, and for trial and error experimentation where controlled experimentation is not possible.

In both Wilson’s book and the Manzi interview (and apparently in Manzi’s book Uncontrolled, which I haven’t read yet), the limits of multivariate analysis of naturally generated data – i.e., almost all econometric analysis – are examined and found wanting. As Manzi explains, “omitted variable bias” is massive when examining data on human systems; the systems are simply too complex to produce reliable, non-obvious predictions via multivariate analysis because you cannot control for all of the possible effects and interactions influencing the data. He suggests that while 90 percent of studies relying on well-designed randomized control experiments are subsequently replicated, that figure drops to 20 percent or so for studies relying primarily on well-designed multivariate analysis.

In a post on the deterrence effect of the death penalty, Timothy Taylor provides an example of the difficulties of using multivariate analysis to examine social policy. Taylor draws on a recent National Research Council study on the topic, which like a similar study published in 1978 has concluded “available studies provide no useful evidence on the deterrent effect of capital punishment.” Taylor then explains several reasons why it has been hard to draw firm conclusions from the data. While he doesn’t use the term “omitted variable bias,” it is among the problems that the NRC study finds hampering results in this area.

The views of both Wilson and Manzi, and the case study on the effects of the death penalty, all point to a certain humility concerning our claims to understand how the world works. But humility isn’t the end of the story, it isn’t an argument to stop; it is an argument to trust our beliefs about the social world less conclusively and also to trust them selectively: trust knowledge derived from replicated randomized-control experiments most, trust knowledge from replicated multivariate analysis much less, trust knowledge based on trial and error learning less as well.

These ideas will, once better worked out in my head, probably also mention Vernon Smith’s work on constructivist and ecological rationality. Of course, V. Smith is known to be a fan of experimental approaches to understanding social phenomena as well.

The constructivist way forward: experimentation! testing! social progress!

Adam Smith and mirror neurons paper published

Posted on by

Lynne Kiesling

I mentioned a while ago my working paper on the neuroscience research on mirror neurons and its relevance for Adam Smith’s theory of sympathy developed in The Theory of Moral Sentiments (1759). After revision and some extremely helpful referee guidance, the paper has been published in The Review of Austrian Economics:

Mirror neuron research and Adam Smith’s concept of sympathy: Three points of correspondence

In The Theory of Moral Sentiments, Adam Smith asserts that humans have an innate interest in the fortunes of other people and desire for sympathy with others. In Smith’s theory, sympathy is an imperfectly reflected combination of emotion and judgment when one observes someone (the agent) in a particular situation, and imagines being that person in that situation. That imagination produces a degree of interconnectedness among individuals. Recent neuroscience research on mirror neurons provides evidence consistent with Smith’s assertion, suggesting that humans have an innate capability to understand the mental states of others at a neural level. A mirror neuron fires both when an agent acts and when an agent observes that action being performed by another; the name derives from the “mirroring” of the action in the brain of the observer. This neural network and the capabilities arising from it have three points of correspondence with important aspects of the Smithian sympathetic process: an agent’s situation as a stimulus or connection between two similar but separate agents, an external perspective on the actions of others, and an innate imaginative capacity that enables an observer to imagine herself as the agent, in the agent’s situation. Both this sympathetic process and the mirror neuron system predispose individuals toward coordination of the expression of their emotions and of their actions. In Smith’s model this decentralized coordination leads to the emergence of social order, bolstered and reinforced by the emergence and evolution of informal and formal institutions grounded in the sympathetic process. Social order grounded in this sympathetic process relies on a sense of interconnectedness and on shared meanings of actions, and the mirror neuron system predisposes humans toward such interconnection.

If you are not a subscriber and would like to read the paper, the manuscript version is available on my SSRN page.

Monsters of Grok t-shirts

Posted on by

Lynne Kiesling

Here’s some outstanding geek attire! Monsters of Grok is a line of t-shirts that use rock band t-shirt logo designs, but the names are instead famous scientists and intellectuals such as Ada Lovelace (done as a Ladytron logo), Isaac Newton (as Iron Maiden), and Benjamin Franklin (as Black Flag). I fell over laughing when I first saw these, literally hyperventilating and weeping. Guess that makes me a geek rocker …

Today, to make myself feel better for having such a nasty ear infection (with gratitude to those of you who have sent get well wishes!), I finally broke down and purchased two of them. The first one’s easy to guess if you’re a regular KP reader, the second one is a little more tricky as there were several contenders. If you guess them both you get a gold star!

Economics of power market design compared unfavorably to climate science

Posted on by

Michael Giberson

From the Harvard Electricity Policy Group meeting in February 2011. By convention the meetings are off-the-record, so the speaker’s name is not identified in the summary:

I think the most important distinction between the fields of climate science and economics for me is the question of evidence. Science is characterized by a subtle interplay between conceptual models and the evidence that supports or contradicts them. There’s a rigorous process of analyzing and evaluating evidence and improving or discarding the conceptual models as the evidence dictates. In economics, evidence can often be harder to come by and more ambiguous in nature. This instance is a strong case in point. There is no real precedent. The markets are brand new. And with a few exceptions, the RTO regions have been basically in capacity surplus since the markets came into being for reasons having nothing to do with the capacity markets themselves.

Where evidence is lacking, theorists can find themselves somewhat less constrained. Under these circumstances, whichever side has the loudest voices or the most money or the most impressive resumes can dominate the conversation. This should never be mistaken as proof that their position are correct.

[...]

I’m aware that many will argue, and have argued, that a focus on market efficiency will in the long run lead to the greatest consumer benefit. This may be true in a nonexistent, two-sided perfect market with no barriers to entry. But it is a tenuous article of faith when applied to real electricity markets. And given the untold billions in costs to get to that uncertain future, it’s no wonder that consumer advocates basically unanimously are not eager to take that bet.

The implementation of capacity markets based on these unproven theories has already led, predictably, to the transfer of tens of billions of dollars of ratepayer wealth to generation owners. I say predictably because this outcome was clearly anticipated by all parties and articulated by many. The whole point was to raise costs. On the other hand, there’s not a shred of hard evidence that this process has led to new generation where it is most needed, or to avoided retirements of needed capacity or to cost-saving transmission investments. These are the ostensible purposes of the construct. There is no reason to believe that it would. It’s just too good an arrangement for existing generation owners as it is.

The speaker observes that capacity markets have also spurred development of demand-side resources, but this “positive benefit … has come at an astronomical cost.”

As an alternative to capacity markets, the speaker suggests a combination of state-sponsored investments, long term contracts, and short term spot markets. Not that he presents any evidence that this approach will work better for consumers, it just seems good to him. I wonder, scientifically speaking, why not just examine the existing evidence on prices and investments in “energy only” power markets in Texas, Alberta, and Australia?

Jonah Lehrer on voter ignorance

Posted on by

Lynne Kiesling

It shouldn’t surprise you to find, given my recent working paper on Adam Smith, sympathy, and mirror neurons that I am an avid reader of neuroscience writer Jonah Lehrer. His post today riffs off of President Obama’s birth certificate to muse on voter ignorance. In discussing some research on the subject, he observes

Why does more education lead to less accurate beliefs? The answer returns us to the difference between rational voters (what we think we are) and rationalizing voters (what we really are). It turns out that the human mind is a marvelous information filter, adept at blocking out those facts that contradict what we’d like to believe.

It sounds like he and Bryan Caplan should have a little blog exchange on neuroscience’s implications for the results of Bryan’s book The Myth of the Rational Voter. It also sounds like yet another reason why we should strive to make as few important decisions as possible through political means.

Bainbridge’s broad brush criticisms on empirical legal studies slams all interdisciplinary legal work

Posted on by

Michael Giberson

Criticisms of the growing field of empirical legal studies by UCLA law professor  Stephen Bainbridge were issued in such broad brush strokes that he ended up blasting just about every law academic engaged in any sort of interdisciplinary work, especially so if the academic seeks to examine data of some sort. The main claims showed up recently in a National Law Journal article, which quoted Bainbridge:

“A lot of the people I see who are empiricists, often with doctorates in the social sciences, aren’t very good lawyers,” he said. “I’ve read numerous papers that just got the law wrong. The problem is that we’re hiring people with Ph.D.s in other fields, but their law credentials are middling at best. Someone who is a brilliant economist wants to be in a economics department, so we get second-rate lawyers who are second-rate in their academic field.”

Perhaps phrasing the criticism in that way touched a nerve with Josh Wright, a law professor at George Mason University who holds both a PhD in economics and a law degree from UCLA. Wright responds at Truth on the Market, noting among other things that Bainbridge is asserting many facts about the state of the world without actually pointing to any evidence (much less adequately testing the evidence once it is identified).

John List’s $10 million crazy idea field experiment in education

Posted on by

Michael Giberson

Bloomberg Markets Magazine has a feature on economist John List and his $10 million research project on education. Along the way we get an introduction to List’s work on field experiments in economics, a splash of lab-based economics back story, and the reaction of education specialists who think List’s project is wholly off target.

List, along with collaborators Steven Levitt and Roland Fryer, has obtained a $10 grant for a program which randomly assigned 3-5 year old students to one of three groups: (1) free all-day preschool, (2) “parenting academy” for the student’s parent or guardian, or (3) a control group with neither intervention. The program intends for follow the students into adulthood in order to assess the long-term effects of the intervention.

List says he doesn’t know much about education theory, so he enlisted specialists to consult on the preschool curriculum. One such consultant, Clancy Blair, a New York University professor of applied psychology, says he was astonished by the size of the project and by how it focuses on financial incentives without looking at such variables as how the parents interact with their children.

“That’s a crazy idea,” says Blair, who studies how young children learn. “It’s not based on any prior research. This isn’t the incremental process of science. It’s ‘I have a crazy idea and I convinced someone to give me $10 million.’”

List says too many decisions in fields from education to business to philanthropy are made without any scientific basis. Without experimenting, you can’t evaluate whether a program is effective, he says.

“We need hundreds of experiments going on at once all over the country,” he says. “Then we can understand what works and what doesn’t.” …

“What educators need to know are what are the best ways to educate kids, and this is trying to short-circuit that,” Blair says. “We have fundamental problems in education, and this is sort of a distraction.”

List says he understands the objections. “If I was in the field, I’d hate me, too,” List says in November while driving to his sons’ indoor baseball practice in one of Chicago’s south suburbs. “There should be skeptics.”

Ethanol blather courtesy of the EPA and the Financial Times

Posted on by

Michael Giberson

On Friday, the EPA announced it now believes it is safe to use fuels made with up to 15 percent ethanol in cars, SUVs and light-duty trucks manufactured between 2001 and 2006.  EPA administrator Lisa Jackson said, “Whenever sound science and the law support steps to allow more home-grown fuels in America’s vehicles, this administration takes those steps.” It is, as the Houston Chronicle reported, “a victory for ethanol advocates, including manufacturers, corn farmers and their supporters on Capitol Hill.”

“Sound science and the law?” The EPA action was to grant a waiver on a Clean Air Act-based limitation on selling gasoline with more than 10 percent ethanol for the vehicle classes mentioned. While the Clean Air Act also allows for waivers to be granted, I’d say that the law is no more than neutral on the matter. The essence of  ”sound science” is open inquiry, but it looks like the EPA pursued a rather selective course of study in order to grant this favor to the ethanol industry. From the press release it looks like all that the EPA considered was whether or not use of higher-content ethanol blends would damage emissions control systems. I think a broader inquiry is needed before concluding sound science supports steps to allow more “home-grown fuels.”

The Financial Times Energy Source blog quoted Jackson on “sound science and the law” and followed with quotes from sundry other lobbyists (a “move in the right direction,” “will further increase volatility in food markets,” all mandates “should be repealed,” “takes food out of the mouths of American consumers”). A somewhat limited range of reporting perhaps, but besides lobbyists pro and con who are you going to get to talk about an EPA announcement that comes out on a Friday?

Which is fine except that the FT blogger added, “What nobody is touching on, however, is that the US produces so much ethanol that it has been exporting it.” (The linked story in the FT reports, producers are making so much they are “running out of places to put this ethanol.”) And, the blogger continues, “Raising the content of ethanol in fuels at home will at least put more of the ethanol produced in the US in this country’s vehicles. … if we are going to produce ethanol, it does seem wrongheaded that this country is giving tax credits for the production of a biofuel that is shipped abroad to lower emissions elsewhere.”

Some modest amount of ethanol is used in gasoline as an oxygenate, allowing the gasoline to burn more cleanly (and so providing some local air quality benefits), but the primary alleged public policy reason to further subsidize and mandate use of ethanol is due to its supposed carbon neutrality. Since greenhouse gas accumulations are no respecters of national boundaries, it doesn’t really matter where in the world the ethanol gets burned for us to (allegedly) gain this (supposed) benefit.

I say, let’s let ethanol supporters ship as much as possible overseas. At least then my own vehicle isn’t at risk.

NOTE: Robert Rapier also comments on the decision, adding some informed speculation on how the politics are likely to play out. (Hint: “the ethanol industry usually gets what they wants.”)

Incentives and fake success in medical research and public policy

Posted on by

Michael Giberson

Al Roth quotes from an article in the Atlantic discussing the powerful incentives to publish badly done, probably false medical research dressed up as success. In a sense the problem is the same as with other academic “publish or perish” reward systems except the incentives in medical research can be much, much higher.  The article centers on the work of medical-research researcher Dr. John Ioannidis, who believes part of the problem is that we expect researchers to always find success and continually be right:

“We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough. But as long as careers remain contingent on producing a stream of research that’s dressed up to seem more right than it is, scientists will keep delivering exactly that.

“Science is a noble endeavor, but it’s also a low-yield endeavor,” he says. “I’m not sure that more than a very small percentage of medical research is ever likely to lead to major improvements in clinical outcomes and quality of life. We should be very comfortable with that fact.”

(Emphasis added by Roth)

I wonder if we could say the same thing about public policy. Are innovations in government policy also a “low-yield endeavor”? Should we be very comfortable with the fact that only a very small percentage of policy research is ever likely to lead to major improvements in social and economic outcomes and quality of life?

Should we reward politicians and government bureaus for producing a large stream of innovating, possibly-outcome-improving policies, or only for policy innovations that turn out to be outcome improving? The answer must depend on how difficult it is to judge the quality of “innovating, possibly-outcome-improving policies” and how difficult it is to measure whether a policy innovation was outcome improving.