Cass Sunstein on regulatory analysis and the knowledge problem

Cass Sunstein begins:

With respect to the past and future of regulation, there are two truly indispensable ideas. Unfortunately, they are in serious tension with one another. Potential solutions lie in three reforms, all connected with democracy itself – but perhaps not quite in the way that most people think.

The first indispensable idea is that it is immensely important to measure, both in advance and on a continuing basis, the effects of regulation of social welfare. As an empirical matter, what are the human consequences of regulatory requirements? That is the right question to ask, but inside and outside of government, it is tempting to focus on other things. […]

At the present time, the right way to answer the right question is to try to identify the costs and benefits of regulations, in order to catalogue and to compare the various consequences, and to help make sensible tradeoffs. To be sure, cost-benefit analysis can create serious challenges, and at the present time, it is hardly a perfect tool. Nonetheless, it is the best one we now have. Some people do not love cost-benefit analysis, but they should. If we want to know about the real-world effects of regulation, that form of analysis deserves a lot of love.

The second idea, attributable above all to Friedrich Hayek, is that knowledge is widely dispersed in society. As Hayek and his followers emphasize, governments planners cannot possibly know what individuals know, simply because they lack that dispersed knowledge. The multiple failures of plans, and the omnipresence of unintended consequences, can be attributed, in large part, to the absence of relevant information. Hayek was particularly concerned about socialist-style planning. He contended that even if socialist planners are well-motivated and if the public interest is their true concern, they will fail, because they will not know enough to succeed. Hayek celebrated the price system as a “marvel,” not for any mystical reason, but because it can aggregate dispersed information, and do so in a way that permits rapid adjustment to changing circumstances, values, and tastes.

In sum, while it is immensely important to measure the effects of regulation, we may lack the necessary information.

Sunstein notes, pragmatically, that regulators have access to substantial amounts of information and methods of cost-benefit analysis are well-established and improving. He wrote, “In the day-to-day life of cost-benefit analysis, regulators are hardly making a stab in the dark.” He continues:

Nonetheless, modern followers of Hayek are correct to emphasize what they call “the knowledge problem,” understood as the government’s potential ignorance, which can be a problem for contemporary regulators of all kinds […]

The tension, in short, is that regulators have to focus on costs and benefits (the first indispensable idea), but they will sometimes lack the information that would enable them to make accurate assessments (the second indispensable idea). … In light of the knowledge problem, can they produce reliable cost-benefit analyses, or any other kind of projection of the human consequences of what they seek to do, and of potential alternatives?

Sunstein identified three reforms he said respond to the first indispensable idea while helping overcome or mitigate the limits imposed by the knowledge problem: First, modern “notice-and-comment” rulemaking processes; second, retrospective analysis of regulations; and third, careful experiments and especially randomized control trials. As pragmatic responses to knowledge problems, each of the three have something to contribute.

Is it enough?

What would Hayek say? Sunstein responded to this question in a footnote:

I am not suggesting that Hayek himself would be satisfied. Consider this remarkable passage:

This is, perhaps, also the point where I should briefly mention the fact that the sort of knowledge with which I have been concerned is knowledge of the kind which by its nature cannot enter into statistics and therefore cannot be conveyed to any central authority in statistical form. The statistics which such a central authority would have to use would have to be arrived at precisely by abstracting from minor differences between the things, by lumping together, as resources of one kind, items which differ as regards location, quality, and other particulars, in a way which may be very significant for the specific decision. It follows from this that central planning based on statistical information by its nature cannot take direct account of these circumstances of time and place and that the central planner will have to find some way or other in which the decisions depending on them can be left to the “man on the spot.”

Hayek [“The Use of Knowledge in Society,” AER, 1946]. In my view, the claim here is a mystification, at least as applied to the regulatory context. Statistical information “by its nature” can indeed “take direct account of these circumstances of time and place.” Of course it is true that for some purposes and activities, statistical knowledge is inadequate.

It is an odd note.

Sunstein quoted Hayek, said Hayek’s point is a mystification, then admitted, “Of course it is true….” I guess Sunstein’s point is that it is a true mystery?

In any case, we should note few things about Sunstein’s reforms. First, his focus is not so much on the doing of regulation itself, but rather on regulatory oversight. His three reforms are not to be applied in day-to-day regulation, but rather serve as external correctives. A related and perhaps more fundamental issue concerns the manner in which regulatory operations can facilitate the production and coordination of knowledge in ways that promote better outcomes.

Second, extending this last point, as we compare government and market processes, we note the relative power of feedback processes in the private sector and the weakness of feedback processes in the public sector. If you are dissatisfied with your service in a restaurant, you can take choose to eat elsewhere next time. If you are dissatisfied with your service at the Department of Motor Vehicles (or the Veterans Administration or the local Zoning Board, etc.), you have many fewer options. Feedback processes are essential to the production and coordination of knowledge. How can regulatory agencies learn in an environment in which feedback processes are so significantly attenuated?

Third, Sunstein is operating with a rather flat view of knowledge. That is to say, in his explanation knowledge and information and data are various names for more or less identical things. But if we take seriously Sunstein’s remark “of course it is true that for some purposes and activities, statistical knowledge is inadequate,” then further questions arise. Sunstein does not explore the point, but it is exactly here, for Hayek, that the force of the knowledge problem emerges.

There is a research agenda here.

Obviously Sunstein endorses further research and development of benefit-cost analysis, expansion of notice-and-comment rulemaking processes, retrospective regulatory analysis, and experimental tests of regulation. Benefit-cost analysis has a dedicated academic society and journal and book level treatments. Sunstein discusses efforts to broaden understanding of regulatory proposals and encourage public engagement (for example, the government’s and the university-based outreach project Retrospective regulatory analysis and experimental tests get less attention, but a number of academic programs do at least some retrospective work (for example: the Mercatus Center Regulatory Studies Program at GMU, the Penn Program on Regulation, the George Washington University Regulatory Studies Center). As Sunstein notes, a number of federal agencies have committed to using experiments to help understand the impact of regulation.

How far can these efforts get us in the face of the knowledge problem? For which regulatory “purposes and activities” is it the case that “statistical knowledge is inadequate”? Are there patterns in this inadequacy that bias or undermine regulatory action? Assuming Sunstein’s reforms are fully implemented, what residual knowledge problems would continue to trouble regulation?

A good place to start is Lynne Kiesling’s article “Knowledge Problem” (obviously!) which appeared in the 2014 volume Oxford Handbook of Austrian Economics. Mark Pennington’s book Robust Political Economy examines how knowledge problems and incentive problems can frustrate political activity. Obviously, too, Hayek’s own “The Use of Knowledge in Society” and “Economics and Knowledge” are relevant, as is the later “Competition as a Discovery Process.” Don Lavoie’s “The Market as a Procedure for Discovery and Conveyance of Inarticulate Knowledge” condenses Hayek’s statements on the knowledge problem and further explains why the problem cannot be overcome merely by further developments in computer technology.

Going deeper, one might explore James Scott’s book Seeing Like a State, which emphasizes how the top-down processes of measurement and aggregation can strip meaning from knowledge and result in destruction of value. Then perhaps a work like Hans Christian Garmann Johnsen’s new book The New Natural Resource: Knowledge Development, Society and Economics might have something to say. (I’ve only just begun looking at Johnsen’s ambitious book, so it is too soon to judge.) Complementary to work on institutional frameworks and knowledge would be close studies of government agencies, like the Pressman and Wildavsky classic Implementation: How Great Expectations in Washington are Dashed in Oakland… and broader surveys of policy history such as Michael Graetz’s The End of Energy or Peter Grossman’s U.S. Energy Policy and the Pursuit of Failure.

And more.

Here I have focused just on those parts of the Sunstein article where he bumps up against the knowledge problem most explicitly. He explores each of the three reforms in more detail, providing much more to think about.

Recommended reading for regulatory analysts.


Sunstein’s paper is “Cost-Benefit Analysis and the Knowledge Problem.” Regulatory Policy Program Working Paper RPP-2015-03. Cambridge, MA: Mossavar-Rahmani Center for Business and Government, Harvard Kennedy School, Harvard University.

History of thought course video: Hayek and the knowledge problem

Not surprisingly, given the title of this blog and the focus of my research, the last video in the series for my history of economic thought course provides an introduction to Hayek and the knowledge problem.

Hayek’s work in the 20th century explored a range of ideas, one of the most important of which was the argument that the fundamental economic challenge in a society is the coordination of plans and actions among agents in an economy, all of whom have diverse goals and make choices based on their own perceptions and private knowledge. The knowledge problem has implications for questions from the socialist calculation debate of the 1920s and 1930s to the modern policy analyses of the regulatory state.

Happy birthday Hayek!

Lynne Kiesling

Today’s Hayek’s birthday, a worthwhile landmark for reflection on his work and why it’s important to read. I assign “The Use of Knowledge in Society” in every class I teach, and I recommend it if you haven’t yet read it. Here Hayek argues that the fundamental economic problem societies face is not the allocation of a given set of resources based on a given set of preferences and technical capabilities; instead, the coordination of decisions and actions among interacting individual agents with diffuse private knowledge and plans forms the basis of economic activity. The diffuse and private nature of knowledge hampers such plan coordination, but out of human interaction, institutions emerge that enable decentralized coordination. Prices and market processes compose an institution for coordination in the face of the knowledge problem. Moreover, Hayek argued, knowledge transcends “scientific” information, there is no given and uniform set or distribution of data, and such information fails to capture all knowledge relevant to both static and dynamic decision-making and coordination.

Hayek’s substantial insight in this work, one that has become largely incorporated into mainstream economics, is that the price system operating through market processes is an effective, parsimonious (but not perfect) means of generating, signaling, and aggregating such knowledge. Prices cannot convey all individual knowledge pertinent to a particular economic decision, but they do serve as knowledge surrogates by communicating some private knowledge (Horwitz 2004). Coordination of individual actions and plans emerges as a beneficial consequence of the price system; thus the price system and market processes enable emergent, or unplanned, order.

Hayek characterized the fundamental economic problem not as the static allocation of scarce resources among uses by omniscient agents, but rather as the coordination of actions and plans among dispersed agents with diffuse private knowledge. In his statement that “… the knowledge of the circumstances of which we must make use never exists in concentrated or integrated form, but solely as the dispersed bits of incomplete and frequently contradictory knowledge which all the separate individuals possess” (p. 519), Hayek draws on the earlier arguments of the socialist calculation debate and of his (1937) work. The “man on the spot” (p. 524) has subjective, private knowledge of “the particular circumstances of time and place” (p. 522), and that knowledge is among the decision-relevant data that cannot be aggregated except through a decentralized system of prices and a market process of exchange to determine those prices. Prices economize on the communication and interpretation of knowledge among dispersed agents.

How do individuals learn the plans of others? How do we learn when we are wrong and take action accordingly? Prices and market processes provide feedback channels. Feedback loops, learning, adaptation to a changing environment and changing actions and plans of others, interdependence of agents and their actions in a complex system, and how prices and markets serve as feedback loops making a complex system adaptive – all are important implications of Hayek’s argument. Prices provide profit opportunities and realized profits, and those realized profits serve as feedback that can spur the discovery of new products, services, business models, or other ways to create value through economic activity. Alert entrepreneurs see these opportunities, learn from observed and realized feedback, and adapt their plans accordingly. Prices enable “error detection and correction within the market” (Boettke 1998, p. 135). Markets are processes for social learning and provide feedback channels for entrepreneurial alertness.

This post is drawn from my article, “Knowledge Problem” (SSRN link), which is included in the forthcoming Oxford Handbook of Austrian Economics (Peter Boettke and Chris Coyne, eds.).

New paper: Knowledge Problem

Lynne Kiesling

I have a new paper that may be of interest to KP readers, since the subject of the paper is the same as the name of this site: Knowledge Problem. I am honored to have been invited to contribute this paper to the forthcoming Oxford Encyclopedia of Austrian Economics (Peter Boettke and Chris Coyne, eds.). Here’s the abstract:

Hayek’s (1945) elaboration of the difficulty of aggregating diffuse private knowledge is the best-known articulation of the knowledge problem, and is an example of the difficulty of coordinating individual plans and choices in the ubiquitous and unavoidable presence of dispersed, private, subjective knowledge; prices communicate some of this private knowledge and thus serve as knowledge surrogates. The knowledge problem has a deep provenance in economics and epistemology. Subsequent scholars have also developed the knowledge problem in various directions, and have applied it to areas such as robust political economy. In fact, the knowledge problem is a deep epistemological challenge, one with which several scholars in the Austrian tradition have grappled. This essay analyzes the development of the knowledge problem in its two main categories: the complexity knowledge problem (coordination in the face of diffuse private knowledge) and the contextual knowledge problem (some knowledge relevant to such coordination does not exist outside of the market context). It also provides an overview of the development of the knowledge problem as a concept that has both complexity and epistemic dimensions, the knowledge problemʼs relation to and differences from modern game theory and mechanism design, and its implications for institutional design and robust political economy.

In this paper I analyze the development of the two categories of the knowledge problem — the complexity knowledge problem and the contextual knowledge problem — and explore both the history of the development of these concepts and their application in robust political economy and new institutional economics. As is the hallmark of a good research project, I think on balance I learned more than I created in the process of writing this paper.

One other thing I made sure to include was a discussion of how the knowledge problem and its development relates to game theory and mechanism design, through the work of Oskar Morgenstern (and then through some of the work of Herb Simon and Vernon Smith, among others).

Tying together economics, institutional design, history of thought, and epistemology, I hope you find this paper informative and useful! I’ll also make sure to update when the full volume is available.

An example of what not to do in persuasion

Lynne Kiesling

Alex Tabarrok has an excellent post this morning at Marginal Revolution:

David Warsh and Paul Krugman try to write Hayek out of the history of macroeconomics. …

It is true that many of Hayek’s specific ideas about business cycles vanished from the mainstream discussion under the Keynesian juggernaut but what Krugman and Warsh miss is that Hayek’s vision of how to think about macroeconomics came back with a vengeance in the 1970s. …

… Hayek was an important inspiration in the modern program to build macroeconomics on microfoundations. The major connecting figure here is Lucas who cites Hayek in some of his key pieces and who long considered himself a kind of Austrian.

I offer this as a cautionary “what not to do” note to students in particular, but also to all of us. In the piece to which Alex is responding Krugman chooses his definition of “modern macroeconomics” in a way that clearly maps into his preconceptions and reflects his confirmation bias. Such a rhetorical stratagem is unscientific and anti-intellectual. It’s also easy to critique (no disrespect intended for Alex’s good, pointed critique) by simply looking at the literature and seeing that modern macro encompasses a breadth of ideas and approaches, many of which are substantially informed by models and methodological approaches that Krugman chooses to reject.

Thus both on intellectual grounds and with a view toward crafting an argument that is persuasive to those who don’t already agree with you and share your worldview, don’t do this. Being more ecumenical and treating the contributions of your intellectual opponents with respect will make your arguments more thorough, effective, and persuasive.

On a substantive note, I’d like to echo the recommendation that Jacob Levy made in the comments on Alex’s post; the conclusion of Warsh’s essay is a good one, and suggests that incorporating more of a complexity approach into macro would enable us to build better models:

That said, it is pleasing to think that Hayek himself may yet turn out to have been a very great economist after all, far more significant than Myrdal or Robinson, when seen against the background of a broader canvas. The proposition that markets are fundamentally evolutionary mechanisms runs through Hayek’s work. Caldwell, of Duke University, notes that, starting with the Constitution of Liberty, “the twin ideas of evolution and spontaneous order” become prominent, especially the idea of cultural evolution, with its emphasis on rules, norms, and decentralization.

These are today lively concepts in laboratories and universities around the world. “It could have been that Hayek was running a different race, and the fact that he didn’t do well in the Walrasian race was that he wasn’t running in it—he was running in the complexity race,” says David Colander, of Middlebury College. Hayek may yet enter history as a prophet of evolutionary economics, a discipline dreamt of since the days of Thorstein Veblen and Alfred Marshall in the late nineteenth century but not yet forged, whose great days lie ahead.

UPDATE: See also Pete Boettke on this same theme, motivated by Alex’s post.

Penn Jillette and Hayek: “I don’t know”

Lynne Kiesling

Penn Jillette, the taller and more vocal half of the magic performance duo Penn & Teller, has written a lovely and thoughtful essay as a companion to his appearance last night on Piers Morgan’s CNN show. It defies excerpting, so I encourage you to click through and read it in its entirety.

His theme: “I don’t know”, particularly with respect to religion and to helping the poor, leading him to conclude that he is an atheist libertarian. For example, about helping the poor he writes

Then he asked me what we could do to help poor people. I said I donated money, food, medical care, and services and he said, “No,” he meant, what could society do to solve the problem of poor people. Again, I was stumped.

He said the government had to do it, which I interpreted as another way of saying he didn’t know, but he thought that made me look mean … even though I do care and do try to help. …

And I don’t think anyone really knows how to help everyone. I don’t even know what’s best for me. Take my uncertainty about what’s best for me and multiply that by every combination of the over 300 million people in the United States and I have no idea what the government should do.

In this essay Jillette is channeling some of the most important ideas about the knowledge problem developed by Hayek and others, ideas that are the foundation of what we do here at KP — the limits of individual knowledge, the necessary limits of collective knowledge, and the humility that should arise as a consequence, both individually and in collective action/policy situations. Jillette is also channeling a lot of David Hume’s skepticism, not just in terms of religion but also in terms of empiricism and the limits of human reason.

Jillette’s essay is also charming in tone, reflecting respect for those who disagree with him and those with different life experiences and abilities. A very thought-provoking read.

Quote of the day: Hayek on expediency and freedom

Lynne Kiesling

Pace Don Boudreaux for my shamelessly copying his “quote of the day” meme … my quote of the day is this striking one from Hayek’s essay “Principles or Expediency” (1971):

From the insight that the benefits of civilization rest on the use of more knowledge than can be used in any deliberately concerted effort, it follows that it is not in our power to build a desirable society by simply putting together the particular elements that by themselves appear desirable. Though probably all beneficial improvements must be piecemeal, if the separate steps are not guided by a body of coherent principles, the outcome is likely to be a suppression of individual freedom.

The reason for this is very simple though not generally understood. Since the value of freedom rests on the opportunities it provides for unforeseen and unpredictable actions, we will rarely know what we lose through a particular restriction of freedom. Any such restriction, any coercion other than the enforcement of general rules, will aim at the achievement of some foreseeable particular result, but what is prevented by it will usually not be known. The indirect effects of any interference with the market order will be near and clearly visible in most cases, while the more indirect and remote effects will mostly be unknown and will therefore be disregarded. We shall never be aware of all the costs of achieving particular results by such interference.

And so, when we decide each issue solely on what appears to be its individual merits, we always overestimate the advantages of central direction. Our choice will regularly appear to be one between a certain known and tangible gain and the mere probability of the prevention of some unknown beneficial action by unknown persons.

Certainly captures some themes we’ve been exploring around here lately.