Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society
You have 75 highlighted passages
You have 71 notes
Last annotated on July 7, 2016
I developed a complicated analytical process to predict the size of the sales gain, including qualitative and quantitative consumer research, competitive benchmarking, and internal capability modeling. With great pride I described this plan to a partner in our consulting firm, who responded by saying, “Okay . . . but why wouldn’t you just do it to a few stores and see how it works?” This seemed so simple that I thought it couldn’t be right.Read more at location 77
an experiment really would provide the most definitive available answer to the question.Read more at location 81
Cost changes often could be predicted reliably through engineering studies. But when it came to predicting how people would respond to interventions, I discovered that I could almost always use historical data, surveys, and other information to build competing analyses that would “prove” that almost any realistically proposed business program would succeed or fail, just by making tiny adjustments to analytical assumptions.Read more at location 84
Even after executing some business program, debates about how much it really changed profit often would continue, because so many other things changed at the same time. Only controlled experiments could cut through the complexityRead more at location 87
As an example, consider the deliberations around how to respond to the 2008 economic crisis.Read more at location 91
What would be the effects of any given stimulus proposal on general economic welfare?Read more at location 95
Paul Krugman and Joseph Stiglitz, both Nobel laureates in economics, argued that stimulus would improve economic performance. In fact, they both argued that it should be bigger. On the other hand, James Buchanan, Edward Prescott, and Vernon Smith—all Nobel laureates in economics—argued that the stimulus would not improve economic performance enough to justify the investment, saying that “notwithstanding reports that all economists are now Keynesians . . . it is a triumph of hope over experience to believe that more government spending will help the US today.”Read more at location 98
But the stimulus situation was even worse. It was clear at the time that we would not know which of them were right or wrong even after the fact.Read more at location 107
Suppose Professor Famous Economist X predicted on February 1, 2009, that “unemployment will be about 10 percent in two years without the bill, and about 8 percent with the bill.” What do you think would happen when 2011 rolled around and unemployment was 10 percent? It’s a very, very safe bet that Professor X would say something like, “Yes, but other conditions deteriorated faster than anticipated, so if we hadn’t passed the stimulus bill, unemployment would have been more like 12 percent.Read more at location 108
The key problem is that we have no reliable way to measure the counterfactual—that is, to know what would have happened had we not executed the policy—because so many other factors influence the outcome.Read more at location 112
social sciences have not produced a substantial body of useful, nonobvious, and reliable rules that would allow us to predict the effectRead more at location 116
it should lead us to value the freedom to experiment and discover workable arrangements through an open-ended process of trial and error.Read more at location 118
This is not a new insight, but is the central theme of an Anglo-American tradition of liberty that runs from Locke and Milton through Adam Smith and on to the twentieth-century libertarian thinkers, pre-eminently Sir Karl Popper and F. A. Hayek.Read more at location 119
human society is far more complex than the understanding of the planners. Hayek termed this the “knowledge problem.” By this line of thinking, we need the trial-and-error process created by the free play of markets, social tolerance, and experiments in living—what Popper called the “open society”—toRead more at location 128
In short, we need freedom because we are ignorant.Read more at location 130
They were not (nor were Smith and some of their antecedents) arguing against all market regulations, government investments, lifestyle restrictions, and so forth. Rather, they were arguing against an unwarranted assumption of knowledge by those who would attempt to control society’s evolution.Read more at location 132
But as I’ll describe in detail, the mechanics of genetic evolution provide a clear and compelling picture of how a system can capture and exploit implicit insight without creating explicit knowledge, and this naturally becomes the model for the mechanism by which trial and error advances society’s material interests without conscious knowledge or planning.Read more at location 139
Donald T. Campbell, a twentieth-century social scientist at Northwestern University, to create a theory of knowledge, which he termed “evolutionary epistemology.” It has a practical implication that can be summarized as the idea that any complex system, such as our society, evolves beliefs about what practices work by layering one kind of trial-and-error learning upon another.Read more at location 145
real people try out almost random practices, and those that work better are more likely to be retained.Read more at location 148
This is a modernized and practical version of what Popper called “piecemeal social engineering”: the idea of testing targeted reforms designed to meet immediate challenges, rather than reforming society by working backward from a vision of the ideal.Read more at location 150
This is a much humbler view of social science than what was entertained by the eighteenth-century founders of the discipline, such as Auguste Comte and Henri de Saint-Simon,Read more at location 152
These early pioneers expected that social science eventually would resemble Newtonian physics,Read more at location 153
But this same increasing complexity has another pernicious effect: it becomes far harder to generalize the results of experiments. We can run a clinical trial in Norfolk, Virginia, and conclude with tolerable reliability that “Vaccine X prevents disease Y.” We can’t conclude that if literacy program X works in Norfolk, then it will work everywhere.Read more at location 159
This is because the vast majority of reasonable-sounding interventions will work under at least some conditions, and not under others.Read more at location 167
A brute-force approach to this problem would be to run not one experiment to evaluate whether this program works, but to run hundreds or thousands of experiments to evaluate the conditions under which it works.Read more at location 170
This is the opposite of elegant theory-building, and is even more limited than either Popper’s or Campbell’s version of social engineering.Read more at location 172
Of course, this would require that each experiment be cheap enough to make this many tests feasible.Read more at location 174
The capability has emerged not within formal social science, but in commercial enterprises.Read more at location 175
The method has been to use information technology to routinize, and ultimately automate, many aspects of testing.Read more at location 178
I found myself in the middle of this experimental revolution in business when some friends and I started what eventually became a global software company that produces the tools to apply randomized experiments in certain narrowly defined business contexts.Read more at location 182
there is no magic. I mean this in a couple of senses. First, we are unlikely to discover some social intervention that is the moral equivalent of polio vaccine.Read more at location 188
experimental science in these fields creates only marginal improvements.Read more at location 189
we will never eliminate the need for strategy and some kind of long-term vision.Read more at location 192
1. Nonexperimental social science currently is not capable of making useful, reliable, and nonobvious predictionsRead more at location 198
2. Social science very likely can improve its practical utility by conducting many more experiments,Read more at location 199
3. Even with such improvement, it will not be able to adjudicate most important policy debates.Read more at location 201
4. Recognition of this uncertainty calls for a heavy reliance on unstructured trial-and-error progress.Read more at location 202
5. The limits to the use of trial and error are established predominantly by the need for strategy and long-term vision.Read more at location 203
trying to do something as comparatively trivial as figuring out how many Snickers bars ought to be on a convenience store shelf next week.Read more at location 208
