Imagine that a hypothetical US president is considering his options vis-à-vis Iran’s rapidly developing nuclear weapons program. First a science adviser enters the room and predicts that if the Iranians take a certain quantity of fissile material and compress it into a sphere of a particular size under specific conditions, then it will cause an explosion large enough to destroy a major city. Next a historian enters the room and predicts that if external attempts are made to thwart Iranian nuclear ambitions, then a popular uprising will ensue sooner or later, and force changes in governments until Iran has achieved nuclear capability. The president would be incredibly irresponsible to begin debating nuclear physics with his science adviser, even if the president happened to have trained as a physicist. Conversely, the president would be incredibly irresponsible not to begin a debate with the historian. This likely would include having several historians present different perspectives,Read more at location 1577
Superficially she might sound a lot more like the physicist. She would use lots of empirical data, equations, and technical language.Read more at location 1588
But lots of things would arguably remain outside the grasp of formal models.Read more at location 1590
(1) a priori beliefs about human nature, and conclusions that are believed to be logically derivable from them, (2) analysis of historical data, which is to say, data-driven theory-building, and (3) a review of the track record of prior predictions made using the predictive rule in question.Read more at location 1595
Please view the following film taken from a long series of huge explosions that result when independent evaluators combine the materials I described in the manner I described.Read more at location 1598
The reason the physicist need concentrate only on controlled experiments is that these are accepted as the scientific gold standard for testing theories.Read more at location 1602
so many things change in a macroeconomic event that it is not realistic to isolate the causal impact of any one factor.Read more at location 1605
Mankiw summarized fourteen findings that have achieved widespread acceptance among economists. Among them are: Fiscal policy (e.g., tax cut and/or government expenditure increase) has a significant stimulative impact on a less than fully employed economy. A large federal budget deficit has an adverse effect on the economy. A minimum wage increases unemployment among young and unskilled workers. In fact, 10 to 20 percent of practicing economists disagree with each of these assertions; but more fundamentally, even if we assume them to be correct, they are too vague to really help settle policy arguments.Read more at location 1617
These limitations are inherent to the methodology of nonexperimental social science.Read more at location 1636
although experiments can help and should be aggressively pursued, our scientific knowledge of any human social organization will remain extremely limited even when these experiments are deployed extensively.Read more at location 1644
The Development of Nonexperimental Quantitative Social ScienceRead more at location 1653
Prior to the creation of modern social science, we simply had history, with its tradition of recording facts and making assertions based on these facts plus narrative appeals to commonly held understandings of human motivations and experiences. This was nonscientific, in that it did not make claims for the kinds of reliable, nonobvious, and useful predictive rules that characterize science. In the terms of this book, history is informed common sense.Read more at location 1663
numerous thinkers attempted to apply scientific methods to the study of human social behavior. The French Enlightenment, in particular, was central to the creation of the modern social science ideology. Auguste Comte and Henri de Saint-Simon were explicit in arguing that the methods of natural science provided the model for developing predictive laws for human society. Comte argued that human understanding in various fields proceeded in three stages: theological, metaphysical, and finally, positive.Read more at location 1667
we would say that knowledge proceeds from mythology to philosophy to science.Read more at location 1671
Comte believed that humanity had achieved “positive” (i.e., scientific) understanding in various fields in the order of their complexity: mathematics, astronomy, physics, chemistry, biology, and finally sociology.Read more at location 1672
It was clear to the earliest social scientists that the natural sciences of their era (astronomy, chemistry, and physics) achieved spectacular success by discovering and stating physical laws as equationsRead more at location 1673
He argued that despite the inability to conduct controlled experiments in social sciences, thinkers could reason from introspection to general predictive rules—sayingRead more at location 1679
In an argument somewhat akin to Sir Karl Popper’s doctrine of falsification, Mill argued that one role of empirical observation is to “verify” a given causal rule by “by comparing,Read more at location 1681
For example, a social scientist might promulgate a predictive rule that a US president will fail to win reelection if the unemployment rate exceeds 10 percent. If, in a specific future election, a president were to win reelection with 11 percent unemployment, the social scientist might observe that there was a disturbing cause created by the fact that the nation was at war and the president was viewed as an indispensable leader.Read more at location 1685
But what if there are myriad “disturbing causes,” many of which are as important as the cause of interest in determining the outcome of the situation?Read more at location 1688
The problem of how we can develop nonobvious, reliable predictive rules without controlled experiments has so far been deadly to Comte and Mill’s dream of rational social science.Read more at location 1696
First, both asserted findings are nonobvious, but plausible.Read more at location 1700
Second, each of these should be a very high-quality analysis,Read more at location 1705
The first example is a regression model presented by Princeton public policy professor Larry M. Bartels in his 2008 book, Unequal Democracy.Read more at location 1711
Kammerer Award from the American Political Science Association (APSA)Read more at location 1713
The citation for the award specifically calls out “the care taken in the analysis” and “the rigorous application of controls.”Read more at location 1714
The most widely discussed finding in the book was a regression analysis, based in part on an updated version of analysis from his 2004 academic paper, “Partisan Politics and the US Income Distribution,” which reviews the changes in incomes for the rich versus the poor under Democratic versus Republican presidents from 1948 to 2005. Bartels asserts that the differences in the behavior of Republican versus Democratic presidents have been a leading cause of the rich gaining relative income versus the poor, saying these presidential differences were “the most important single influence on the changing US income distribution over the past half-century.”Read more at location 1715
decisions by the Congress, Supreme Court, and Federal Reserve; changes in international economic competition; technological developments that enhance some people over others; changes in immigration rates and sources; changes in social mores and beliefs; and being at war or peace.Read more at location 1721
“The price of oil and the increasing participation of women in the labor forceRead more at location 1724
because these long-term trends have been so glacial, and so intertwined, it is very difficult to discern their distinct effects on the shape of the income distribution.”Read more at location 1727
“Fortunately, from the standpoint of political analysis, the very fact that these social and economic trends have been gradual and fairly steady implies that their effects are unlikely to be confounded with the effects of alterations in control of the White House.”Read more at location 1729
The obvious question is how we know the net causal effect of a great many intertwined social and economic forces has been gradual, fairly steady, and not confounded.Read more at location 1730
Based on these assertions, Bartels builds a set of regression models that attempts to explain changes in income inequality in any given year as a function of six variables.Read more at location 1732
One of these variables is last year’s party of the president.Read more at location 1734
last year’s change in the price of oil, and last year’s change in female labor force participation.Read more at location 1736
The fourth variable is last year’s income growth at the 95th percentile (i.e., growth “trickles down”).Read more at location 1737
Finally, he understands that this short list of four factors cannot conceivably describe all of the “great many economic and social forces”Read more at location 1739
he adds two variables to his equations to fit his curves to the historical trend of the data, rather than to explain this historical trend as a function of underlying causes. One of these trend variables is the number of years since 1948 for each year, and the final variable in the model is the square of the number of years since 1948 for each year. And that’s it.Read more at location 1742
Bartels is claiming he has isolated the effect of presidential party on inequality with those six variables.Read more at location 1745
assumption that presidential actions affect income distributions for only one yearRead more at location 1747
There are plausible long-term causal mechanisms that could have almost no effect for years,Read more at location 1748
obviously including appointments to the Supreme Court and Federal Reserve,Read more at location 1749
Did Reagan’s effects on the change in distribution of incomes in America really end in 1989?Read more at location 1750
assumption that out of all the potential confounding causes for inequality, only oil prices and female labor participation should be included in the model as specific causes, and that the model has captured all of the other possible causal factors through his “linear and quadratic trend terms.”Read more at location 1751
Using the raw Census data tables, I observed that income inequality does tend to rise under Republican presidents (lagged one year—e.g., Jimmy Carter gets credit for 1981) and fall under Democratic presidents (lagged one year). But when I did the simple test of changing the lag to two years, the entire apparent effect disappears:Read more at location 1760
he cites two academic papers that he believes show his assumption is “consistent with macroeconomic evidence regarding the timing of economic responses to monetary and fiscal policy changes.” But first note that a president can affect a far broader range of policies than monetary and fiscal policy—for example, regulatory decisions, Supreme Court and Federal Reserve appointments, negotiating trade treaties, antitrust enforcement, seeking out or settling wars,Read more at location 1766
And these papers don’t appear to claim a one-year rather than a two-year window for the impacts they do analyze. One paper estimates that (1) the peak impact of a tax shock on GDP should be reached by one to two years after the taxes change, and thereafter continue indefinitely; and (2) the peak impact of a spending shock should not be reached until two to four years after the spending change, and then continue indefinitely. The other paper estimates that numerous effects of monetary shocks extend for two years or more.Read more at location 1772
For this kind of a social reality, such model-tuning (for example, the one-year lag versus a two-year lag; including oil prices and female labor force participation versus the myriad other potential control variables; using a linear plus quadratic trend terms versus searching for additional explicit control variables, etc.) is inevitable, because the complexity of the real world overwhelms the tool of regression analysis.Read more at location 1780
Steven Levitt, a distinguished economics professor at the University of Chicago. Levitt was awarded the John Bates Clark Medal as the best American economist under forty.Read more at location 1788
Among the most widely discussed passages in Freakonomics was Levitt’s assertion that a significant fraction of US crime reduction in the 1990s can be linked to changes set in motion by Roe v. Wade in 1973. The basic asserted causal mechanism is that the increase in abortions disproportionately eliminated potential future criminals.Read more at location 1789
several fertility control technologies—most importantly the birth control pill—plus a huge variety of social trends that plausibly affect abortion rates and/or crime emerged in the same era as legalized abortion. The argument Levitt makes in his professional publications is that we can control for these other effects. But this is extremely difficult if these other effects became evident at the times, in the places, and for the population subgroups where abortion legalization had its first effects.Read more at location 1798
Freakonomics presents the results of a natural experiment: the five states that liberalized abortion laws prior to Roe (Levitt terms these “early legalizers”) experienced a crime reduction prior to the nonrepeal states.Read more at location 1803
showing that the states with the highest abortion rates in the 1970s experienced the greatest crime dropsRead more at location 1805
Finally, they note that Australia and Canada have seen similar results.Read more at location 1807
The logic of the various regression analyses in the paper was (as with the Bartels analysis) to “hold all other factors equal” and isolate the causal change created by the change in the abortion rate.Read more at location 1812
other academics published alternative versions of the same analysis, using slightly different assumptions, that did not show any such effect. Levitt and Donohue, of course, quickly replied by arguing that one should use their preferred specifications.Read more at location 1815
two Federal Reserve economists published a crucial criticism in which they showed that the software implementation of the equations presented in DL 2001 had an important error and that once this was corrected and some other technical changes were made, the asserted effect of abortion on crime was no longer evident.Read more at location 1817
using a different data set massaged differently to reflect better how people moved among various states after having abortions—inRead more at location 1821
Other academics then attempted to replicate the same analysis for the effect of legalization of abortion in the United Kingdom. They also discovered that depending on the exact specification of data sets and assumptions made in the regression model, the effect on crime would sometimes appear, and sometimes not.Read more at location 1826
The fragility of the results in this paper serve to emphasize the difficulty researchers have in identifying causal effects of social change such as abortion legalization on crime rates some years hence, particularly given the myriad of other social changes occurring over the same time and which may dilute any effect.Read more at location 1830
Once again, regression analysis cannot tell us the effects of abortion on crime, because different reasonable assumptions for the analysis lead to completely different answers.Read more at location 1833
One way to get around all of this confusion would be to run an experiment. A purposeful experiment to force a random sample of states to implement abortion legalization has never happened in American history, and almost certainly never will.Read more at location 1837
New York declines 35 percent, while Alaska increases 50 percent; California is down 14 percent, and Hawaii is up 11 percent; Washington is almost exactly flat. The total rate across the early legalizers goes down versus the rest of the country only because New York and California are so much larger than the other three states. The natural experiment cannot resolve the question of the causal impact of abortion on crime, either.Read more at location 1857
First, causal density is very high, so sample size is critical, but many natural experiments have far too few data points.Read more at location 1861
Second, a national society is holistically integrated; therefore, it is hard to get causal impermeability between the test and control groups. In the abortion-crime debate, for example, I indicated that a significant technical issue was how to account for the reality that people move between states.Read more at location 1866
Third is the possibility of systematic, unobserved bias between the individuals or places that are subject to the treatment in the natural experiment as compared to those that are not. Consider the abortion-crime example. All kinds of plausible differences in political culture, social evolution, rational expectation for future challenges, and so on could vary between the early legalization states and the rest of the country.Read more at location 1873
This is the irreducible problem for any such social natural experiment that does not use strict randomization for assignment to the test population, no matter how large the same size.Read more at location 1880
three problems: small sample size, integrated complexity, and potential selection bias.Read more at location 1882
The actual event that inspired this observation was that, one day in 1961, Lorenz entered .506 instead of .506127 for one parameter in a climate-forecasting model and discovered that it produced a wildly different long-term weather forecast.Read more at location 1887
Businesses are notoriously practical and results-oriented, and have sunk vast resources into trying to develop useful, reliable predictions for behavior in the absence of experiments. In doing so, they have run into the same problems and hit the same dead ends. I know, because I spent years doing it.Read more at location 1893
