La “Buona Scuola” come missione impossibile
Null Hypothesis in education by Arnold Kling
***
Tesi: tra insegnamento e apprendimento non esiste una relazione chiara e quantificabile. E’ questo che dice la “null hypothesis. La portata di una simile ipotesi è vasta, per esempio: le risorse investite nell’educazione sono uno spreco. Oppure: la valutazione degli insegnanti è velleitaria. Eccetera.
***
In 2010, Teach For America (TFA) launched a major expansion effort, funded in part by a five-year Investing in Innovation (i3) scale-up grant of $ 50 million from the U.S. Department of Education. This study examines the effectiveness of TFA elementary school teachers in the second year of the scaleup, relative to other teachers in the same grades and schools. The study found that, on average, TFA corps members hired in the first two years of the scale-up period were as effective as other teachers in the same high-poverty schools in both reading and math.
MELISSA CLARK E GLI INTERVENTI SUGLI INSEGNANTI
We use administrative data on Swedish lottery players to estimate the causal impact of wealth on players’… The effects on most other child outcomes, which include drug consumption, scholastic performance, and skills, can usually be bounded to a tight interval around zero. Overall, our findings suggest that correlations observed in affluent, developed countries between (i) wealth and health or (ii) parental income and children’s outcomes do not reflect a causal effect of wealth…
DAVID CESARINI: RICCHEZZA E ISTRUZIONE
In 1960, countries with an education level of 8.3 years of schooling were 5.5 times richer than those with 2.8 year of schooling. By contrast, countries that had increased their education from 2.8 years of schooling in 1960 to 8.3 years of schooling in 2010 were only 167% richer. Moreover, much of this increase cannot possibly be attributed to education, as workers in 2010 had the advantage of technologies that were 50 years more advanced than those in 1960. Clearly, something other than education is needed to generate prosperity….
RICARDO HAUSMAN SU RICCHEZZA E ISTRUZIONE
A new study of 10,000 teachers found that professional development— the teacher workshops and training that cost taxpayers billions of dollars each year— is largely a waste.
LINDSEY LAYTON. INVESTIRE SUGLI INSEGNANTI È UNO SPRECO
The Null Hypothesis is this: 1. Take any educational intervention 2. Conduct a rigorous controlled experiment. 3. Look for results that do not fade out within a year or two. 4. If you find apparent success, try to replicate it 5. You will not find significant effects in all of steps (1)– (4)
DEFINIZIONE DI NULL HYPOTHESIS
For teacher training to work, you have to deal with the null hypothesis at two layers. First, you have to find an intervention that significantly affects teachers. Then that intervention has to significantly effect students. This is the null hypothesis compounded.
ESEMPIO DI NH COMPOSTA
The Chicago public-school system implemented a “whole-school” intervention called the Teacher Advancement Program (TAP), which was supposed to raise the quality of its teacher workforce. Teachers were given the opportunity to earn performance pay and promotions through a special system of mentoring and observation. Although the literature on TAP is extensive, IES focused on the most reliable study— an experimental evaluation conducted by Mathematica a few years ago. There are questions about how faithfully TAP was implemented in Chicago, and the control group was small, but here is the study’s bottom line: There were no significant gains in student test scores, regardless of grade level or subject area.
JASON RICHWINE
Andrew Flowers reports on the Chetty, et al study showing differences in teacher value added, Their numbers are being replicated in many different settings. Even in Rothstein’s paper critiquing their method, he replicated their results using data from North Carolina public schools. “I’m not aware of another area of social science where there has been so much replication, in such a short time, and they’ve all found the same result,” Kane said.
CONTRO LA NH… CHETTY
As is evident, pre-K and control children started the pre-K year at virtually identical levels. The TNVPK children were substantially ahead of the control group children at the end of the pre-K year (age 5 in the graph). By the end of kindergarten (age 6 in the graph), the control children had caught up to the TNVPK children, and there were no longer significant differences between them on any achievement measures.
DALE FARRAN SULLA PRESCUOLA
This will create some cognitive dissonance for progressives who have faith in universal pre-K and also believe in using rigorous social science to guide policy. And some cognitive dissonance for James Heckman.
PROBLEMI PER HECKMAN
Although family disadvantage is strongly correlated with schools and neighborhood quality, the SES gradient in the sibling gender gap is almost as large within schools and neighborhoods as it is between them.
DAVID AUTOR SULL’AMBIENTE SCOLASTICO
I view the quoted sentence as throwing some cold water on Raj Chetty’s view that neighborhoods make a big difference.
AUTOR VS CHETTY
In summary: teacher quality probably explains 10% of the variation in same-year test scores. A + 1 SD better teacher might cause a + 0.1 SD year-on-year improvement in test scores. This decays quickly with time and is probably disappears entirely after four or five years, though there may also be small lingering effects. It’s hard to rule out the possibility that other factors, like endogenous sorting of students, or students’ genetic potential, contributes to this as an artifact, and most people agree that these sorts of scores combine some signal with a lot of noise… This seemingly unbelievable finding has been replicated in quasi-experiments and even in real experiments and is difficult to banish. Since it does not happen through standardized test scores, the most likely explanation is that it involves non-cognitive factors like behavior….
SCOTT ALEXANDER SU QUANTO PESA LA QUALITÀ DEGLI INSEGNANTI
I think that the right way to put it is that no one should pretend to know very much or have great confidence about claims about how teaching affects outcomes, particularly in the long term.
LA MISTERIOSA RELAZIONE INSEGNAMENTO APPRENDIMENO
In other service businesses, we let the customer make a subjective judgment of quality. You do not pick your hair stylist or your doctor or your auto mechanic based on some distant economist’s regression analysis. The reason we don’t use customer judgment in education is that we let government, rather than the customer, pick the service provider.
NH E SOCIALIZAZIONE DEL SERVIZIO
charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while regular charter schools decrease test scores, four-year college enrollment, and earnings. Using school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings.
WILL DOBBIE SULLE CHARTER SCHOOL
My guess is that their findings are random variations around the null hypothesis.
COME VALUTARE I STRANI RISULTATI DI DOBBIE
Studies that show significant effects of educational interventions are right in this wheelhouse. That is why until they are scaled, replicated, and shown to have durable effects, you should view accounts of such studies with skepticism.
COME VALUTARE REALMENTE UN PROGRAMMA
The results demonstrate that expenditures and related school inputs have very weak associations not only with test scores in the sophomore and senior years of high school but also with high school graduation and subsequent college entry… The overall conclusion of the Coleman Report— that family background is far and away the most important determinant of educational achievement and attainment— is as convincing today as it was fifty years ago….
STEPHEN MORGAN… FAMIGLIA E SCUOLA
Why do the academic effects of early childhood education so often fade out?
THIMOTY TAYLOR
“The gains to children in Massachusetts charters are enormous. They are larger than any I have seen in my career,” [education researcher Susan] Dynarski wrote.
SUSAN DYNARSKI… SU RICERCHE A BREVE
Zimmerman (2014) compares students whose GPAs are either just above or just below the threshold for admission to Florida International University, a four-year school with the lowest admissions standards in the Florida State University System. This study finds that “marginal students” who are admitted to the school experience sizable earnings gains over those who just miss the cutoff and are thus unlikely to attend any four-year college, translating into meaningful returns net of costs and especially high returns for low-income students.
ZIMMERMAN SUL MERO COLLEGE PREMIUM
One of the Obama administration’s signature efforts in education, which pumped billions of federal dollars into overhauling the nation’s worst schools, failed to produce meaningful results, according to a federal analysis. Test scores, graduation rates and college enrollment were no different in schools that received money through the School Improvement Grants program— the largest federal investment ever targeted to failing schools— than in schools that did not.
EMMA BROWN SUI FALLIMENTI DI OBAMA NELLA SCUOLA
At some of the most prestigious flagship universities, test results indicate the average graduate shows little or no improvement in critical thinking over four years. Sounds like another win for the null hypothesis.
WSJ
Neerav Kingsland writes, Overall, CMOs are delivering +. 03 SD effects over three years in both reading and math. These gains are driven by the fact that students benefit from CMOs the longer they stay in them In this context, CMOs are charter school management organizations, not collateralized mortgage obligations. I still think that the overall picture tends to support the Null Hypothesis, although I believe that charter schools are capable of saving a lot of money while producing the same (null) effect as government schools.
CHARTER: NH + RISPARMI
I would speculate that our school system probably makes more efficient use of resources than would a system in which schools did not exist. I would speculate that it makes less efficient use of resources than would a system of vouchers and competition rather than government-managed schools. Would I go far as to say that the only difference that schools make is in resource allocation, not in outcomes? I doubt that such an extreme position is warranted. But statistically, educational interventions tend to affect resource allocation much more than outcomes. For educational interventions within roughly the current institutional setting, the null hypothesis is not an iron law, but it is an empirical regularity.
REFRAME NH
