Tanto per essere chiari, questo lavoro mostra che gli stessi metodi con i quali si è preteso di quantificare la creazione di valore da parte degli insegnanti, spiegano anche l' effetto degli insegnanti medesimi sull'altezza dei loro studenti. Poiché tale effetto nella realtà è presumibilmente pari a zero, è lecito dubitare anche del primo calcolo.primo calcolo. https://www.nber.org/papers/w26480.pdf

https://feedly.com/i/entry/Nkn6RK6HwBgWrvMj84SxHg63I5Wn8O87ZvoPCQT30Mw=_16ec66c169a:2179ba0:97b0c8f7

I BUONI INSEGNANTI ESISTONO?

Riccardo Mariani

Adesso · 

I BUONI INSEGNANTI ESISTONO?

Com’è fatto un buon insegnante? Conta l’esperienza, i titoli, le certificazioni, il genere? Difficile dirlo, nemmeno sappiamo se esistono i buoni insegnanti. Sia chiaro, per tutti noi esistono, ma quando poi si tenta di “prenderne le misure” spariscono. Quantificare l’opera di un buon insegnate è arduo, persino se semplifichi al massimo assumendo che la scuola abbia come unico fine il superamento di certi test formali. In teoria il buon insegnate è quello che ti migliora, che ti fa passare da un punteggio ad uno più elevato. Ma come puoi capire se già non eri destinato a quel miglioramento? In fondo le classi di allievi non si compongono casualmente, c’è una certa omogeneità, ci sono le scuole dei quartieri bene e quelle degli altri quartieri, ci sono le scuole di ottima fama e di fama pessima. Cio’ significa che attribuire meriti e colpe agli insegnanti sulla base del miglioramento nei punteggi dei test rischia di essere ingiustificato, quanto più gli studenti della classe si somigliano tanto meno è visibile la mano dell’insegnate. Qualcuno ha pensato bene di sfruttare i trasferimenti chiesti dai docenti vedendoli come una sorta di esperimento naturale: se un insegnate lavora bene in una scuola e poi lavora bene anche nell’altra dove si trasferisce allora forse il miglioramento dei ragazzi è merito suo. E per un certo periodo – qualcuno ricorderà gli articoli di giornale – lo abbiamo creduto. Ma, ahimé, a quanto pare nemmeno i trasferimenti sono frutto del caso, la correlazione di molti fattori significativi tra classi scolastiche di provenienza e classi scolastiche di di destinazione invalida di fatto l’esperimento naturale fondato sui trasferimenti. Come sono fatti allora i buoni insegnanti? Esistono? Purtroppo sono domande ancora inevase. Gli anni passano e io comincio a pensare al peggio.

IRLE.BERKELEY.EDU

irle.berkeley.edu

Alla ricerca del buon insegnante

Alla ricerca del buon insegnante

Teachers: Much More Than You Wanted To Know By scott alexander

***

Domanda: esiste “il buon insegnante”? Risposta: sembrerebbe di no.

I.

Newspapers report that having a better teacher for even a single grade (for example, a better fourth-grade teacher) can improve a child’s lifetime earning prospects by $ 80,000.

AVERE UN BUON INSEGNANTE

if the test scores of two kids in the same teacher’s class were on average no more similar than the test scores of two kids in two different teachers’ classes, then teachers can’t matter very much.

UN TEST X VALUTARE IL PESO DELL’INSEGNANTE

they all agree pretty well that individual factors are most important, followed by school and teacher factors of roughly equal size. Teacher factors explain somewhere between 5% and 20% of the variance. Other studies seem to agree, usually a little to the lower end. For example, Goldhaber, Brewer, and Anderson (1999) find teachers explain 9% of variance; Nye, Konstantopoulos, and Hedges (2004) find they explain 13% of variance for math and 7% for reading. The American Statistical Association summarizes the research as “teachers account for about 1% to 14% of the variability in test scores”, which seems about right.

ESITO DI TRE STUDI: CONTA POCO

So put more simply– on average, individual students’ level of ability grit is what makes the difference. Good schools and teachers may push that a little higher, and bad ones bring it a little lower, but they don’t work miracles.

SINTESI DELL’EVIDENZA

it’s much easier to say “this is 10% dependent on school-level factors and 10% based on teacher-level factors”

%

In terms of observable teacher-level effects, the only one they can find that makes a difference is gender (female teachers are better). Teacher certification, years of experience, certification, degrees, et cetera have no effect. This is consistent with most other research, such as Miller, McKenna, and McKenna (1998). A few studies that we’ll get to later do suggest teacher experience matters; almost nobody wants to claim certifications or degrees do much.

GENDER E ESPERIENZA

The most robust finding in the research literature is the effect of teacher verbal and cognitive ability on student achievement. Every study that has included a valid measure of teacher verbal or cognitive ability has found that it accounts for more variance in student achievement than any other measured characteristic of teachers (e.g., Greenwald, Hedges, & Lane, 1996; Ferguson & Ladd, 1996; Kain & Singleton, 1996; Ehrenberg & Brewer, 1994).

ABILITÀ COGNITIVE

Teachers account for about 10% of variance in student test scores, it’s hard to predict which teachers do better by their characteristics alone, and schools account for a little more but that might be confounded.

CONFUSIONE IN AGGUATO

II.

Suppose you want to figure out which teachers in a certain district are the best. You know that the only thing truly important in life is standardized test scores [citation needed], so you calculate the average test score for each teacher’s class, then crown whoever has the highest average as Teacher Of The Year.

INDIVIDUARE IL MIGLIORE. FACILE?

But you’ll probably just give the award to whoever teaches the gifted class.

PROBLEMA OVVIO

So okay, back up. Instead of judging teachers by average test score, we can judge them by the average change in test score.

INCREMENTO NEI TEST

this is the basic idea behind VAM (value-added modeling), the latest Exciting Educational Trend and the lynchpin of President Obama’s educational reforms. If you use VAM to find out which teachers are better than others, you can pay the good ones more to encourage them to stick around.

VAM HIGHT STAKE

A teacher whose VAM is two standard deviations above the mean should have students who score on average 0.2 standard deviations above the mean.

CAMBIARE CLASSE

what happens if we compound this and give this student the best teachers many years in a row? Sanders and Rivers (also Jordan, Mendro, and Weerasinghe) argue the effects are impressive and cumulative.

UNO STUDENTE MEDIO NELLE MANI DEI MIGLIORI

A RAND education report criticizes these studies as “using ad hoc methods” and argue that they’re vulnerable to double-counting student achievement… Sanders and Rivers] provide evidence of the existence and persistence of teacher or classroom effects, but the size of the effects is likely to be somewhat overstated”….

CRITICHE

Just by eyeballing and playing around with it, it looks like most of the gain from these “three consecutive great teachers” actually comes from the last great teacher. So the superadditivity might not be quite right, and Sanders and Rivers might just be genuinely finding bigger teacher effects than anybody else. At what rate do these gains from good teachers decay? They decay pretty fast. Jacob, Lefgren and Sims find that only 25% of gains carry on to the next year, and only 15% to the year after that… Kane and Rothstein find much the same. A RAND report suggests 20% persistence after one year and 10% persistence after two…. All of this contradicts Sanders and Rivers pretty badly….

DATI SOSPETTI: I BENEFICI DECADONO

None of these studies can tell us whether the gains go all the way to zero after a long enough time. Chetty does these calculations and finds that they stabilize at 25% of their original value. But this number is higher than the two-year number for most of the other studies, plus Chetty is famous for getting results that are much more spectacular and convenient than anybody else’s. I am really skeptical here.

CHETTY SULLA DECADENZA

remember Louis Benezet’s early 20th century experiments with not teaching kids any math at all until middle school– after a year or two they were just as good as anyone else, suggesting a dim view of how useful elementary school math teachers must be.

INSEGNARE AI 13ENNI ANALFABETI

In summary, I think there’s pretty strong evidence that a + 1 SD increase in teacher VAM can increase same-year test scores by + 0.1 SD, but that 50%– 75% of this effect decays in the first two years.

SUMMARY

III.

I expected teachers’ groups and education specialists to be pushing all the positive results. After all, what could be better for them than solid statistical proof that good teachers are super valuable? In fact, these groups are the strongest opponents of the above studies… They argue that VAM is biased and likely to unfairly pull down teachers who get assigned less intelligent lower-grit kids….

L‘OPPOSIZIONE DEGLI INSEGNANTI AGLI STUDI

I think they have some good points about how VAM isn’t always a good measure. First, it seems to depend a lot on student characteristics; for example, it’s harder to get a high VAM in a class full of English as a Second Language students.

PRIMO DIFETTO DEL VAM

Also, a lot of VAM models control for student race, gender, socioeconomic status, et cetera. I guess this is better than not doing this, but it seems to show a lack of confidence– if controlling for prior achievement was enough, you wouldn’t need to control for these other things. But apparently people do feel the need to control for this stuff, and at that point I bring up my usual objection that you can never control for confounders enough,

INDIZI SULLA MANCANZA DI FIDUCIA

Maybe because of this, there’s a lot of noise in VAM estimates. Goldhaber & Hansen (2013) finds that a teacher’s VAM in year t is correlated at about 0.3 with their VAM in year t + 1. A Gates Foundation study also found reliabilities from 0.19 to 0.4, averaging about 0.3. Newton et al get slightly higher numbers from 0.4 to 0.6; Bessolo a wider range from 0.2 to 0.6. But these are all in the same ballpark, and Goldhaber and Hanson snarkily note that standardized tests aimed to assess students usually need correlations of 0.8 to 0.9 to be considered valid (the SAT, for example, is around 0.87).

NOISE DEL VAM

Even if VAM is a very noisy estimate, can’t the noise be toned down by averaging it out over many years? I think the answer is yes, and I think the most careful advocates of VAM want to do this, but President Obama wants to improve education now and a lot of teachers don’t have ten years worth of VAM estimates.

VAM MEDIO

Opponents argue that it might not be, and cite Paufler and Amrein-Beardsley‘s survey of principals, in which the principals all admit they don’t assign students to classes randomly.

ASSEGNAZIONE STUDENTI

Rothstein (2009) tries to “predict” students’ fourth-grade test scores using their fifth-grade teacher’s VAM and finds that this totally works. Either schools are defying the laws of time and space, or for some reason the kids who do well in fourth-grade are getting the best fifth-grade teachers.

I MIGLIORI CON I MIGLIORI

It would be nice to be able to draw all of this together by saying that teachers have almost no persistent effects, and the genetic component identified by Plomin and pointed at by Rothstein represents the 15– 25% “permanent” gain identified by Chetty and others which so contradicts my lack of line dancing memories.

GENETICA E INSEGNANTI

In summary, there are many reasons to be skeptical of VAM. But some of these reasons contradict each other, and it’s not clear that we should be infinitely skeptical.

GIUSTO ESSERE SCETTICI?

IV.

let’s go back to that study that says that a good fourth grade teacher can earn you $ 89,000. The study itself is Chetty, Friedman, and Rockoff (part 1, part 2).

TORNIAMO A CHETTY

This sounds impressive, but imagine the average kid works 40 years. That means it’s improving yearly earnings by about $ 1,000.

1000 ALL’ ANNO

They found that such teachers improved yearly earnings by about $ 300, but their study population was mostly in their late twenties and not making very much, and they extrapolated that if good teachers could increase the earnings of entry-level workers by $ 300, eventually they could increase the earnings of workers with a little more experience by $ 1000.

LA TESI CHETTY. PIÙ CHE ALTRO ESTRAPOLAZIONI

But really, who cares? The fact that having a good fourth grade teacher can improve your adult earnings any measurable amount is the weird claim here. Once I accept that, I might as well accept $ 300, $ 1,000, or $ 500,000.

PRIMA CRITICA A CHETTY

Everyone else has found that teacher effects on test scores decay very quickly over time. Chetty has sort of found that up to 25% of them persist, but he doesn’t really seem interested in defending that claim and agrees that probably test scores just fade away. Yet as he himself admits, good teachers’ impact on earnings works as if there were zero fadeout of teacher effects.

CHETTY NN PRENDE SUL SERIO IL DECADIMENTO

V.

Project STAR (Student Teacher Achievement Ratio) was a big educational experiment in the 80s and 90s to see whether or not smaller class size improved student performance… So Chetty, Friedman, Higer, Saez, Schanzenbach, and Yagan analyzed the STAR data….

CHETTY SFRUTTA IL L’ASSEGNAZIONE RANDOM DI STAR

When they’re using teacher quality to predict the success of specific students, they use the average of all the test scores

CON IL RANDOM IL VAM NON SERVE

They find that the average test score of all the other students in your class, compared against the average score of all the students in other randomly assigned classes in your school, predicts your own test score. “A one percentile increase in entry-year class quality is estimated to raise own test scores by 0.68 percentiles, confirming that test scores are highly correlated across students within a classroom”.

CONTANO I PEER… QUINDI L’INSEGNANTE

This fades to approximately zero by fourth grade, confirming that the test-score-related benefits of having a good teacher are transient and decay quickly.

MA ECCO TORNARE LA DECADENZA

Carrell finds that “exposure to a disruptive peer in classes of 25 during elementary school reduces earnings at age 26 by 3 to 4 percent”… If this is a big factor in the differences in performances between classes, then so-called “teacher quality” might be conflated with a measure of how many children in their classes are behavioral problems,…

PEER O INSEGNANTE?

Again, everybody finds that test score gains do not last nearly that long. So it can’t be that kindergarten teachers provide you with a useful fund of knowledge which you build upon later.

ASILO

The test scores gains from pre-K are notorious for vanishing after a couple of years, but a few really big preschool studies like the Perry Preschool Program found that such programs do not boost IQ but may have other effects (though to complicate matters, apparently Perry did boost later-life standardized test scores, just not IQ scores, and to further complicate matters, other studies find children who went to pre-K have worse behavior).

STUDI SIMILI.

there’s strong evidence that parents have relatively little non-genetic impact on their childrens’ life outcomes, but now we’re saying that even a kindergarten teacher they only see for a year does have such an impact?

ALTRO PARADOSSO CHE NDEBOLISCE I RISULTATI SUGLI ASILI

VI.

teacher quality probably explains 10% of the variation in same-year test scores. A + 1 SD better teacher might cause a + 0.1 SD year-on-year improvement in test scores. This decays quickly with time and is probably disappears entirely after four or five years, though there may also be small lingering effects. It’s hard to rule out the possibility that other factors, like endogenous sorting of students, or students’ genetic potential, contributes to this as an artifact… For some reason, even though teachers’ effects on test scores decay very quickly, studies have shown that they have significant impact on earning as much as 20 or 25 years later, so much so that kindergarten teacher quality can predict thousands of dollars of difference in adult income….

SUNTO: IMPATTO DECADENZA E PARADOSSI

I really don’t know whether to believe this and right now I say 50-50 odds that this is a real effect or not– mostly based on low priors rather than on any weakness of the studies themselves.

50/50

La “Buona Scuola” come missione impossibile

La “Buona Scuola” come missione impossibile

Null Hypothesis in education by Arnold Kling

***

Tesi: tra insegnamento e apprendimento non esiste una relazione chiara e quantificabile. E’ questo che dice la “null hypothesis. La portata di una simile ipotesi è vasta, per esempio: le risorse investite nell’educazione sono uno spreco. Oppure: la valutazione degli insegnanti è velleitaria. Eccetera.

***

In 2010, Teach For America (TFA) launched a major expansion effort, funded in part by a five-year Investing in Innovation (i3) scale-up grant of $ 50 million from the U.S. Department of Education. This study examines the effectiveness of TFA elementary school teachers in the second year of the scaleup, relative to other teachers in the same grades and schools. The study found that, on average, TFA corps members hired in the first two years of the scale-up period were as effective as other teachers in the same high-poverty schools in both reading and math.

MELISSA CLARK E GLI INTERVENTI SUGLI INSEGNANTI

We use administrative data on Swedish lottery players to estimate the causal impact of wealth on players’… The effects on most other child outcomes, which include drug consumption, scholastic performance, and skills, can usually be bounded to a tight interval around zero. Overall, our findings suggest that correlations observed in affluent, developed countries between (i) wealth and health or (ii) parental income and children’s outcomes do not reflect a causal effect of wealth…

DAVID CESARINI: RICCHEZZA E ISTRUZIONE

 In 1960, countries with an education level of 8.3 years of schooling were 5.5 times richer than those with 2.8 year of schooling. By contrast, countries that had increased their education from 2.8 years of schooling in 1960 to 8.3 years of schooling in 2010 were only 167% richer. Moreover, much of this increase cannot possibly be attributed to education, as workers in 2010 had the advantage of technologies that were 50 years more advanced than those in 1960. Clearly, something other than education is needed to generate prosperity….

RICARDO HAUSMAN SU RICCHEZZA E ISTRUZIONE

A new study of 10,000 teachers found that professional development— the teacher workshops and training that cost taxpayers billions of dollars each year— is largely a waste.

LINDSEY LAYTON. INVESTIRE SUGLI INSEGNANTI È UNO SPRECO

The Null Hypothesis is this: 1. Take any educational intervention 2. Conduct a rigorous controlled experiment. 3. Look for results that do not fade out within a year or two. 4. If you find apparent success, try to replicate it 5. You will not find significant effects in all of steps (1)– (4)

DEFINIZIONE DI NULL HYPOTHESIS

For teacher training to work, you have to deal with the null hypothesis at two layers. First, you have to find an intervention that significantly affects teachers. Then that intervention has to significantly effect students. This is the null hypothesis compounded.

ESEMPIO DI NH COMPOSTA

The Chicago public-school system implemented a “whole-school” intervention called the Teacher Advancement Program (TAP), which was supposed to raise the quality of its teacher workforce. Teachers were given the opportunity to earn performance pay and promotions through a special system of mentoring and observation. Although the literature on TAP is extensive, IES focused on the most reliable study— an experimental evaluation conducted by Mathematica a few years ago. There are questions about how faithfully TAP was implemented in Chicago, and the control group was small, but here is the study’s bottom line: There were no significant gains in student test scores, regardless of grade level or subject area.

JASON RICHWINE

Andrew Flowers reports on the Chetty, et al study showing differences in teacher value added, Their numbers are being replicated in many different settings. Even in Rothstein’s paper critiquing their method, he replicated their results using data from North Carolina public schools. “I’m not aware of another area of social science where there has been so much replication, in such a short time, and they’ve all found the same result,” Kane said.

CONTRO LA NH… CHETTY

As is evident, pre-K and control children started the pre-K year at virtually identical levels. The TNVPK children were substantially ahead of the control group children at the end of the pre-K year (age 5 in the graph). By the end of kindergarten (age 6 in the graph), the control children had caught up to the TNVPK children, and there were no longer significant differences between them on any achievement measures.

DALE FARRAN SULLA PRESCUOLA

This will create some cognitive dissonance for progressives who have faith in universal pre-K and also believe in using rigorous social science to guide policy. And some cognitive dissonance for James Heckman.

PROBLEMI PER HECKMAN

Although family disadvantage is strongly correlated with schools and neighborhood quality, the SES gradient in the sibling gender gap is almost as large within schools and neighborhoods as it is between them.

DAVID AUTOR SULL’AMBIENTE SCOLASTICO

I view the quoted sentence as throwing some cold water on Raj Chetty’s view that neighborhoods make a big difference.

AUTOR VS CHETTY

In summary: teacher quality probably explains 10% of the variation in same-year test scores. A + 1 SD better teacher might cause a + 0.1 SD year-on-year improvement in test scores. This decays quickly with time and is probably disappears entirely after four or five years, though there may also be small lingering effects. It’s hard to rule out the possibility that other factors, like endogenous sorting of students, or students’ genetic potential, contributes to this as an artifact, and most people agree that these sorts of scores combine some signal with a lot of noise… This seemingly unbelievable finding has been replicated in quasi-experiments and even in real experiments and is difficult to banish. Since it does not happen through standardized test scores, the most likely explanation is that it involves non-cognitive factors like behavior….

SCOTT ALEXANDER SU QUANTO PESA LA QUALITÀ DEGLI INSEGNANTI

I think that the right way to put it is that no one should pretend to know very much or have great confidence about claims about how teaching affects outcomes, particularly in the long term.

LA MISTERIOSA RELAZIONE INSEGNAMENTO APPRENDIMENO

In other service businesses, we let the customer make a subjective judgment of quality. You do not pick your hair stylist or your doctor or your auto mechanic based on some distant economist’s regression analysis. The reason we don’t use customer judgment in education is that we let government, rather than the customer, pick the service provider.

NH E SOCIALIZAZIONE DEL SERVIZIO

charter schools increase test scores and four-year college enrollment, but have a small and statistically insignificant impact on earnings, while regular charter schools decrease test scores, four-year college enrollment, and earnings. Using school-level estimates, we find that charter schools that decrease test scores also tend to decrease earnings, while charter schools that increase test scores have no discernible impact on earnings.

WILL DOBBIE SULLE CHARTER SCHOOL

My guess is that their findings are random variations around the null hypothesis.

COME VALUTARE I STRANI RISULTATI DI DOBBIE

Studies that show significant effects of educational interventions are right in this wheelhouse. That is why until they are scaled, replicated, and shown to have durable effects, you should view accounts of such studies with skepticism.

COME VALUTARE REALMENTE UN PROGRAMMA

The results demonstrate that expenditures and related school inputs have very weak associations not only with test scores in the sophomore and senior years of high school but also with high school graduation and subsequent college entry… The overall conclusion of the Coleman Report— that family background is far and away the most important determinant of educational achievement and attainment— is as convincing today as it was fifty years ago….

STEPHEN MORGAN… FAMIGLIA E SCUOLA

Why do the academic effects of early childhood education so often fade out?

THIMOTY TAYLOR

“The gains to children in Massachusetts charters are enormous. They are larger than any I have seen in my career,” [education researcher Susan] Dynarski wrote.

SUSAN DYNARSKI… SU RICERCHE A BREVE

Zimmerman (2014) compares students whose GPAs are either just above or just below the threshold for admission to Florida International University, a four-year school with the lowest admissions standards in the Florida State University System. This study finds that “marginal students” who are admitted to the school experience sizable earnings gains over those who just miss the cutoff and are thus unlikely to attend any four-year college, translating into meaningful returns net of costs and especially high returns for low-income students.

ZIMMERMAN SUL MERO COLLEGE PREMIUM

One of the Obama administration’s signature efforts in education, which pumped billions of federal dollars into overhauling the nation’s worst schools, failed to produce meaningful results, according to a federal analysis. Test scores, graduation rates and college enrollment were no different in schools that received money through the School Improvement Grants program— the largest federal investment ever targeted to failing schools— than in schools that did not.

EMMA BROWN SUI FALLIMENTI DI OBAMA NELLA SCUOLA

At some of the most prestigious flagship universities, test results indicate the average graduate shows little or no improvement in critical thinking over four years. Sounds like another win for the null hypothesis.

WSJ

Neerav Kingsland writes, Overall, CMOs are delivering +. 03 SD effects over three years in both reading and math. These gains are driven by the fact that students benefit from CMOs the longer they stay in them In this context, CMOs are charter school management organizations, not collateralized mortgage obligations. I still think that the overall picture tends to support the Null Hypothesis, although I believe that charter schools are capable of saving a lot of money while producing the same (null) effect as government schools.

CHARTER: NH + RISPARMI

I would speculate that our school system probably makes more efficient use of resources than would a system in which schools did not exist. I would speculate that it makes less efficient use of resources than would a system of vouchers and competition rather than government-managed schools. Would I go far as to say that the only difference that schools make is in resource allocation, not in outcomes? I doubt that such an extreme position is warranted. But statistically, educational interventions tend to affect resource allocation much more than outcomes. For educational interventions within roughly the current institutional setting, the null hypothesis is not an iron law, but it is an empirical regularity.

REFRAME NH