The Debate over Teacher Value-Add and Student Outcomes in Adulthood
Can the relationship between high performing teachers and students be observed and assessed at the individual level? On 15 January an article by Alicia Wood (Putting a dollar value on having top teachers, SMH) pronounced that good teachers can influence the earning power, teenage pregnancy rates and university enrolments of their students.
Can the relationship between high performing teachers and students be observed and assessed at the individual level?
On 15 January an article by Alicia Wood (Putting a dollar value on having top teachers, SMH) pronounced that good teachers can influence the earning power, teenage pregnancy rates and university enrolments of their students.
This is a heading bound to make us take notice. It is based on the findings of a recently published study by economists from Harvard and Columbia universities.
This study followed 2.5 million students over 20 years, using their test scores (NAPLAN types measures) and mapped this to their adult lives using the SES ratings of their geo-location and teenage pregnancies.
The actual claims made by the researchers are that when a high value add (top 5%) teacher enters a school, end of school year test scores in the grade in which he or she teachers rise immediately and students assigned to such high value add teachers are more likely to go to college, earn higher incomes and less likely to be teenage mothers. They claim that, on average, having such a teacher for one year raises a child's cumulative lifetime income by $50,000.
The article cites Bob Lipscombe from the NSW Teachers Federation who labeled the research simplistic: ''It is extremely difficult, if not impossible, to isolate teachers in that way ... A typical high school student has seven or eight teachers each year - which teacher takes credit for the results?
More goes into making an excellent teacher than just test results.''
On the other hand Helen Walton, President of the NSW Parents and Citizens Associations, suggested that: "The study's findings could empower parents who are concerned their child's teacher is not performing well. You need quality teachers in every classroom, in front of every child ... At the moment, if a parent or a group of parents are concerned that a teacher is not performing, the process required to put that teacher through an improvement program can take up to 12 months. By that time, it's too late.''
The research also attracted significant commentary in the US.
This article profiles some of the more interesting responses
To Ravitch one of the things most notable things about the treatment of this study is that it "was reported on Page One of The New York Times, covered on the PBS Newshour, and lauded by Nicholas Kristof in the Times. While the study itself did not have specific policy recommendations, one of the authors told the Times: "The message is to fire people sooner rather than later."
In that sense it was taken up by conservative education reformers immediately and was seen as a vindication of economists like "Erik Hanushek of the Hoover Institution, who has been arguing for several years that the key to improving education is to fire the bottom 5 to 10 percent of teachers based on the test scores of their students. In theory, if a "bad" teacher is replaced by an average teacher, then scores go up".
Ravitch states that, teachers are really important. They make a lasting difference in the lives of their students. Some teachers are better than other teachers. Some are better at raising students' test scores. The problems of the study are not technical, but educational.
...[They} point us to an education system in which tests become even more consequential than they are now. Teachers would work in school systems with no job protection, and their jobs would depend on the rise or fall of their students' test scores.
.....There would be even less time in our schools than now for the arts, history, civics, geography, the sciences, foreign languages, health, and physical education. There would be less time to read challenging literature. There would be less time for science experiments. There would be less time for field trips to museums or historical sites. There would be less time for anything other than getting ready for the state tests.
There would be less time for extracurricular activities. There would be less time for chorus or band or dramatics or painting or film-making. There would be less time to read books, whether novels or histories. None of these things is directly related to raising test scores.
What matters most is getting the right answer on the test. Divergent thinking would be discouraged because divergent thinking might produce wrong answers. So would originality, creativity, ingenuity, or any other display of independence or critical thinking.
We can expect that some teachers will find ways to avoid teaching the most challenging students and to avoid the most difficult schools and districts. Isn't that the way incentives work?
When you put all these likely outcomes together, it's hard to imagine that we will have better education for more kids. We might or might not have higher test scores, but at what cost? Under these circumstances, who will want to teach? Is there a large pool of average, good, or great teachers waiting in the wings?
... Now, to clear up any doubt, let me make it clear that I don't believe any school should hire or retain incompetent or "bad" teachers. If teachers can't teach, they should be fired. No one who is incompetent should be awarded due process rights. Teachers who are having problems should be evaluated by their (hopefully, experienced) principal and peers, offered help, and if they don't or can't improve, they should be terminated.
She also notes that one of the key conclusions of the study is ignored by the media.
"While these calculations show that good teachers have great value, they do not by themselves have implications for optimal teacher salaries or merit pay policies. The most important lesson of this study is that finding policies to raise the quality of teaching--whether via the use of value-added measures, changes in salary structure, or teacher training-is likely to have substantial economic and social benefits in the long run."
No one could disagree with that statement, certainly not me. As for me, I prefer deliberate efforts to raise entry standards into teaching, to improve teacher preparation, and to ensure that every school has a significant number of experienced teachers who are masters of their craft. That seems to be what the high-performing nations do. The goal would be to make teaching a prestigious profession, rather than a job that any college graduate-with only minimal preparation-can do.
Sadly, if you read the NY Times article, you see the predictable call to make it easier or quicker to fire the weak teachers. Wouldn't it be nice if, instead of fantasizing about terminations, some fiscal conservatives out there saw these types of studies and said, "You know, if we spent a little more money right now trying to improve the quality of teaching for every teacher, in every classroom, we'd be multiplying these effects by millions of teachers and hundreds of millions of students over the next several decades and we'd be adding billions of dollars in value! Let's do it!"
I don't have a problem acknowledging the following:
1) Good teaching helps students learn, poor teaching does not help, and can even hinder student learning2) Teaching is to some extent a stable trait, at least it seems so in our current school environment3) Some teachers are better than other teachers4) Some of this variation in teaching "ability" is can be reflected in student test scores, and in long term student and adult outcomes.
Teachers matter. I wouldn't be a teacher if I didn't feel this to be true. As a college teacher, on one hand I (every day) confront evidence of my powerlessness in the face of student circumstances. On the other, I have had a few students tell me how I have changed their lives, and I (sort of) believe them. I certainly think that some of my teachers changed my life.
I am a teacher and I try to improve. But I know that there are some people who start out better than I did, improve faster, and will always be superior [However] I don't find competitive teacher rankings by score (such as VAM) motivating, or acknowledge that they are the best way to measure my success.
I am adamant that this approach, as reflected in the quotes in the article, is toxic and corrosive to good teaching and to improving education in our country.
Dana Goldstein's response
Given the widespread, non-ideological worries about the reliability of standardized test scores when they are used in high-stakes ways, it makes good sense for reform-minded teachers' unions to embrace value-added as one measure of teacher effectiveness, while simultaneously pushing for teachers' rights to a fair-minded appeals process. What's more, just because we know that teachers with high value-added ratings are better for children, it doesn't necessarily follow that we should pay such teachers more for good evaluation scores alone. Why not use value-added to help identify the most effective teachers, but then require these professionals to mentor their peers in order to earn higher pay? That's the sort of teacher "career ladder" that has been so successful in high-performing nations like South Korea and Finland, and that would guarantee that excellent teachers aren't just reaching twenty-five students per year but are truly sharing their expertise in a way that transforms entire schools and districts.
Michelle Rhee's response
So, given the potential impact our teachers have on our kids and society, isn't it time to rethink how we assign, retain, evaluate, and pay educators? Shouldn't we take a hard look at teacher-layoff and teacher-tenure policies?
Let's consider pay. The average teacher in America makes roughly $55,000 a year. That's pretty paltry when you consider what's at stake. What's more, the way salaries increase over a teacher's career is outdated. Teachers typically receive salary bumps for time on the job or for earning advanced degrees that aren't actually linked to student achievement. The new study confirms that what matters most, and what teachers really ought to be rewarded for, is the ability to help kids make academic progress. Given the link between effective educators and their students' later earnings, shouldn't we be putting more money into our best teachers' paychecks now?
Valerie Strauss - in response to Michele Rhee
The authors say the data come from the years 1989 to 2009, which to me implies that the standardized test scores used are recent. Actually, the scores are from the 1990s, well before the No Child Left Behind era ushered in those high-stakes standardized tests you like to use to hold students and teachers and schools accountable.
The only data that come from 2009 concern the incomes of students who turned 28 in that year. Now, students who turned 28 in 2009 were born in 1981. Since the researchers used test data from grades four to eight, the students in question would have been 10-year-old fourth graders in 1991 and 14-year-old eighth graders in 1995.
So the authors used value-added data from 1991 to 1995 and then followed specific students from that period until they were 28 in 2009 and measured their income and other factors.
The executive summary, incidentally, never says the test data are so old. And on Page 5 of the actual report, the authors wrote: "An important limitation of our analysis is that teachers were not incentivized based on test scores in the school district and time period we study."
"Not incentivized" is a euphemism in this case for old data.
The following sentence on Page 5 says, "The signal content of value-added might be lower when it is used to evaluate teachers because of behavioral responses such as cheating or teaching to the test."
Signal content? That means, in this context, "a less reliable predictor."
This all leaves me wondering exactly how reliable any of this actually is.
Now you might be wondering, why am I telling you all of this? Why does it matter?
Well, it matters a lot. It shows that students going to school in the early 1990s - in the days before No Child Left Behind gave high-stakes tests such dangerous importance in accountability systems starting in 2002 - wound up doing quite well. They earned more, lived in better places, got into better colleges and didn't wind up as teenage parents.
So I ask you: Why exactly do we need high-stakes testing when the old tests seemed to be working just fine in helping students succeed? High-stakes tests have had many awful consequences: narrowed curriculum, cheating scandals, etc. This study seems to me to show we don't need them!
I should note that Footnote 9, which starts on Page 5, and Footnote 64 on Page 50 say that even in the low-stakes tests that were the basis of this study, there's a tendency for the top 2 percent of teachers ranked by value-added to have patterns of test-score gains that are consistent with cheating - and this percentage is, of course, much higher in the high-stakes era. You surely know all about the cheating scandal in Atlanta that pushed out the superintendent and others in a bunch of cities. In fact, there are investigations now into cheating in D.C. schools when you were chancellor!
Cheating distorts the outcome, which leaves one to wonder why the authors put this important factor in a few small footnotes. Hmm.
Give NCLB another ten years and it will get rid of innovative and creative elementary programs ...[and great teachers]... Not because Jason Kamras didn't come along and make them rich. Not because they got tired of waiting for savior Michelle Rhee to come along and sweep their lower performing colleagues out of the way. Not because no one came and measured their VAM. No, they got worn down by the bullshit that ALL teachers had to put up with in DCPS. They got tired of being told, in so many ways, big and little, that their effort didn't matter. I saw them often frustrated, even as I loved to learn from them.
So how do we acknowledge that good teaching matters? How do we retain and celebrate good teachers and get more of them and have fewer bad teachers? Here's my modest proposal:
- Provide decent working conditions for all teachers
- Prioritize improving teaching practice, not evaluating teacher quality.
- Pay all teachers a little bit more, but mostly help them do what they came to the profession to do: inspire their students to love learning.
- Respect and support a broad array of inquiry and knowledge, so that teachers can teach a rich and varied curriculum.
Do this, and you'll find bad teachers get better, good teachers want to continue teaching, and more people want to enter teaching.
Or, you could increase class sizes, increase high stakes testing, give a rare $20,000 bonus after making teachers compete with each other for salary gains based on VAM, and get going with that firing project. Let me know how that goes.
A recent study by Harvard and Columbia economists found that students with effective teachers are less likely to become pregnant, more likely to go to college, and more likely to get higher-paying jobs. ... Great teachers make an enormous difference and ineffective teachers are hurting our students' futures - we can't allow that.
We need to be able to identify those ineffective teachers and give them the support they need to grow. And if that doesn't work, we need to be able to move them out.
A real evaluation system that is based on measurable improvement in student performance and principal assessment ...is the only way we can do that.
But when we tried to get approval for such a system for just 33 struggling schools - 33 out of 1,700 - the UFT insisted on provisions that would make it even harder to remove ineffective teachers. Not the same - harder. As a result, those 33 schools $58 million in School Improvement Grants from the State.
Well, I can tell you this: We're not going to accept that. And we're not going to allow ineffective teachers to remain in those 33 schools or in any school.
Actually, ugh, no. What economists Raj Chetty and John N. Friedman of Harvard and Jonah Rockoff of Columbia want to do, apparently, is to identify and fire "weaker" teachers, for the sake of a barely perceptible increase in students' "lifetime income." Nobody has actually tried this yet; the report doesn't describe an experiment. It's just the conclusion they draw from their analysis of massive amounts of data gathered from public schools in New York City and cross-referenced against IRS records and the like.
Great! So how much more likely are these students to attend college, according to this paper? One-half of one percent more likely. And...If there are 25 kids in this average classroom, that means about ten grand per kid, per lifetime. Say they work for forty years: that comes to $250/year.
The aggregate figure is estimated and rejoiced over, but the teacher in question is an individual, not an abstract quantity. This is not enough money to be crowing about in the newspaper; still less is it enough to be contemplating firing anybody over. That the authors would deliberately spin the number in this way seems like headline-baiting.
To read the original research article go to: http://www.nber.org/papers/w17699.ack