Percentiles do not work

Some Concerns About the LA Times’ Attempt at Teacher Evaluation
[Via Mike the Mad Biologist]

The LA Times has taken upon itself to rate school teachers in Los Angeles. To do this, the LA Times has adopted the ‘value-added’ approach (italics mine):

Value-added analysis offers a rigorous approach. In essence, a student’s past performance on tests is used to project his or her future results. The difference between the prediction and the student’s actual performance after a year is the “value” that the teacher added or subtracted.

For example, if a third-grade student ranked in the 60th percentile among all district third-graders, he would be expected to rank similarly in fourth grade. If he fell to the 40th percentile, it would suggest that his teacher had not been very effective, at least for him. If he sprang into the 80th percentile, his teacher would appear to have been highly effective.

Any single student’s performance in a given year could be due to other factors — a child’s attention could suffer during a divorce, for example. But when the performance of dozens of a teacher’s students is averaged — often over several years — the value-added score becomes more reliable, statisticians say.

While I laud the attempt to approach this issue quantitatively, I have serious doubts about their methods (Note to LA Times: methodological issues don’t make an approach “controversial”; they can make it wrong). Let us count the ways:

[More]

The first thing I noticed was that using percentiles could be a problem. One nice trick I have learned over the years is to look at the extremes and see if things get weird.

Let’s take the case – low that it might be – that every single 4th grade teacher is a superstar and takes every 3rd grader they get and move them up to 5th grade levels. That is, the entire cohort is moved en masse two years ahead. If everything stays the same, then there will be no change in the percentiles at all. Someone who was in the 10th percentile of his cohort will still be there. Someone who is in the 99th will still be ahead of 99% of all their colleagues, just at a 5th grade level.

That makes no sense. As does the other, what is all the 4th grade teachers are so bad that no one progresses beyond a 3rd grade education. The percentiles again will not change.

Using percentiles is a zero-sum game. If someone moves up, someone else has to move down. That is the nature of percentiles. In addition, the ends of the curve are bounded in ways that make little sense for this approach. The smartest cannot get any smarter – the 99% can not become 101%. They simply stay on top.

So a teacher who happens to get a group at 90% could do a great job but see no improvement at all. The kids are still better than 90% of their fellows. The only way we would see anything is if the teacher somehow caused some of the 90% students to do much worse. Then they would get dinged.

Why would a teacher want a class of really smart kids? They only thing they would be able to accomplish is to hurt their own future. Much better to teach a bunch at 10%. Even if the teacher are lousy, they will not drop and the teacher might get lucky and drive a few up to 20%, making themselves look like a great teacher.

So this system would provide no real chance for a teacher with exceptional students but great chances for a teacher with awful students. SOmehow I think this is the worng incentive to have.

One thought on “Percentiles do not work

Comments are closed.