Percentiles do not work

Some Concerns About the LA Times’ Attempt at Teacher Evaluation
[Via Mike the Mad Biologist]

The LA Times has taken upon itself to rate school teachers in Los Angeles. To do this, the LA Times has adopted the ‘value-added’ approach (italics mine):

Value-added analysis offers a rigorous approach. In essence, a student’s past performance on tests is used to project his or her future results. The difference between the prediction and the student’s actual performance after a year is the “value” that the teacher added or subtracted.

For example, if a third-grade student ranked in the 60th percentile among all district third-graders, he would be expected to rank similarly in fourth grade. If he fell to the 40th percentile, it would suggest that his teacher had not been very effective, at least for him. If he sprang into the 80th percentile, his teacher would appear to have been highly effective.

Any single student’s performance in a given year could be due to other factors — a child’s attention could suffer during a divorce, for example. But when the performance of dozens of a teacher’s students is averaged — often over several years — the value-added score becomes more reliable, statisticians say.

While I laud the attempt to approach this issue quantitatively, I have serious doubts about their methods (Note to LA Times: methodological issues don’t make an approach “controversial”; they can make it wrong). Let us count the ways:

[More]

The first thing I noticed was that using percentiles could be a problem. One nice trick I have learned over the years is to look at the extremes and see if things get weird.

Let’s take the case – low that it might be – that every single 4th grade teacher is a superstar and takes every 3rd grader they get and move them up to 5th grade levels. That is, the entire cohort is moved en masse two years ahead. If everything stays the same, then there will be no change in the percentiles at all. Someone who was in the 10th percentile of his cohort will still be there. Someone who is in the 99th will still be ahead of 99% of all their colleagues, just at a 5th grade level.

That makes no sense. As does the other, what is all the 4th grade teachers are so bad that no one progresses beyond a 3rd grade education. The percentiles again will not change.

Using percentiles is a zero-sum game. If someone moves up, someone else has to move down. That is the nature of percentiles. In addition, the ends of the curve are bounded in ways that make little sense for this approach. The smartest cannot get any smarter – the 99% can not become 101%. They simply stay on top.

So a teacher who happens to get a group at 90% could do a great job but see no improvement at all. The kids are still better than 90% of their fellows. The only way we would see anything is if the teacher somehow caused some of the 90% students to do much worse. Then they would get dinged.

Why would a teacher want a class of really smart kids? They only thing they would be able to accomplish is to hurt their own future. Much better to teach a bunch at 10%. Even if the teacher are lousy, they will not drop and the teacher might get lucky and drive a few up to 20%, making themselves look like a great teacher.

So this system would provide no real chance for a teacher with exceptional students but great chances for a teacher with awful students. SOmehow I think this is the worng incentive to have.

Sounding strident and desperate for a reason

by divemasterking2000
Letters About Hate-Mongering Science Bloggers
[Via Mike the Mad Biologist]

After l’affaire Heffernan, I was curious to see what, if any, letters to the editor would appear in the NY Times. The Sunday Magazine printed two letters, both critical of Heffernan, which suggests to me, that there were very few, if any, supporters of Heffernan’s position (An aside: anyone know if the letters actually make their way to Heffernan? Just wondering).

I like this point:

If some bloggers sound desperate and strident, it’s possibly because even into the 21st century, only 39 percent of Americans believe in evolution and one out of five believes the Sun revolves around the Earth.

Indeed.

[More]

We sound desperate and strident because dealing with this level of managed ignorance puts tremendously unnecessary stress on our ability to solve complex problems. Their viewpoint throws a wrench into the DIKW cycle, stopping it cold. They know what the answer is and want no further knowledge to be found.

Many people do not want any further understanding, they do not want any further knowledge to be created. So they use every trick to keep understanding from moving forward.

Science is all about understanding the world around us. It is always cranking the DIKW cycle, trying to get to knowledge, where an action can be taken, and eventually to wisdom, where the right action can be taken.

I tend to get upset with people who want to throw a wrench into the process we need to solve the complex problems around us. I am only human and even when I try, I can come off didactic, condescending and angry.

We need to crank the cycle rapidly and from many different angles; we need to synthesize a large amount of knowledge to get to wisdom. That is how will will solve the complex problems we have today.

People who willfully try to disrupt that process are threatening our ability to overcome so many hurdles and potentially could have huge ramifications on our ability to survive.

That is why I may sound desperate and strident sometimes. Too many people are actively working to prevent the discovery of knowledge so that they can stay comfortably inside their own Cargo Cult Worlds.

By doing so, they jeopardize the future for everyone. I have to do my best to prevent that, even though a lot of research indicates that it is very hard to get people to leave the luxurious walls of a Cargo Cult World for the unreasonableness of the Real World.

Is this a cynical view of the company that claims not to do evil, or not?

by DownTown Pictures Is Google Discouraging Paid Android Apps?
[Via Daring Fireball]

Royal Pingdom:

You can only pay for apps in 13 out of the 46 or so countries where Android phones are available. For those of you who like stats, 13 in 46 works out to less than 30%. Contrast this with Apple’s App Store, which supports paid apps in 90 countries. This is a huge advantage iPhone developers currently have over Android developers.

The cynical view: Google prefers free Android apps over paid ones, because free apps try to make money through ads, and Google serves nearly all the in-app ads for Android apps.

[More]

In addition, 57% of all Android apps are free, as opposed to Apple, which is about 25%.

The true customers of Google, like the dying Main Street Media, are its advertisers. It tries to bring consumer eyeballs to the ads so that it makes money. It tries to get developers to help drive the eyeballs to ads. Google does not directly make money from those eyeballs, which is why it can afford to give away all of its products. They all drive ad eyeballs.

So, the needs of Google are not necessarily congruent with those of customers or developers. If there is a conflict between customers and/or developers and advertising, Goggle has to select advertising. That is who pays the bills.

TANSTAAFL. Apple gets 30% of every paid app but nothing from free apps. So it is easy to see why it might push for more paid apps (This might change with iAd where Apple will get 40% of the revenue.) But Apple’s model has been to serve the customer and developer directly, and to get paid directly by them.

It does not support itself by the indirect inflow of money from an outside source, whose needs do not parallel those of the users.

There is an interesting book called Good Work which looks at areas where ethics and excellence overlap. It looked at the Newspaper industry and found little overlap, mainly because the media’s customer was the advertising interests. They paid the bills, so when anything conflicted between the needs of the readers/reporters/editors and those of the advertisers, the advertisers eventually won.

Something similar may very well happen to Google. Their fate is really in the hands of the advertisers not the people who rely on it.

An example of the wreck that is our Congress

by avinashkunnath

Google, Verizon warned not to “cable-ize” the Internet
[Via Ars Technica]

Four influential members of Congress, fed up with the Google/Verizon “industry-centered net neutrality policy framework,” are now demanding that the government act in the public interest to enforce real network neutrality.

More specifically, they want the FCC to act. Even though passing legislation through Congress would be cleaner and less legally fraught than implementing new rules at the FCC, current political realities make that impossible. It’s up to the FCC to git-er-done.

[More]

Few Republicans are for net neutrality, mostly wanting the large corporations to carve of the Internet and control it. But the Democrats are not much better with 73 of them against net neutrality.

Net neutrality means that bits of data I sent are treated just the same as bits of data sent by Google or Verizon or Apple. BIg companies do not get to make the internet work better and faster for them. In addition, they do not get to slow down competitor’s data.

Data is data and everyone’s is treated the same. Large corporations do not want that. They want to turn the Information Superhighway into a bunch of farm-to-market roads and contruct tollways for any real use of the Internet. from the article:

A Democratic legislative aide tells Ars that the votes simply aren’t in place, that members will grow even more skittish about passing controversial legislation as the November elections approach, and that no bill stands a realistic chance of moving through Congress unless the major carriers support it.

So, the Democrats will not pass anything, even if a majority want to. And, nothing passes unless the big corporations want it (This is one reason why many countries have more people connected to the Internet than we do and also are well on the way to full 4G cell technology while we are still puttering around trying to get 3G most places. Anytime we rely on the corporations to determine what legislation gets supported, you can bet that the final product will have few benefits for the people. Not unless the corporations are paid handsomely for it.)

So, again we have to rely on the Unitary President to get anything done. In fact, the Congress wants it done this way. Completely shirking their legislative duty, they empower the executive branch which has gathered power over the last 30 years.

Sometimes I feel like I am living during the last few years of the Roman Republic, before Julius Caesar crossed the Rubicon.

Corrupt organizations doing what they do best, I guess

NAB, RIAA seek to push FM radio into iPods and iPhones
[Via AppleInsider]

Radio broadcasters and music labels are seeking to legally mandate FM radio reception as a feature in all consumer mobile devices in an effort to expand the market for radio.

[More]

So, a dying organization wants to have the government force other companies to support its business model by adding extra complexity to product. This is not capitalism – there is no free market battle here – and is not even socialism – there is no public ownership, or even public need, of any part. It is a corruption.

The right way to do it is to make it worth Apple’s while. Pay Apple directly to put the chip in.

The corrupt way is to use the government to do it without giving any recompense to Apple. I’m sure they tried to get Apple to do it and Jobs told them to pound sand. Perhaps adding the chip would allow less room for the battery.

Luckily, it is not part of the legislation that got out of committee. Let’s hope it does not make it into the final legislation.

You know to watch your pocket when industry groups are saying they are doing this for the benefit of the public and their safety. As the article states, this is all about getting more money and forcing other companies to support it.

I believe these same groups were behind the attempts to kill internet radio by charging outrageous fees. Same old – same old.

How does this help the fans?

by oddsock

Pink Floyd albums removed from iTunes after record contract expires
[Via AppleInsider]

Numerous albums from legendary rock band Pink Floyd have disappeared from Apple’s iTunes store, along with other digital storefronts, following the expiration of a contract between the band and its record label.

[More]

Well, now the band has control of some important albums. But they are no longer available online. The longer they take to fix this, the less money they make.

Especially for albums several decades old. Most everyone who wanted the songs has already gotten them.

Of course, the same thing could be said about the Beatles catalog and we have been waiting years for that.

Without anything available online, an artist can quickly disappear.

[Listening to: Pigs (Three Different Ones) from the album "Animals" by Pink Floyd] But Links my no longer work

[Listening to: One of These Days from the album "Meddle" by Pink Floyd] Links still work