A great story of how science checks itself and deals with apparent fraud by coming up with an entirely new, cheaper and more workable method to determine the results. Science got better because it had to overcome problems with previous models.
In 2015, two grad students at the University of California, Berkeley set out to study the art of political persuasion. It’s not easy to do this kind of research as it happens (or doesn’t happen) in the real world. Most of the time, when scientists study persuasion, they do it in a lab where the situation can be tightly controlled and the variables accounted for. But David Broockman and Joshua Kalla thought they had an easy solution. Little did they know, that simple fix would lead them to set off a scandal and, eventually, help advance political science.
A UCLA grad student named Michael LaCour and Columbia University political science professor Donald Green had just published some game-changing research showing that straight people who met a real, live gay person and listened to that person’s story could be persuaded to support marriage equality, and that this change of heart could be long lasting. The study was a big deal. The news made The New York Times. It was all over the web. It became the basis for an episode of “This American Life.”
Broockman figured he could save time and effort by just using LaCour and Green’s methodology. He even partnered with the same political organization, the Los Angeles LGBT Center. He studied LaCour’s paper like a Talmudic scholar, seeking insights that would help him set up his own experiments. And that was how David Broockman figured out Michael LaCour likely fabricated his results.
“We weren’t getting a great response rate, and we had no idea why,” Broockman told me. “We figured there must be some magic recruitment approach that the survey vendor had used. So we called up uSamp” — the third-party company LaCour said he used to conduct his surveys — “and we say, ‘Hey, whatever you did for him, can you do that for us?’ And they say, ‘Uuuh, we never did that for him.’”
Instead of Broockman being the student, learning from LaCour’s example, he became LaCour’s accuser. In May 2015, Broockman published a 27-page report that included evidence LaCour had failed to survey participants after they were exposed to the persuasion technique and had, instead, simply made up the results. LaCour has not admitted to this. FiveThirtyEight reached out to him for a comment, but he has not yet offered one.
The key innovative thing they did was to use an online survey to get an idea of people’s views before hand and use the results to set up a blinded control and experimental groups. So, instead of having to survey a lot of people where only few would actually take the survey, they started out with a group who were already predisposed to being interviewed.
This dropped the costs of the survey from $1million to about $25,000.
And to combat any attempts to suggest their work might be fraudulent, the researchers have been incredibly transparent with their work, making just about everything available to all to see.
They believe that, not surprising to me, that it is the skill of the interviewer in leading an open conversation that changes minds, not the personal story of the interviewer.
So, even though replication indicates the original study was very flawed, it lead to a new and much better approach. That is how science is supposed to work.
Image: Steve Rainwater