On the non-reproducibility of psychological studies

Nobel laureate Daniel Kahneman calls for ‘daisy chain’ of psychology replications
[Via Not Exactly Rocket Science]

Last Wednesday, Nobel laureate Daniel Kahneman sent an email to a group of a dozen or so psychologists telling them that the credibility of their field was in danger. The recipients all worked on social priming – the study of how subtle unconscious cues can influence our behaviour. It’s an area that has attracted controversy of late, due to failed replications of classic results, the outing offraudulentresearchers, and a more general concern among psychologists about the validity of their field’s results.

Kahneman meant the email as helpful advice, but his wording couldn’t have been stronger:  “Your field is now the poster child for doubts about the integrity of psychological research… I believe that you should collectively do something about this mess.” His solution: a “daisy chain” of replications, where laboratories collaborate to check the results of their neighbours, in an open, transparent, and pre-established way.

Kahneman requested that the email be sent to anyone relevant and, presumably because I have written several pieces on this topic, a copy landed in my Inbox on Wednesday.  I interviewed Kahneman about it, and my story about his …


Reproducibility is a problem in many areas of science, especially ones at the true cutting edge. We often do not understand all the complexities so very small differences in execution can result in different results.

Nowhere is this more seen than in studies of humans. 

It is not that these studies are wrong. But they may not be very robust. It turns out that all sorts of supposedly inconsequential things can alter the experiment.

Such as the person doing the experiment actually knowing what the purpose of the experiment is. It may well be that they give away certain things in their voice or body position that predispose the subject.

Read up on Clever Hans and you can see that this effect can so easily produce inaccurate results.

2 thoughts on “On the non-reproducibility of psychological studies

  1. Are you sure this isn’t a social experiment itself to show that even “scientists” don’ trust social science?

    1. But then how do I know that your reply is not some sort of social experiment to show how much scientists trust “scientists” who don’t trust social science? ;-)

Comments are closed.