Perception and Publication Bias
[Via NeuroLogica Blog]
The psychological literature is full of studies that demonstrate that our biases affect our perception of the world. In fact psychologists have defined many specific biases that affect not only how we see but how we think about the world. Confirmation bias, for example, is the tendency to notice, accept, and remember data that confirms what we already believe, and to ignore, forget, or explain away data that is contradictory to our beliefs.
Balcetis and Dunning have published a series of five studies that add to this literature by showing what they call “wishful seeing.” In their studies they found that people perceive desirable items as being physically closer to them than less desirable items. This finding is plausible and easy to believe for a skeptic steeped in knowledge of cognitive flaws and biases. But is this finding itself reliable? Psychologists familiar with the history of this question might note that similar ideas were researched in the 1950s and ultimately rejected. But that aside, can we analyze the data from Balcetis and Dunning and make conclusions about how reliable it is?
Recently Gregory Francis did just that, revealing an interesting aspect of the “wishful seeing” data that calls it into question. Ironically the fact that Balcetis and Dunning published the results of five studies may have weakened their data rather than strengthen it. The reason is publication bias.
Read the whole article because it does a nice job demonstrating one of the reasons to be skeptical of small studies. And it helps explain things lke why successful Phase 2 clinical trials – small trials looking for efficacy – fail in Phase 3 – large scale trials.
Small studies should have greater variation in them. So if you repeated the same small study multiple times, you should get them varying on both sides of the ‘real’ value.
As you increase the size, the variation decreases and you get closer to the real value.
Think of flipping a coin. Let’s do it 6 times. You might get 2H-4T. Do it again. 3H-3T. Again, 1H-5T.
Even though we know the real answer should be 3H-3T we have a wide variety of results around this value.
Now imagine that if you had a 1H-5T result you could get it published while a 5H-1T would not be. That is publication bias.
Only further research would reveal that the 5H-1T result was not correct.
This is why most scientists hold little belief in a single small study. We have seen too many of them that eventually revert to the ‘real’ value which is often uninteresting.
It has to be the totality of several investigations into a complex area to get their attention.
The problem is that the media simply responds to publication. They never withhold judgement.