Why some science reports should be taken with a grain of salt

The new phrenology [Pharyngula]
[Via Pharyngula]

Morphological variation is important, it’s interesting…and it’s also common. It’s one of my major scientific interests — I’m actually beginning a new research project this spring with a student and I doing some pilot experiments to evaluate variation in wild populations here in western Minnesota, so I’m even putting my research time where my mouth is in this case. There has been some wonderful prior work in this area: I’ll just mention a paper by Shubin, Wake, and Crawford from 1995 that examined limb skeletal morphology in a population of newts, and found notable variation in the wrist elements — only about 70% had the canonical organization of limb bones.

taricha.jpeg
aorta.jpg

I’ve also mentioned the fascinating variation in the morphology of the human aorta. Anatomy textbooks lay out the most common patterns, but anyone who has taught the subject knows that once you start dissecting, you always find surprises, and that’s OK: variation is the raw material of evolution, so it’s what we expect.

The interesting part is trying to figure out what causes these differences in populations. We can sort explanations into three major categories.

  1. Genetic variation. It may be the the reason different morphs are found is that they carry different alleles for traits that influence the developmental processes that build features of the organism. Consider family resemblances, for instance: your nose or chin might be a recognizable family trait that you’ve inherited from one of your parents, and may pass on to your children.

  2. Environmental variation. The specific pattern of expression of some features may be modified by environmental factors. In larval zebrafish, for instance, the final number of somites varies to a small degree, and can be biased by the temperature at which they are raised. They’re also susceptible to heat shock, which can generate segmentation abnormalities.

  3. Developmental noise. Sometimes, maybe often, the specific details of formation of a structure may not be precisely determined — they wobble a bit. The limb variation Shubin and others saw, for example, was almost entirely asymmetric, so it’s not likely to have been either genetic or environmental. They were just a consequence of common micro-accidents that almost certainly had no significant effect on limb function.

When I see variation, the first question that pops into my head is which of the above three categories it falls into. The second question is usually whether the variation does anything — while some may have consequences on physiology or movement or sexual attractiveness, for instance, others may really be entirely neutral, representing equivalent functional alternatives. Those are the interesting questions that begin inquiry; observing variation is just a starting point for asking good questions about causes and effects, if any.

I bring up this subject as a roundabout introduction to why I find myself extremely peeved by a recent bit of nonsense in the press: the claim that liberal and conservative brains have a different organization, with conservatives having larger amygdalas (“associated with anxiety and emotions”) and liberals having a larger anterior cingulate (“associated with courage and looking on the bright side of life”).

Gag.

[More]

This leads into a wonderful examination detailing why this report not only does not really answer any interesting questions but also has many aspects that make any scientist skeptical of the reported results. I wrote about some of these aspects yesterday but this report has several that are more worrisome.

The paper has not even come out and no one knows which journal it will be in. You can almost bet that any science research that is reported by press release and the media before it has been seen by other scientists will be hyped to an extreme. Just look to the recent ‘arsenic-life form’ discussion. Lots of scientists spent a lot of time debunking much of the media reports of that work.

And that was for a paper that had been published in a prestigious journal. Here we have a report that has not seen the light of day, that no one has seen any real data for and that could simply be an outlier that will disappear when examined in greater detail.

As an example, scientists say something is statistically significant if there is less than a 1 in 20 chance it could happen by random. But, if you look at 30 different things – fishing for a difference – then even a 1 in 20 chance becomes possible, purely through random chance. Here is an easy way to visualize it.

Say a lottery has a 1 in 7 million chance of winning. This is because there are 7 million possible number combinations. Very small odds. But if you bought 7 million tickets – that is, you go fishing – you will definitely find a winning ticket. That does not mean anything. It is still random. You just looked long enough and did enough fishing to find some thing.

This may be what is going on here. We don;t really know because we have not seen the data.

So, do not trust this yet because the paper has not been published, we have only seen media reports and there has not been the necessary vetting of the work. Finally, publishing this sort of ‘data’ without any sort of understanding of why, except for some sort of handwaving, means that it should be taken with very little salt.

Data without an underpinning model that can be examined further does us little good in understanding the world around us. And, as often happens, that data becomes worthless under further scrutiny. Give us data and a model that permits greater exploration, vetting and possible destruction. That is what scientists do.

Publicity hounds publish data in the media before publishing it in a journal.



2 thoughts on “Why some science reports should be taken with a grain of salt

  1. But Clarence Page, an esteemed columnist for the Chicago Tribune, wrote a column about this. How does he know about it if it hasn’t been published? My, my. Have you all looked in an UFO magazine? Or one like that? You proper scientists may be missing an article published in Vampires Today or similar journal.

    1. He knows abut it because it was more or less commissioned by a British tabloid and they published the results, before it has been published in a journal. That is what I meant by publishing by press release.

      In my experience, the greater the hype from an article before it has even been seen in print, the greater the likelihood that everything written about it will be wrong.

      Fir instance, no mention of error bars, size of sample, etc. No statistics. All nuance from a paper is gone in order to hype the newest ‘finding.’

Comments are closed.