I have learned to largely ignore headlines about the latest study proving that “X” is actually bad for you or that you should never do “Y”. And for good reason: I’ve grown sick of seeing a subsequent study that claims to prove the opposite.
A new set of studies (irony noted) has shown the merits of my cynicism. A group of social scientists set out to replicate the findings of 21 influential studies from the prestigious general science journals, Nature and Science. That means that independent researchers would conduct the experiments again and see if they got similar results. If so, then the study results should be considered valid. If not, they would be suspect.
Of the 21 replication tests 13 replicated successfully. However, even in the studies that replicated, the magnitude of the effect of the test decreased by about half. In other words, the impact was half of what the original experiment suggested.
The takeaway here is that no matter the pedigree of the researchers, their institution, or the journal in which they’re published, take the results of scientific experiments with a grain of salt until they are replicated.
Moving this from the social sciences to medicine ups the stakes dramatically, but here again, reader beware. Kevin Drum of Mother Jones magazine reports that back in the 90s, researchers sought genetic links to depression, and lo and behold, they thought they found the suspect genes. This year, however, a large new study came out that basically poo-pooed all of that 90s research. The 90s researchers thought they had found a set of alleles (called 5-HTTLPR) responsible for producing depression in people under stress. However, “recent research has shown that it’s wildly unlikely for a single gene variant to have a detectible impact on a complex disorder like depression. So a new study was done not just on 5-HTTLPR, but on all the other genes that were associated with depression, using modern tools that could work on samples of hundreds of thousands.”
The result? “…the effects of these gene variants are tiny, and (b) they’re so tiny we can’t say they really exist at all. There’s certainly no way that researchers with sample sizes of only a few hundred people had even a remote chance of discovering these effects, assuming they ever existed in the first place.”
Drum quotes a psychiatrist explaining how far researchers went with this imaginary link.
What bothers me isn’t just that people said 5-HTTLPR mattered and it didn’t. It’s that we built whole imaginary edifices, whole castles in the air on top of this idea of 5-HTTLPR mattering. We “figured out” how 5-HTTLPR exerted its effects, what parts of the brain it was active in, what sorts of things it interacted with, how its effects were enhanced or suppressed by the effects of other imaginary depression genes. This isn’t just an explorer coming back from the Orient and claiming there are unicorns there. It’s the explorer describing the life cycle of unicorns, what unicorns eat, all the different subspecies of unicorn, which cuts of unicorn meat are tastiest, and a blow-by-blow account of a wrestling match between unicorns and Bigfoot.
So I’ll remain skeptical on the next banner headline insisting that coffee will make you live forever, or a glass of wine will kill you. I’ll generally wait to take action until the study is replicated.
There is one set of scientific facts you can act on right now, though. If you stick with the main food groups of chocolate, meat, and diet soda, you’ll be just fine. Trust me, I’ve done the research.
Leonce Gaiter – VP Content & Strategy
