It's news, but is it true?

There's no such thing as a slow news day on the health beat. Even if there are no killer viruses on the loose, and no deadly bacteria spreading in the food chain, there is always "a new study" to report.

Every day we check science news sites, like this one, that give reporters advance notice about studies about to be published.  It's not unusual to see up to a hundred studies stacked like planes over an airport, each one hoping to land in the headlines, sometime over the next few days.

Experience tells me that if a study is about chocolate, coffee or red wine, it has an excellent chance of getting in the news no matter what the finding. It's the same for studies in high profile journals, such as Nature, Science, Lancet, and certain conditions like breast cancer, Alzheimer's, obesity, and autism. And if the study claims to have made a novel, 'first ever' discovery, then it will probably be in the news.

But what if a study found that most studies end up being wrong? Would that study make the news?  Well, it happened.  A paper with the startling title: "Why Most Biomedical Findings Echoed by Newspapers Turn Out to be False" was published a few weeks ago by a French research group. And no, that study did not make it into the headlines.

The researchers, lead by neurobiologist François Gonon, examined the way newspapers reported on a number of high profile studies on attention-deficit hyperactivity disorder (ADHD). They asked the question: do scientific claims reported in the media end up being proven true over time? Their answer: in most cases, no. Then they asked: do the media go back and set the record straight? No again.

In other words, we, in the media, make a big deal over a new research finding, but when it turns out to be less exciting, or even wrong after future research, we don't tend to report that. 'Never mind' doesn't usually make it into the news.

How often is research disproven?

It happens all the time, according to epidemiologist John Ionannidis. "There is increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," he wrote, in a 2005 article with the provocative title: "Why most published research findings are false."

He based that conclusion on his own research, published in a peer reviewed paper in the Journal of the American Medical Association. In that study, Ionannidis followed up 45 high-profile medical research findings published over the previous 13 years, each claiming to have found something that worked (vitamin E, estrogen, various drugs or surgical techniques). Ionannidis concluded that when those findings were tested further, in larger and better controlled studies, almost a third turned out to be either wrong, or much weaker than the initial claims. Less than half of the findings were able to be replicated, and a quarter were never re-tested. Now the French group has taken that research one step further.

"After Ionannidis' team convincingly showed that most initial biomedical findings are either refuted or strongly attenuated by subsequent studies," Gonon told me in an email this week, he said his group decided to investigate how research claims are treated in the media. He concluded that newspapers "preferentially report on initial findings" but don't report later studies that disprove or weaken them. 

'The media can inadvertently oversimplify, misrepresent, or overdramatize scientific results.' — Misinformation and Its Correction

By "initial findings," he is referring to research that claims to be the first time anyone has found the 'thing,' (that molecule A kills cancer cells, for example). And because a 'first ever' discovery is more newsworthy than one that 'confirmed what we already thought,' most research findings that make news are 'initial findings.' And initial findings are extremely vulnerable to becoming weaker after further research, (when molecule A only works in mice, or when molecule A turns out to be too toxic to give to humans and is therefore not likely to be a cure for cancer, after all.)

On top of that, Gonon says "initial studies leading to null observations are seldom published." In other words, research journals don't tend to publish studies that show nothing happened ("null observations" — when molecule A did not kill any cells at all, or didn't do it any better than molecule B), even though proving that 'something isn't there' is extremely important in the overall advancement of knowledge.

What it means in the real world is that although science shows up in the headlines in black and white, it should instead appear in the infinite shades of grey that more accurately reflect the inherent uncertainty.

Does the accuracy of the science in the media matter?

Yes, according to more research about research, which also did not end up in the news. An Australian  paper published last week called "Misinformation and Its Correction" explored the ways people come to be misinformed, pointing out that once a mistaken impression about a research finding gets out there, it’s difficult to get it back.

"Human culture strongly depends on people passing on information," the authors wrote. And the more surprising, or disturbing the information, the more eagerly it is spread.

"People seem to mainly pass on information that will evoke an emotional response in the recipient," they wrote. "Thus, stories containing content likely to evoke disgust, fear, or happiness are spread more readily from person to person."

And if enough people read about it, heard about it and talked about it, it's impossible to get back to all of them and say 'Oops, that thing we told you about turns out not to be true.'

"There are cognitive variables within each person that render misinformation 'sticky,' they wrote. "The second factor is purely pragmatic, and it relates to the ability to reach the target audience."

So who's responsible for all this scientific exaggerating and misinforming? The Australian paper lists the media as a major source of misleading scientific information.

"The media can inadvertently oversimplify, misrepresent, or overdramatize scientific results," they wrote.

But scientists share some of the responsibility too.

"I believe that journalists are not the only ones to be blamed." François Gonon, told me. "Scientists, and scientific editors are also responsible. Indeed, scientists make desperate efforts to have their articles accepted by prestigious scientific journals and overstate the interest of their findings," he said.

It's an observation that is supported by yet another paper, coincidently published in the last few weeks, also looking at research in the news, and also, like the others, absent from the news headlines.

This paper looked for 'spin' in media reports about randomized controlled trials, the so-called 'gold standard' of research trials. It found 'spin' in half of the stories and press releases. Who did the spinning? The authors determined that the main factor associated with spin was "the presence of spin in the article abstract conclusion." In other words, the scientists, who wrote the abstract, started the spin, by stressing the beneficial aspects of whatever treatment or drug they were testing.

"This tendency," the authors wrote, "in conjunction with other well-known biases such as publication bias, selective reporting of outcomes, and lack of external validity, may be responsible for an important gap between the public perception of the beneficial effect and the real effect of the treatment studied."

Of course all of these studies about the accuracy of science in the media could also be flawed, and subject to the same laws of "never mind."

Disturbing questions about the incompatibility of science and news.

Science 'evolves,' but news 'happens.' As reporters, we want to be able to tell you the 5 Ws, the Who, What, Where, When and Why of the story, with absolute certainty, even though in science  it's almost impossible to be eternally certain about anything. Which is why I keep this "three scientists" joke pinned to the bulletin board over my desk:

"Three scientists were on a train that had just crossed the border into Scotland. A black sheep was grazing on a hillside. The first scientist peered out of the window and said, 'Look! Scottish sheep are black.' The second scientist said, 'No no, some Scottish sheep are black.' The third scientist with an irritated tone in his voice said, 'My friends, there is at least one field, containing at least one sheep, of which at least one side is black some of the time.'"

One final note: when I asked François Gonon how much media coverage he received for his study about the accuracy of science in the news, he told me:

"I have tested the media coverage of our study using the data base Dow Jones Factiva and I have found no newspaper article echoing on our study."  He added, "The fact that the media coverage of our article is so small does not surprise me because our article has not been published in one of the few most famous scientific journals (e.g. Nature, Science, The Lancet)."

But the study was not completely ignored. A few days later this article appeared in The Economist, under the heading "Journalistic deficit disorder."