Diet research built on a 'house of cards'?

The next time a headline about diet and disease catches your eye, check the fine print of the study.

If it is based on a food questionnaire — and there's a good chance it will be — then the conclusions should be handled with caution.

We're talking about hundreds if not thousands of papers published every year.- David Allison, University of Alabama at Birmingham

That's because people don't tell the truth, not even to scientists.

Scientists know this. Research has proven it. It's been an awkward problem in nutrition science for more than 40 years.

And that's long enough, according to an international group of nutrition researchers. They've launched a campaign to end the use of one of the most common research tools in nutrition science.

• Read more on the group's report

"All of these studies, if they are based on self-report estimates of energy intake, really don't contain scientifically meaningful information," said David Allison, a prominent obesity researcher at the University of Alabama at Birmingham and lead author on a report signed by 45 scientists from around the world.

This matters, the expert group warned, because the flawed data could result in public health policies that are not science-based, but built on a "house of cards," Allison said.

"We're talking about hundreds if not thousands of papers published every year."

Asking who ate what, and when

Is a high-fat diet linked to breast cancer?  Do fruit and vegetables protect against Alzheimer's disease?  How much of obesity can be explained by eating too many calories?

To get those answers, scientists need to gather data about who ate what, and when. So they ask research subjects to self-report, to think back, sometimes over months and years, and fill in a detailed food questionnaire about what they ate.

Other studies ask people to keep food diaries. And sometimes the researchers do random spot checks, calling subjects and asking them to remember everything they've eaten for the last 24 hours. 

The correct answer is that, on average, obese people eat more than non-obese people. The wrong result was found and that confused the field for many years by relying on self-report.- David Allison

But because humans are very bad at admitting or remembering what they ate, all the self-reported methods have shown evidence of bias.

For years, scientists were puzzled by paradoxical findings that obese subjects were eating fewer calories than people of normal weight.

"That was completely wrong," Allison said. "The correct answer is that, on average, obese people eat more than non-obese people. The wrong result was found and that confused the field for many years by relying on self-report."

There's also a tendency for research subjects to tilt their dietary recall towards healthier behaviour.

What about a recent study concluding that Americans are eating fewer calories?  It was based on self-reported data.

"We can't say they're wrong, they could be right. We don't know and, frankly, they don't know. And no one can know because we don't have the legitimate information," Allison said.

He admits the group is taking a controversial stand that has sparked mixed reaction from the nutrition science community.

'A refreshing point of view'

"It varies a lot with who you ask," Allison said. "Much of the scientific community applauds us and says it's a refreshing point of view."

"And as you might expect, some individuals who have based much of their research career and program on the use of these methods say we can't abandon them because in part we don't have anything better."

Marian Neuhouser of the Fred Hutchinson Cancer Research Centre in Seattle, who has done some of the research that has exposed the bias in self-reported data, agrees the data is flawed. But she says if researchers perform a backup biomarker study they can correct for some of those biases.

Biomarker studies using urinalysis are accepted methods of accurately measuring energy intake. But that requires research subjects to be brought into a lab for a urine test. Researchers say that's too expensive and impractical for large-population studies.

Considering the obesity epidemic and the level of chronic disease, surprisingly little work is being done to improve methods, said Ross Prentice, a professor of biostatistics at the Hutchinson Centre.

Technical solutions being considered include chin-mounted "chew monitors," or wrist monitors that measure hand-to-mouth movements.  Some have suggested using smart phones to take photographs of food.

Allison prefers a biologically based approach, perhaps a test of breath or urine for products of digestion, which does not depend on the honour system and the fallibility of human memory.