Is Nutrition Research Seriously Flawed?

Epidemiological research in nutrition needs a complete overhaul, John P. A. Ioannidis, MD, DSc, argues in a recent JAMA commentary. Ioannidis, a professor of medicine and health research and policy at Stanford University School of Medicine in California, does not see value in continuing down the same path nutrition studies have followed for decades.

"The key message is that the noise is much stronger than the signal, even under the best circumstances," he told Medscape Medical News. The problem is not researchers themselves — the best scientists can't overcome ineffectual methods, he says — but the field's reliance on epidemiology to answer questions far too complex for epidemiological methods to address. "That whole literature doesn't get us anywhere," he says.

Problematic Associations and Relative Risk

Ioannidis argues that the nonstop barrage of associations reported between various foods and chronic diseases inappropriately implies causation and potentially causes more harm than good to the public.

"Unfounded beliefs that justify eating more food, provided 'quality food' is consumed, confuse the public and detract from the agenda of preventing and treating obesity," Ioannidis writes. Current guidelines may even contribute to some obesity, he suggests, if people consume too many "healthy food" calories, but don't spend them.

"Epidemiology can be valuable," he acknowledged to Medscape Medical News. And it has led to major progress, such as establishing the link between smoking and lung cancer. But those successes did not face the odds that nutritional studies are up against. Epidemiological studies and meta-analyses are too inadequate to provide new or valuable information about how different foods and nutrients actually affect long-term health risks, he posits.

"It's an intractable problem that's just too much for the tools that we have," he told Medscape Medical News.

But leading nutrition researcher Walter Willett, PhD, a professor of epidemiology and nutrition at Harvard's T. H. Chan's School of Public Health in Boston, Massachusetts, finds Ioannidis's argument more than unconvincing. Willett says it is inconsistent, unrealistic, and, perhaps ironically, given Ioannidis's own claims, potentially harmful to public understanding of science.

"He makes many fundamental errors and misrepresentations of existing literature and doesn't seem to understand even basic epidemiologic principles," Willett told Medscape Medical News.

Willett points specifically to Ioannidis's characterization of research linking hazelnuts to a longer life span. In his commentary, Ioannidis chastised inappropriate use of causal language among both study authors and the media, as these assertions mislead consumers into believing, for example, that eating 12 hazelnuts a day will extend life by 12 years.

Willett perceives this and similar examples as a fundamental inability to interpret epidemiological findings. And Willett isn't alone in his concerns: Ioannidis himself told Medscape Medical News that other researchers raised similar objections to what they perceived as his oversimplification of reported epidemiological findings.

In particular, Willett and others object to the apparent extrapolation of relative risk findings into lifetime benefits that appropriate statistical calculations, such as a life table or survival analysis, do not support. A more accurate interpretation would roughly translate to living maybe 1 more year if a person had that daily serving of hazelnuts, assuming the relationship is causal, Willett says.

But Ioannidis, who said he sent a clarification to JAMA about those concerns, says his intentions were misunderstood: He used the hazelnut example, as well as others, to illustrate the absurdity of the public's frequent misinterpretations of study results. He argues that reporting findings in relative risk is part of the problem precisely because those findings are so often misused in the way he described them, and hence the need for a new paradigm in researching nutrition and communicating results of that research to the public.

Nutrition's Complexity Simply Too Great, Ioannidis Says

Even so, Willett sees Ioannidis as grossly exaggerating the challenges of epidemiological nutrition research.

"We agree that we have measurement error, but it's not to a degree that would make the information not useful," Willett said. "Of course we understand that we cannot measure anything perfectly, but your study doesn't depend on perfect measurement."

The key to meaningful epidemiological nutrition studies, Willett says, is reducing measurement error through correction and repeating measurement over time with replication, thereby dampening or averaging out measurement error.

But to Ioannidis, those corrections are laughable, given the herculean task of researching nutrition in all its complexity.

"There are some valiant efforts," Ioannidis told Medscape Medical News, "but the number and types of confounding are so intricately related to so many dimensions of lifestyle, social environment, economic status, education, personal experiences, social connectedness, our whereabouts and local circumstances, our personal circumstances.... What and how we eat is dictated by zillions of things, and there's no way we can measure them."

The challenges of teasing out risks and benefits associated with specific food and nutrients harken back to the early days of genetic research, Ioannidis writes, when thousands of papers crudely linked chromosomal areas to disease risk.

"According to current knowledge, these previous efforts were doomed: each chromosomal area contains thousands of genetic variants," he writes. "Linkage scans resulted in numerous articles, but limited useful information."

Just as relying on a couple hundred microsatellite markers to understand an entire genome was, in retrospect, naive, so is relying on self-reported nutrition data to understand the complexities of an entire diet's interaction with an individual's genome and behavioral and anthropomorphic characteristics, as well as the effects of that diet on those interactions, he says. And adjusting for broad factors, such as socioeconomics, does not do justice to such extreme complexity, he says.

"We have 250,000 different foods and 300,000 edible plants alone, and each time we eat something, it's different," Ioannidis told Medscape Medical News. "Our genome is very complex and has many variants, but it's fixed in our whole lifetime. Nutrition changes from one meal to the next."

Such vast possibilities also open the door to selective reporting, Ioannidis told Medscape Medical News, alluding to P-value hacking and researchers' choices of statistical methods in studies with multiple comparisons.

"We have zillions of ways we can analyze the data," Ioannidis said. "If you have even the slightest preference for one result, that one type of nutrient can be good or bad, you can get that result from any data set if you analyze it a certain way."

Unlike Willett, Ioannidis has little optimism about the usual self-correcting nature of research unless a major paradigm shift occurs. Genomic research eventually matured into more sophisticated methodology, but the early results never directly affected public policy and people's everyday decisions the way nutrition research does, Ioannidis says.

Further, epidemiological nutrition research methodologies haven't substantially improved, yet their continued publication "spuriously affect[s] guidelines, and confuse[s] the public through heated advocacy by experts and nonexperts," Ioannidis writes.

Nutrition Research Isn't Broken, Willett Contends

But Willett points to epidemiological successes such as discovering that it is trans fat, not total fat, that harms health. This conclusion and others arose from multiple studies over the years that honed nutritional knowledge.

"Despite all the noise on the web and among some self-appointed 'experts' who actually have not done any nutrition research, we do know the basics of healthy diets, even though there are many details [yet] to be investigated," Willett told Medscape Medical News.

That doesn't mean Willett doesn't see room for improvement. He shares Ioannidis's concerns about meta-analyses, for example, as he wrote in an article published in JAMA in October 2017.

"There's a lot of bad meta-analyses floating around, and part of it is that anyone with an Internet connection can do a meta-analysis," he told Medscape Medical News. But high-quality pooled data analyses reveal highly consistent, reproducible results across studies, he said.

In fact, he sees the application of research findings as a bigger problem than the findings themselves, with physicians and the public slow to follow the nutrition advice that is available.

He also acknowledges how industry funding can muddle the field.

"We do have a broad mix of well done and very weak studies," he said. "Some of this is purposeful, because of industry funding of studies that are destined by design to get misleading results."

Although these studies can contaminate poorly conducted meta-analyses, a simultaneous increase in high-quality prospective studies has strengthened the evidence base for today's nutritional guidelines, Willett argues. He points to the broad evidence base that supports a Mediterranean-type diet's health benefits as one example.

"I think we are generally on a good path that uses multiple research approaches and that is actively working to enhance methods," he said. "The main problem is only modest funding compared to what is being invested in areas like genomics and drug development. We need research on both further refining our knowledge on diet and health and also on translation of current knowledge into practice."

Large RCTs or More Prospective Observational Studies?

Ioannidis pushes for a completely different remedy: a full-fledged moratorium on epidemiological nutrition studies with a pivot to more "large-scale, long-term, randomized trials on nutrition...especially for assessing diet patterns," such as the Prevención con Dieta Mediterránea (PREDIMED) trial of a Mediterranean diet, he writes.

Although PREDIMED's publication initially suffered randomization problems that required retraction and republication, its ultimate finding of no survival benefit is useful, as other negative findings from large pragmatic trials might be, Ioannidis argues.

"[T]heir outcomes may help inform nutritional guidelines with some pragmatic 'intention-to-eat' data," he writes. Ioannidis wants to see more experimentation, even if it seems to lead to a dead end.

"Don't underestimate the complexity, and be open to ideas from scratch," he told Medscape Medical News. "It may be that nothing works. That would still be useful to know."

But Willett is deeply critical of Ioannidis's push for large RCTs, arguing that the inevitable lack of adherence to such studies would render them useless.

"You really do want to know if there is efficacy of the intervention," Willett told Medscape Medical News. He even coauthored a perspective piece in Advances in Nutrition in July 2018 that directly addressed the problems with using large RCTs instead of prospective epidemiological studies.

After countering the four "myths" about nutrition research that directly align with Ioannidis's critiques, Willett and coauthors said switching to RCTs would not be feasible and was "unlikely to advance nutritional sciences or improve policies."

"To move forward, we should continue to improve study design and diet assessment methodologies, reduce measurement errors, and leverage new technologies," Willett and his colleagues wrote. "Advances in the field lie in coalescing evidence from multiple study designs, methodologies, and technologies, and translating what we already know into policy and practice, so we can improve diet quality and enhance health in an equitable and sustainable manner across the world."

Where Ioannidis and Willett most agree is that nutrition research must be high quality and clearly communicated to the public if it is to benefit people.

"Physicians should know that dietary choices can have a major impact on the well-being of their patients, and physicians need to be more engaged in guiding their patients to healthy diets," Willett told Medscape Medical News.

Ioannidis and Willet have disclosed no relevant financial relationships.

JAMA. Published online August 23, 2018. Extract

For more news, join us on Facebook and Twitter

Comments

Commenting is limited to medical professionals. To comment please Log-in.

Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.