The Certainty of Uncertainty in Medicine

Hello and welcome. I am Dr George Lundberg and this is At Large at Medscape.

Do you remember the old cartoon that shows a corporate CEO diving under his desk or running out the back door when told by the office receptionist that "60 Minutes is here to talk"? That is the way American academics may feel when they learn that John Ioannidis may be examining their published work. The bigger they are, the more scared they become—and with good reason.

John Ioannidis (Disclosure: we are members of the same department at Stanford University) has made a career of questioning conventional wisdom and medical dogma that is based upon research, often by "good investigators" at "the best institutions," funded by the biggest agencies, and published in "the best journals." He has found much of these research findings to be either partially or completely false.

His 2005 paper, "Why Most Published Research Findings Are False,"^[1] is in the open-access journal PLOS Medicine. It has become the most accessed article in the history of the Public Library of Science, known as PLOS. John argues persuasively that many forms of bias operate in affected domains, fields, cultures, people's heads, reward systems, and ecologic groups. This plays out in small sample sizes; excessive flexibility in designs, definitions, outcomes, and analytic modes; and financial and other conflicts of interest. The mathematics, the psychology, and a misunderstanding or deliberate ignoring of the overriding importance of positive predictive values negate the validity of most—and he means most—published research.

John Ioannidis is not a one-hit wonder. He has published and lectured many times, but often along the same lines.

The Conventional Wisdom Bias

Now there is a new book, called Willful Ignorance; The Mismeasure of Uncertainty,^[2] by prominent statistician Herb Weisberg of the firm Causalytics in Needham, Massachusetts. I recommend this book to all serious medical thinkers who are concerned about the frequency with which we in medicine simply do not know the explanation for any particular disease, how to prevent it, how to diagnose it, how to treat or manage it, and how our "conventional wisdom" is so often turned upside down by new findings.

In some ways, there is nothing new about this. Many of us will recall being told by some professors in medical school that half of what we were being taught was wrong but that the professor (and the poor students) could not know which half. I have lived a life in medicine long enough to fully appreciate the wisdom of that professor, even though I thought at the time that the statement was made largely in jest.

Weisberg points out the vast amount of research that is being mass-produced and how little useful insight is generated by it. In addition, he emphasizes the gulf between quantitative research methodology results and clinical practice. One of the problems is the lower priority given to attempting to replicate published research, instead believing that an issue is settled. This bias applies to investigators, funding agencies, the academic reward system, peer reviewers, and medical editors.

How to Fix the Mess

Back to Ioannidis. In 2014, John helpfully proposed how to fix the mess.^[3] He is not opposed to research and he values scientific studies that have ultimately turned out to be valid and have benefitted the human condition, but he still alleges that an estimated "85%" of research resources are wasted.

Here is Ioannidis's description of research practices that may help to increase the proportion of research findings that are indeed true:

More large-scale collaborative research;
Adoption of a culture of replication;
Registration of studies, protocols, analysis codes, datasets, raw data, and results;
Sharing of data, protocols, materials, software, and other tools;
Enhanced practices of reproducibility;
Containment of conflicts of interest of sponsors and authors;
More appropriate statistical methods;
Better training of the scientific workforce in methods and statistical literacy;
Standardization of definitions and analyses;
More stringent thresholds for claiming discoveries or "successes";
Improvement of standards of study design;
Improvements in peer review, reporting, and dissemination of research results.

Okay, but that is a load of change for people, processes, institutions, finances, education, readers, promoters, and organizations—an ingrained, self-perpetuating culture. Lots of luck with that. But the alternative is continued "willful ignorance" just as Weisberg says. Let's get on with this sea change. Thank you, John and Herb.

That is my opinion. I am Dr George Lundberg, at large at Medscape.

Comments

Commenting is limited to medical professionals. To comment please Log-in.

Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.