Look at Your Garbage Bin: It May Be the Only Thing You Need to Know About Statistics

Andrew J. Vickers, PhD

Disclosures

November 03, 2006

In This Article

J'accuse

J'accuse: Many of the medical research papers you read will be wrong, not as a result of methodologic flaws, poor design, or inappropriate statistics, but because of typing errors.

I was doing some research on sample size calculation and was reading up on how such calculations were reported in the very best journals. The very first paper I read was in the prestigious British Medical Journal: the authors stated that they were looking for the drug to improve pain by 16 points on a pain scale that had a standard deviation of 8. Now you may notice that this is a difference of 2 standard deviations, an absolutely massive effect (indeed, an adequately powered trial would require only 12 patients). When I emailed the authors, they apologized and said that they meant to give a figure of 18 rather than 8 for the standard deviation. The second paper I read, which was also published in the British Medical Journal, was also somewhat confounding: anxiety, depression, and fatigue had improved in the treatment group, but quality of life was worse. After investigating the issue with the authors, it turned out that quality of life was indeed better after treatment but that a minus sign had been omitted from the table of results.

I had provided advice to a researcher about a trial and promptly forgotten about it until he sent me a copy of the final manuscript: Could I read the final version and sign the copyright form before he submitted it with my name as an author? I told him that I usually do not put my name to a paper unless I had personally checked the statistical analysis and so asked for a copy of the study database. Although I reproduced the analyses described in the paper, something still felt odd, so I asked for copies of the actual questionnaires on which patients had reported their symptoms. My caution was vindicated: I found 4 data entry errors in the record of the very first patient I checked; I also noticed that the study codes had been handwritten on the questionnaires such that the data for Patients 14 and 19 were likely reversed. I suggested repeating the data entry from scratch. When the paper was eventually published, the results were very different from those included in the paper I had originally seen.

A colleague showed me a paper pointing out that I could conduct a secondary analysis that would interest me. The authors generously sent me the raw data, but when I started my analysis, I immediately noticed some anomalies. The results included data on 2 biomarkers that are inversely correlated -- that is, you normally see high levels of one or the other, but not both. Yet a small number of patients did indeed have high scores for both biomarkers. When I asked the investigators about this, they said that they had checked the records for these patients, had found the data to be correct, and that double-positives sometimes happened. So I finished my analysis and presented the results to my colleague. She pointed out that the results were difficult to interpret without information on clinical stage. Now it turns out that very few early-stage patients are positive for one of the markers, and when we received the stage data, we noticed that several early-stage patients were marker positive; in fact, these were exactly the same "double-positive" patients we had asked about previously. When we raised the issue again, the investigators wrote back saying that on further checking there had been data entry errors, and that the values of the biomarkers had been reversed in some cases. Note that this was an important National Institutes of Health-funded study conducted at a major university and that the primary goal concerned the impact of the biomarkers on outcome.

I was asked to help a surgeon conduct an analysis of the effects of obesity on complication rates. The very first line of the spreadsheet he sent to me described a female patient who was 6 feet tall and weighed 135 pounds, roughly the anorexic look favored by the typical fashion magazine. Yet her body mass index was given as 49, which puts her in the category of the super-obese. It turned out that the surgeon had typed numbers from the surgical charts into a Web-based body mass index calculator and then cut and pasted the results back into the spreadsheet; inevitably, mistakes had occurred.

I could go on to give further examples, but reading about data entry errors in an article is almost as boring as having to deal with them in practice. So, for the rest of my time here, I'll focus on some potential solutions.

One of the principal characteristics of science is that it involves systematic attempts to avoid error. This is why we do everything from test tube washing to experimental replication and blind observation. What the medical researcher must do, therefore, is to put in place systems that decrease the probability of typing errors. These systems can be implemented at all stages of the research process.

Comments

3090D553-9492-4563-8681-AD288FA52ACE
Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as:

processing....