Statistics Is Unscientific! (As Practiced by Clinicians: Part II)

Andrew J. Vickers, PhD


March 03, 2008

A key characteristic of science is reproducibility (on which point, to prove to you just how scientific I am, I would like to state that a key characteristic of science is reproducibility). Consider: Two literature PhD students writing theses about, say, Hamlet, would not be expected to reach similar conclusions; indeed, there would likely be somewhat of a problem if they did. On the other hand, if 2 molecular biologists wrote dissertations with differing findings about the same cell pathway, their hopes for a tenure-track position would likely go the way of Rosencrantz and Guildenstern.

The reason why the 2 biologists should be expected to come to similar conclusions about a phenomenon is related to a second key characteristic of science, the systematic attempt to avoid error. We wash test tubes, calibrate instruments, use randomization in clinical trials, and tape-record interviews because we have found out that, if we don't, we often get misleading results.

Statistical analysis as typically conducted by most clinicians is neither reproducible nor involves a systematic attempt to avoid error. It is, thus, arguably unscientific. Let's take a typical study published in the typical clinical journal, such as a small clinical trial or a surgical case series. The statistical analysis presented is generally nonreproducible because clinical researchers are almost never willing to share clinical data (just try asking!). If I don't have your data set, I can't run analyses to see whether I get the same results as you. Moreover, many of the statistical analyses conducted by clinicians involve no systematic attempt to avoid error. I know this because I have seen all too many of them. Here is a quick guide to statistical procedures as implemented by, say, the typical surgery fellow:

  1. Download the data set from the research database into a spreadsheet, such as Microsoft Excel;

  2. Notice a couple of errors in the data set and make corrections directly onto the spreadsheet;

  3. Notice some missing data, pull chart notes, and type what is missing onto the spreadsheet;

  4. Cut and paste in a couple of columns of data from a different spreadsheet;

  5. Use spreadsheet functions to create some new data columns (eg, create a "time-to-recurrence" variable by subtracting "date of surgery" from "date of recurrence");

  6. Cut and paste the spreadsheet into a simple software package such as SPSS;

  7. Delete (or lose) the original spreadsheets;

  8. Use the pull-down menus on SPSS to run some analyses; and

  9. Cut and paste the analyses into a word processing document for the journal paper.

It is not hard to see how error can be introduced at just about every stage of this process. Indeed, the analysis could probably not be reproduced at all. In a previous paper in this series [please see Look at Your Garbage Bin: It May Be the Only Thing You Need to Know About Statistics ], I discussed some of the techniques you can use to avoid error. Here, I would like to focus on just one of them, which is statistical programming. (Okay, you might feel like stopping reading right here. Mentioning "statistical programming" isn't normally the sort of thing you do to pique someone's interest and is definitely not advisable on a first date, but bear with me, it is only a couple of sentences.) In brief, steps 1 through 9 can be achieved by writing statistical code. As a trivial example, here are 3 lines of code from 1 of my projects (it is not unusual to write 500 lines of code for an analysis):

* COMMENT: Correct data entry error for clinical stage
* COMMENT: Some patients with missing data on clinical stage were coded as "99"
replace stage="Missing" if stage = = "99"

Even from this short example, you can see how use of programming code allows complete reproducibility of a statistical analysis and involves a systematic attempt to avoid error. The code ensures that patients with missing clinical stage are appropriately labeled; "making corrections directly onto the spreadsheet" (the typical approach of the clinician in step 2) can, of course, easily lead to mistakes. Clinicians might find it somewhat depressing that they need to learn statistical programming code before they can run analyses, but I am sure their patients would prefer that if clinicians are going to do research, they at least try to do it properly.

The next article in this series will address issues of reproducibility in science, with a special emphasis on statistical programming.


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: