Are You Smarter Than a Fifth-Grade Scientist?

Andrew J. Vickers, PhD


March 31, 2010

I've seen some great science projects at my children's elementary school. My favorite was a study of music preferences in dogs which found that, at least for the canine cohort under study, Led Zeppelin was preferable to Mozart. Without delving into the methodology of that particular study, let me say that I have been extremely impressed by the kids' working knowledge of basic scientific principles: They seem to understand a hypothesis, an experiment, data analysis, and even the concept of replication.

On this final point, replication, the kids seem to be doing a good deal better than some of my scientific colleagues. A main focus of my own research is molecular markers for prostate cancer. I have read dozens of papers over the years on markers that I had never heard of before and which I then never hear of again. As an example, I just typed "prostate cancer marker" into Medline and here is what I found in the first few papers alone: MSMB, KLK3-stimulating peptide B-2, 2-nitrotyrosine, CD 147, rs1447295, DG8S737, serum chromogranin A, Y-box protein-1. As someone active in the field, I have heard of only 2 of these before: MSMB and rs1447295.

To check whether I might just have been missing something, a colleague and I looked for papers on prostate cancer markers published in a major journal (to focus on the most important papers) in 2005 (to allow time for the papers to be replicated). We found 3 papers, including 1 in The New England Journal of Medicine, which claimed that autoantibodies were almost perfectly able to distinguish prostate cancer from benign disease. Searching the literature since 2005, we were unable to find any subsequent papers on autoantibodies or on the topic covered in the second paper. The third paper included 6 different markers, all of which had been subject to additional research. Four of the markers had no association with outcome in the subsequent paper; the other 2 did have a significant effect on risk, but in the opposite direction to that initially reported. So much for replication.

As it turns out, though, at least some of the blame may lay with the journals. Indeed, I have had marker papers rejected because they were replication studies. As Clinical Cancer Research puts it in its instructions to authors, submissions "must be original, not confirmatory." A fifth grader knows that you have to replicate, but a major cancer journal won't publish it if you do. We might also look to our career structure: We celebrate those who make scientific discoveries but not those who replicate their work (quick: who replicated Fleming's findings on penicillin?). And grants to replicate findings don't tend to generate much excitement among review committees.

All of which is to say that perhaps it would be better if we were a little bit more like fifth-grade scientists. A dog's preference for Jimmy Page power chords sounds a good deal less scientific than "serum chromogranin A," but at least the fifth grader seems to understand the need for replication.

If you liked this article, you'll love Andrew Vickers' collection of stories on statistics, What is a p-value anyway?


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.
Post as: