Meta-Analysis in the Mirror of Its Quotations: Science, Scepticism, Scorn, and Sarcasm

Franz H. Messerli MD; Sripal Bangalore; Adrian W. Messerli


Eur Heart J. 2019;40(40):3290-3291. 

In This Article


Perhaps the first meta-analysis was done by Pearson in 1904 on inoculation of enteric fever, attempting to statistically combine the Indian experience with the South African War experience.[2] The author reasoned 'Many of the groups in the South African experience are far too small to allow of any definite opinion being formed at all, having regard to the size of the probable error involved. Accordingly, it was needful to group them into larger series'. Pearson also critically observed that heterogeneity of the included studies was prone to cast doubt on the validity of the results in stating 'Even thus the material appears to be so heterogeneous, and the results so irregular, that it must be doubtful how much weight is to be attributed to the different results'. Until this very day, heterogeneity of the included studies has remained one of the most common pitfalls because there is near certainty that study design, demographics, and clinical characteristics of patients, specific therapeutic strategies, and outcomes vary from one trial to another. It becomes obvious that meta-analysis is only properly applicable if the data analysed are homogeneous—thus, therapy, patients, and endpoints must be similar or at least somewhat comparable.

Webster's Dictionary defines the term meta as: 'more comprehensive, transcending—usually used with the name of a discipline to designate a new but related discipline designed to deal critically with the original one'. Thus, a meta-analysis can be considered a discipline that comprehensively reviews and statistically analyses original publications, which are most often prospective randomized controlled trials. As stated by the Wall Street Journal 'Meta-analysis begins with scientific studies, usually performed by academics or government agencies, and sometimes incomplete or disputed. The data from the studies are then run through computer models of bewildering complexity, which produce results of implausible precision'.[1] In other words, computer models of bewildering complexity intend to distil multiple imprecise studies into a single precise answer. A busy practitioner might then erroneously believe that she no longer has to painstakingly analyse study after study in the medical literature, but rather that she may simply await the take-home message of a meta-analysis and practice medicine accordingly.

Consider the extensively quoted meta-analysis by Law et al.[3] on the use of blood pressure-lowering drugs in the prevention of cardiovascular disease. It consisted of a total of 147 randomized controlled trials enrolling almost half a million patients with varying inclusion and exclusion criteria, as young as 34 years to as old as 84 years, followed for <0.5–8.4 years. The spectrum of patients in these trials varied from those with or without hypertension, others post-myocardial infarction (MI), some with coronary artery disease without MI, some with heart failure, treated with a myriad of agents from placebo to various antihypertensive agents, at varying doses and frequency, with different blood pressure effect with dissimilar definitions of endpoint but still resulting in one simple conclusion—blood pressure measurement is useless! 'Our results indicate the importance of lowering blood pressure in everyone over a certain age, rather than measuring it in everyone and treating it in some'. Clearly, effect sizes computed from such exceedingly heterogeneous data could hardly be ascribed much validity, yet these data are often cited as establishing the efficacy of antihypertensive therapy. This may be a classic example of what Eyseneck would have called 'an exercise in mega-silliness'.[4]

Little question, as stated by the Nobel Laureate Ronald Coase 'If you torture the data long enough, it will confess'.[5] However, as with all torture, the important question is, confess exactly what?