Association Is not Causation: Treatment Effects Cannot be Estimated From Observational Data in Heart Failure

Christopher J. Rush; Ross T. Campbell; Pardeep S. Jhund; Mark C. Petrie; John J.V. McMurray


Eur Heart J. 2018;39(37):3417-3438. 

In This Article

Abstract and Introduction


Aims: Treatment 'effects' are often inferred from non-randomized and observational studies. These studies have inherent biases and limitations, which may make therapeutic inferences based on their results unreliable. We compared the conflicting findings of these studies to those of prospective randomized controlled trials (RCTs) in relation to pharmacological treatments for heart failure (HF).

Methods and results: We searched Medline and Embase to identify studies of the association between non-randomized drug therapy and all-cause mortality in patients with HF until 31 December 2017. The treatments of interest were: angiotensin-converting enzyme inhibitors, angiotensin receptor blockers, beta-blockers, mineralocorticoid receptor antagonists (MRAs), statins, and digoxin. We compared the findings of these observational studies with those of relevant RCTs. We identified 92 publications, reporting 94 non-randomized studies, describing 158 estimates of the 'effect' of the six treatments of interest on all-cause mortality, i.e. some studies examined more than one treatment and/or HF phenotype. These six treatments had been tested in 25 RCTs. For example, two pivotal RCTs showed that MRAs reduced mortality in patients with HF with reduced ejection fraction. However, only one of 12 non-randomized studies found that MRAs were of benefit, with 10 finding a neutral effect, and one a harmful effect.

Conclusion: This comprehensive comparison of studies of non-randomized data with the findings of RCTs in HF shows that it is not possible to make reliable therapeutic inferences from observational associations. While trials undoubtedly leave gaps in evidence and enrol selected participants, they clearly remain the best guide to the treatment of patients.


Randomized controlled trials (RCTs) are widely acknowledged to be the gold standard test of whether or not a drug is beneficial.[1–4] Although the biases and limitations of non-randomized, observational studies have been recognized for decades (Figure 1), studies of this type purporting to describe the effects of treatment continue to be published, even in high-impact journals.[5–10] Indeed, the 'comparative effectiveness' and 'big data' movements have given non-randomized studies a new respectability in some peoples' eyes.[11–13] Advocates point to the use of more sophisticated analytical techniques than in the past and increasingly larger 'real-world' datasets.[14–17] If the findings of observational studies could validly determine the effect of treatments, such information would clearly be of considerable value. On the other hand, if such analyses are inherently flawed they serve only to cause confusion, e.g. the association between hormone replacement therapy and decreased risk of coronary heart disease (CHD)[18,19] (Figure 2), and maybe worse, e.g. lead to discontinuation of effective therapy by physicians or patients misled by the findings.[20]

Figure 1.

Confounding in non-randomized studies.

Figure 2.

Examples of confounding in non-randomized studies.

There is a particularly strong evidence base for pharmacological treatments in heart failure (HF), making it an appropriate condition in which to compare treatment effects established in RCTs with those reported in non-randomized studies. We have, therefore, compared the conflicting results of non-randomized studies of the 'effect' of pharmacological treatments with those of RCTs using the same therapies for HF. Although many publications of this type have used the word 'effect', more correctly they have actually described associations between treatments and outcomes.