Studies Show Cardiovascular Devices Often Earn FDA Approval Without High-Quality Clinical Data

Reed Miller

December 30, 2009

December 30, 2009 (San Francisco, CA and Boston, MA) - Two separate reviews of cardiovascular device premarket approval (PMA) applications to the FDA identify weaknesses in many of the studies that supported the PMAs, but the studies’ authors appear to disagree on the definition of a "high-quality clinical trial" [1,2].

In the December 30, 2009 issue of the Journal of the American Medical Association (JAMA),Dr Sanket Dhruva (University of California, San Francisco) and colleagues report their findings from a systematic review of the 123 publicly available summaries of safety and effectiveness data (SSED) for 78 PMAs for high-risk cardiovascular devices that the FDA approved between January 2000 and December 2007.

"The findings in this study raise questions about the quality of data on which some cardiovascular device approvals are based. There is a balance between getting new drugs and devices to market quickly and ensuring the evidence of benefit is sufficient before FDA approval and marketing," Dhruva et al say. "However, the bar for evidence of benefit should be higher for devices because they are implanted and cannot simply be discontinued, as drugs can. In addition, although devices can be lifesaving, they also have great potential for risk and adverse events."

About 27% of the 123 studies reviewed by Dhruva et al were randomized, and 14% were blinded. Just over half of the total end points measured in the studies included a control comparison, and 31% of the controls were retrospective. About 88% of the end points were surrogate measures of effectiveness, and 78% of the studies showed a discrepancy between the number of patients enrolled in the study and the number analyzed for the primary outcomes. Only 80% of the 123 SSED summaries report the number of participants enrolled in the study. The average number enrolled was 308 patients.

The authors highlight that 65% of the PMAs were approved based on a single clinical study. The average number of studies listed in the SSED summaries for each PMA was 1.6, with a range from one to five.

Characteristics of 78 PMA studies reviewed

Characteristic of PMA n %
At least 1 randomized study 24 31
At least 1 blinded study 10 13
No studies with more than 1 US center 28 36
At least 1 primary end point reported 71 91

Dhruva et al identify several potential sources of bias in these studies. Retrospectively selected controls or post hoc analysis may allow for the selection of control groups or end points that favor the device. Surrogate end points may not be reliable predictors of actual patient benefit if they are not linked to a clinically meaningful end point. Composite outcomes, which are also common in cardiovascular trials, have been shown to comprise individual end points that often vary in clinical significance and do not contribute equally to the composite measure, they point out. Also, a discrepancy between the number of enrolled patients and the number analyzed for the primary end point raises the possibility that the study has been biased because patients with less favorable outcomes were lost to follow-up.

PMAs represent the FDA's most rigorous device approval process for devices that are often necessary for sustaining a patient's life but also carry the highest risk because they are usually implantable and often permanent. Yet, "although there has been recent scrutiny of evidence used in the US Food and Drug Administration drug approval process, less attention has been paid to the approval process for medical devices. Medical devices are less likely than drugs to have demonstrated clinical safety before they are marketed," Dhruva et al argue.

Separately, Dr William Maisel (Beth Israel Deaconess Medical Center, Boston, MA), director of the Medical Device Safety Institute, and scientists at the FDA's Division of Cardiovascular Devices, Center for Devices and Radiological Health reviewed all of the original cardiovascular device PMAs that the FDA ruled on between January 1, 2000 and December 31, 2007 to assess the quality of clinical investigations submitted by manufacturers. Results of their study were published online December 29, 2009 in the American Journal of Therapeutics.

Unlike the study by Dhurva et al, which was based on only the publicly available SSEDs, Maisel and the FDA researchers had access to the entire submission file for each PMA.

In total, the researchers evaluated 88 cardiovascular device PMAs--77.3% of which were for permanent implants--encompassing 132 clinical studies with a total of 37 328 study subjects and 29 408 device recipients. The average age of the patients was 61, just over one-third of the subjects were women, and over 86% were white.

For this study, Maisel and colleagues deemed the effectiveness and safety end points to be "high quality" if they were clearly defined and associated with a specific time point for analysis. High-quality subject accounting was defined as at least 90% of the original cohort accounted for at study conclusion.

The researchers deemed the primary effectiveness end points, primary safety end points, and subject accounting to be high quality in 81.8%, 60.2%, and 77.3% of pivotal studies, respectively. They note that key cardiovascular comorbidities were infrequently reported. For example, coronary artery disease was reported in about 51% of studies, diabetes was reported in 37%, and tobacco use was reported in only 32%. The patients' race was reported in only about 15% of cases.

The authors also broke out their results by type of cardiovascular device--interventional, electrophysiological, or surgical--but caution that there are a wide variety of devices and study designs within each group.

Data quality by device type, n (%)

Measure Intervention Electrophysiology Surgery Total p
Number of PMAs 42 29 17 88
High-quality primary effectiveness end point 37 (88.1) 27 (93.1) 8 (47.1) 72 (81.8) <0.05b,c
High-quality primary safety end point 24 (57.1) 23 (79.3) 6 (35.3) 53 (60.2) <0.05b
High-quality primary safety and effectiveness end points 21 (50) 23 (79.3) 6 (35.3) 50 (56.8) <0.05a,b
High-quality subject accounting 37 (88.1) 20 (69) 11 (64.7) 68 (77.3) <0.05a,c

a. comparison of interventional vs electrophysiology (EP) groups
b. comparison of EP vs surgery groups
c. comparison of interventional vs surgery groups

What makes a trial "high quality?"

Maisel told heartwire that he believes the authors of the study in JAMA appear to be inappropriately applying the definitions of quality for drug studies to device studies. "Devices are not drugs. There is no one size fits all.

"It's hard to generally characterize devices and lump them all together. So when we were thinking about how to measure quality, we tried to pick things that we felt should apply to every device no matter what type of device," he explained. "That's a little bit different from the JAMA article, where they talk about things like randomization and blinding and number of studies [as indicators of quality]. We thought about those things, but we didn't think that any of them [should be] applied ubiquitously to all devices."

The relatively low percentage of surgical-devices trials with high-quality end points underlines the difficulty of studying this type of device, Maisel said. They are particularly challenging to study because it is difficult to randomize patients to surgery vs nonsurgery and/or blind the patients or physicians. Surgical devices are "a great example of how randomization and blinding aren't the measure by which quality should be measured. Because, for surgical studies, it's just extremely difficult to do that. It's impractical and in some cases, unethical."

But that's not necessarily an excuse for researchers of surgical devices to collect only poor-quality data, Maisel said. "There's no reason that, prior to conducting the study, you couldn't predefine the effectiveness and safety end points and pick the time at which they're going to be studied. So despite the challenges and the diversity, we think the quality measures that we picked should apply equally to surgical devices."

Maisel also pointed out that, compared with drugs, a lot more useful information can be collected in nonclinical trials. "There are things that you can learn about devices by doing engineering studies or bench testing. Sometimes you can learn more about them on the bench than in real life. Examples would be repetitive-motion testing . . . or for devices that need to convey electrical current . . . you can push them past the extreme to test their limits. Device approval is based on more than just the clinical trials."

In a set of "talking points" provided to heartwire by FDA spokesperson Meghan Scott, the agency acknowledges that the study by Maisel and the FDA scientists "identified several problems with clinical trials in support of cardiovascular device PMAs, including: suboptimally defined safety and effectiveness end points, poor patient accounting, and incomplete collection of important patient comorbidities [that] make device safety and effectiveness assessments more challenging."

However, the FDA makes many of the same criticisms of the Dhruva study that Maisel raised. According to the FDA, Dhruva et al "did not fully consider the feasibility of device clinical trials as well as the totality of the evidence available to the agency in PMA submissions. Congress has recognized that devices are not drugs and that device trials require consideration of unique design considerations. The standard drug metrics used to assess quality need to be modified appropriately for assessment of clinical-trial quality for devices."

The FDA argues that "the Dhruva study assumes that the length of the study is associated with the quality of the data. However, the key issues are the clinical-trial question that needs to be assessed and the optimal time point for assessment." For example, vascular-closure devices that that seal the femoral-artery puncture site need to do their job almost immediately to prevent major bleeding, so a short follow-up period is the appropriate study time frame, not a weakness of the study.

The FDA also rejects Dhruva et al's contention that measuring surrogate instead of clinical end points necessarily undermines the validity of trial results. "Appropriate use of valid surrogate and composite end points allows trials to be of a reasonable size and duration such that safe and effective medical devices can reach patients and providers in a timely way." For example, a trial of a device in a relatively low-risk population that specified death as the primary end point would need so many trial subjects to achieve statistical significance that "timely completion of the study with results relevant to patient care and before the device would likely become obsolete would not be feasible." The FDA also points out that it already evaluates composite end points to ensure that each component of the composite is supportive of the overall finding based on the composite.

Summaries of safety and effectiveness appear to be lacking

The FDA also argues that the study by Maisel and the FDA scientists provides a more accurate picture of the quality of studies reviewed by the FDA because it is based on the full PMA filings and not just the SSED summaries. According to the agency, SSED summaries "reflect the intensive review performed by the FDA lead reviewer, statistician, clinician, and engineer. However, in no way can it be used as a 'surrogate' for examination of the actual reviews."

One of the coinvestigators of the Dhruva study, Dr Rita Redberg (University of California, San Francisco), told heartwire that her group relied on the SSED summaries because FDA's own guidelines for creating SSED summaries state that they are "intended to present a reasoned, objective, and balanced critique of the evidence that served as the basis of the decision to approve or deny the PMA."

"It is highly unlikely that a trial would have been randomized or blinded, for example, and that fact would not have been included in the SSED," Redberg points out. However, she said that she and her colleagues certainly support greater access to complete information from FDA reviews.

"SSEDs are the only FDA-reviewed evidence availablefor clinicians, and they form the sole basis of data that canbe used for systematic reviews and guideline development. Thisstudy reinforces the need for improved access to complete FDAreviews for both pharmaceutical and device data," Dhruva et al argue.

FDA commits to continuous improvements

FDA says it is implementing several recommendations based on the study by Maisel and colleagues. In the future, study end points for safety and effectiveness must be clearly defined in the PMA, with a specific statement regarding the time point at which they will be evaluated and the relative contribution of each component of a composite end point. Also, PMA sponsors will be required to clearly and accurately detail their patient accounting practices in order to ensure interpretability of study results, according to the agency.

Redberg agrees that "clear and consistent guidelines on the strength of evidence--type of primary end point, number of patients, length of follow-up, and study design, such as randomized or blinded, would all help to strengthen the evidence base on which the FDA approves these important devices."

In addition to committing to supporting more broad evaluations of clinical studies that support PMAs, the FDA is working with stakeholders to increase participation of women, children, and minorities in clinical trials. For example, the agency has hosted workshops with academic physicians, industry, and other experts to discuss strategies for incorporation of gender-specific analyses in PMA applications, and the FDA is currently developing a guidance document on increasing enrollment of women in clinical trials. An example of its effort to improve research of devices for children is the planned workshop on pediatric heart-valve development scheduled for January 12. The FDA is also participating in the Duke sudden cardiac arrest think tank that directly addresses enrollment of minorities in sudden-cardiac-death trials.