A Nationwide Flash-Mob Study for Suspected Acute Coronary Syndrome

Angel M.R. Schols, MD, PhD; Robert T. A. Willemsen, MD, PhD; Tobias N. Bonten, MD, PhD; Martijn H. Rutten, MD; Patricia M. Stassen, MD, PhD; Bas L. J. H. Kietselaer, MD, PhD; Geert-Jan Dinant, MD, PhD; Jochen W.L. Cals, MD, PhD


Ann Fam Med. 2019;17(4):296-303. 

In This Article


This was the first nationwide flash-mob study in family medicine, which serves as a proof of concept for large-scale clinical diagnostic research among FPs within a short time frame. We showed that flash-mob research in family medicine is feasible by collecting data on 258 patients with suspected ACS in only 2 weeks. We were able to involve and motivate many organizations and FPs by mobilizing nearly 1 in 5 FPs (approximately 1,800 FPs) throughout the country via ambassadors (see Figure 1 and the Supplemental Appendix, http://www.AnnFamMed.org/content/17/4/296/suppl/DC1/, for more information). Recruiting this number of FPs in the context of traditional research would have been extremely challenging and both resource and time consuming. Prior studies on signs and symptoms and clinical decision rules in similar populations (including nonreferred patients, allowing for a larger group eligible for recruitment) showed inclusion rates of less than 30 to 40 patients per week.[1–3,18–20] In addition, traditional studies often have a limited number of participating physicians, which may lead to clustering of data collected by individual physicians. The large number of different FPs and the widespread inclusion of patients scattered throughout the country strengthens the external validity of the present study. Moreover, FPs did not receive additional instructions or training; thus, the study conditions reflected aspects of daily practice.

Strengths and Limitations

Some selection bias might have occurred given the fact that very ill patients might not have been included in this study. For these severe cases, however, the use of a clinical decision rule would not be advised. We asked FPs to only register referred patients to allow for the determination of a final diagnosis based on assessment in secondary care, in contrast to prior studies that also included nonreferred patients using a reference standard based on clinical follow-up and thereby risking diagnostic misclassification.[1–3,7] Cardiologists did not perform a uniform assessment, yet all patients received a standard diagnostic work-up according to current clinical practice. We did not provide an exact definition of myocardial infarction or unstable angina but asked FPs to base their answer on the final diagnosis in the cardiologist's final report and to allow for a follow-up of at least 6 weeks. Cardiologists were blinded with regard to the MHS results, and FPs were not asked to calculate MHS values. A substantial proportion of FPs, however, did fill out the CRF as well as the final diagnosis form. The FP's assessment was registered at the end of the CRF and might therefore have been slightly biased by the preceding items. The significant association between ischemic changes on ECG and ACS was influenced by incorporation bias because ECG abnormalities are part of the reference standard. Although MHS values could only be determined for 186 patients owing to missing items, sensitivity analyses showed only minor differences in test characteristics after imputation. Even though one-half of patients without an MHS had no chest pain, we allowed them to be included in an attempt to develop an adapted clinical decision rule for all referred patients with suspected ACS. We did not specifically investigate clinically relevant conditions other than ACS. There is an ongoing debate as to whether unstable angina is a justified diagnosis in the setting of high-sensitivity troponin assays, yet FPs do not have troponin assays available at the point of care, and unstable angina is still mentioned as part of ACS in the Dutch FP guideline.[21,22]

Comparison With Existing Literature

In contrast to prior studies of the MHS, we found an insufficient diagnostic accuracy of the MHS to safely rule out ACS at a cut-off value of ≤2 (NPV 88.0% compared to 97.7% to 98.1% in prior studies).[1–3] This difference might be explained by spectrum bias because the present study was performed with a referred population with a greater incidence of ACS (19.4%) compared to prior MHS studies (3.7% and 2.5%).[1–3,23] In addition, in those prior studies, a lower-risk population was represented because patients with stable coronary artery disease were included along with patients with unstable presentations of chest pain. This explanation is in line with our present finding that the MHS could accurately rule out ACS in the subgroup that was estimated to be at low risk according to the FP and is comparable with the finding of a prior study in a primary care population with an ACS incidence of 22%.[8] That study also found that a clinical decision rule could be used to safely rule out ACS in patients considered to be at low risk according to the FP's assessment. The overall AUC we found for FP assessment was also in line with that study, which showed an AUC for FP risk estimation of 0.75 (95% CI, 0.68–0.82) compared to 0.72 (95% CI, 0.63–0.81) in the present study.[8]

Implications for Clinical Practice and Future Research

The MHS and FP assessments individually showed insufficient diagnostic accuracy to safely rule out ACS in referred patients. When combined, however, they safely reduced the number of referrals by 19% by applying the MHS only for referred patients considered to be at low risk of having an ACS by the FP's assessment. Yet, such a strategy meets practical limitations. FPs would have to apply this strategy after they made the decision to refer, meaning FPs should incidentally correct their decision to refer'. Therefore, the suggested strategy should be validated in a sufficiently large cohort including both referred and nonreferred patients with suspected ACS.

Although we consider flash-mob research in family medicine feasible and that it may be considered for use as a new research method, it should be noted that the flash-mob method is not suitable for all diagnostic research in family medicine. It should only be used if (1) the research question can be answered with a small data set per patient, (2) the research question is relatively simple and is considered to be relevant and urgent according to FPs, (3) the patient selection and data collection are sufficiently robust and self-explanatory such that very few instructions are necessary, and (4) the flash-mob study can be performed in a large region to include patients among a large number of FPs.