Diagnostic Accuracy of Virtual Cognitive Assessment and Testing

Systematic Review and Meta-analysis

Jennifer A. Watt MD PhD; Natasha E. Lane MD PhD; Areti Angeliki Veroniki PhD; Manav V. Vyas MBBS MSc; Chantal Williams MSc; Naveeta Ramkissoon MPH; Yuan Thompson PhD; Andrea C. Tricco PhD; Sharon E. Straus MD MSc; Zahra Goodarzi MD MSc

Disclosures

J Am Geriatr Soc. 2021;69(6):1429-1440. 

In This Article

Results

Characteristics of Included Studies

We screened 9200 article titles and abstracts and 417 full-text articles, which resulted in 121 included studies (15,832 participants) (Figure 1). We identified 84 virtual cognitive tests (some of which have different versions or contain subtests). Most studies implemented virtual cognitive testing via telephone (69.4%) (Table 1 and Table S1). The TICS or modified TICS were the most utilized virtual cognitive tests (reported in 33.9% of studies) (Table 1). Virtual cognitive tests were studied in at least 13 languages (Table 1). Sixty-nine (57.0%) of the studies reported correlations between virtual and in-person cognitive test scores (Table S2). Risk of bias from patient flow through the study and timing of reference and index tests was high in 46.3% of studies and represented the greatest threat to study validity (Table S3 and Figure S1).

Figure 1.

PRISMA flow diagram

Accuracy and Reliability of Virtual Compared With In-person Cognitive Assessment for Diagnosing Dementia

We found three studies describing the accuracy or reliability of videoconference compared with in-person cognitive assessments based on DSM-IV criteria.[16–18] In a prospective cohort study of 16 Washington State Veterans' Home residents who did not have a history of dementia, geriatric psychiatrists had 100% agreement (i.e., 100% sensitivity and specificity because the in-person assessment was the reference standard) between videoconference and in-person assessments in diagnosing 12 residents with dementia (type not specified).[16] However, this study was conducted in a highly selected patient population: residents had a high probability of having dementia based on the 7-Minute Screen.[16] Further, all residents had a MMSE completed prior to in-person and videoconference cognitive assessments, and MMSE results were reviewed by the geriatric psychiatrists. In a prospective study of older adults (aged 50 years and older) with undiagnosed cognitive problems who were referred to a memory disorder clinic, videoconference-based cognitive assessments demonstrated good reliability compared with in-person cognitive assessments when completed by specialist physicians experienced in diagnosing dementia (weighted kappa 0.51, 95% CI 0.41–0.62).[17] Further, in a separate study comparing reliability between videoconference and in-person assessments for diagnosing dementia (type not specified) in a cohort of older adults (aged 50 years and older) referred to a memory disorder clinic, Martin-Khan et al., reported a weighted kappa of 0.63 (95% CI 0.4–0.9) between videoconference and in-person assessments conducted by a geriatrician and psycho-geriatrician.[18] Both geriatricians were provided with cognitive testing results before their assessments, which were completed by a clinic nurse. However, it was unclear if these two geriatricians were blinded to each other's findings.[18] We did not find any studies comparing the diagnostic accuracy of a telephone compared with in-person cognitive assessment based on DSM criteria.

Virtual Cognitive Test Cut-offs Suggestive of Dementia or Mild Cognitive Impairment and Correlations Between Virtual and In-person Cognitive Test Scores

We describe the three most reported virtual cognitive tests in our systematic review: the TICS, MMSE, and MoCA (Table 1). Only the TICS (maximum score of 41) and modified TICS (maximum score of 50) were compared with a reference standard in >2 studies and had extractable effect estimates that could be meta-analyzed (study-specific estimates of diagnostic accuracy for all cognitive tests are found in Table S2).

Telephone Interview for Cognitive Status (TICS)

We included 10 studies (1673 participants) in our meta-analysis to derive the optimal TICS (maximum score of 41) cutoff value supportive of a diagnosis of dementia (Table 2).[5,19–27] The optimal TICS cutoff suggestive of dementia ranged from 22 to 33, but it was 28 or 30 when the TICS was conducted in English. At the optimal cutoff value of 26, the pooled sensitivity was 0.80 (95% CI 0.61–0.92), the pooled specificity was 0.90 (95% CI 0.72–0.98), and the AUC was 0.92 (Figure 2(A)). In a subgroup of studies where cognitive testing was conducted in English, the optimal cutoff value was 29; the pooled sensitivity at this cutoff value was 0.72 (95% CI 0.40–0.92), the pooled specificity was 0.85 (95% CI 0.62–0.96), and AUC was 0.86. In a subgroup of studies where the mean years of formal education was >8, the optimal cutoff value was 27; the pooled sensitivity at this cutoff value was 0.83 (95% CI 0.58–0.96), the pooled specificity at this cutoff value was 0.92 (95% CI 0.67–0.99), and AUC was 0.94. In a sensitivity analysis where we removed studies at high risk of bias from the conduct or interpretation of the reference standard, the optimal cutoff value was 25; the pooled sensitivity at this cutoff value was 0.78 (95% CI 0.59–0.91), the pooled specificity at this cutoff value was 0.92 (95% CI 0.75–0.98), and AUC was 0.92. Two studies reported cutoffs for the TICS supportive of a diagnosis of MCI: Manly et al. determined the optimal cutoff was 29 and Seo et al. determined the optimal cutoff was 28.[23,25]

Figure 2.

Summary receiver operating characteristic curves for telephone interview for cognitive status (TICS) and modified TICS. Panel A is the summary receiver operating characteristic curve (SROC) associated with the optimal cutoff threshold for the Telephone Interview for Cognitive Status (TICS) (maximum score 41) suggestive of dementia. Panel B is the SROC associated with the optimal cutoff threshold for the modified TICS (maximum score 50) suggestive of dementia. Panel C is the SROC associated with the optimal cutoff threshold for the modified TICS suggestive of mild cognitive impairment. SROC curves are indicated by the solid line. The optimal cutoff threshold is indicated by the '+'. Individual data points from each study are indicated by uniquely colored circles. The 95% confidence interval (CI) for sensitivity, given specificity, is indicated by the solid gray line. The 95% CI for specificity, given sensitivity, is indicated by the interrupted gray line

Modified Telephone Interview for Cognitive Status (TICS)

We included three studies (435 participants) in our meta-analysis to derive the optimal modified TICS (maximum score of 50) cutoff value supportive of a diagnosis of dementia (Table 2).[25,28,29] At the optimal cutoff value of 27, the pooled sensitivity was 0.85 (95% CI 0.61–0.96), the pooled specificity was 0.89 (95% CI 0.68–0.98), and the AUC was 0.94 (Figure 2(B)). We included three studies (525 participants) in our meta-analysis to derive the optimal modified TICS cutoff value supportive of a diagnosis of MCI (Table 2).[25,28,30] At the optimal cutoff value of 30, the pooled sensitivity was 0.77 (95% CI 0.55–0.91), the pooled specificity was 0.68 (95% CI 0.45–0.86), and the AUC was 0.80 (Figure 2(C)).

Mini-mental State Examination (MMSE)

Two studies reported sensitivity and specificity values, compared with a reference standard, for modified versions of the Mini-Mental State Examination (MMSE) supportive of a diagnosis of dementia (Table S2).[31,32] At a cutoff of 13 (maximum possible score 22), the sensitivity of the MMSE conducted on the telephone in Brazilian Portuguese for diagnosing dementia was 0.9 and specificity 1.[31] At a cutoff of 13 (maximum possible score 26), the sensitivity of the MMSE conducted on the telephone in Cantonese for diagnosing dementia was 1 and specificity was 0.67.[32] One study reported sensitivity and specificity values, compared with a reference standard, for a modified version of the MMSE supportive of with a diagnosis of MCI: at a cutoff of 89 (maximum possible score 100), the sensitivity of the Telephone Modified MMSE in German was 0.83 and specificity was 1.[33] Pearson correlations for telephone compared with in-person MMSE testing ranged from 0.54 to 0.88.[34–36] Pearson correlations for videoconference compared with in-person MMSE testing ranged from 0.89 to 0.92.[37–39]

Montreal-cognitive Assessment (MoCA)

Two studies reported sensitivity and specificity values, compared with a reference standard, for modified versions of the MoCA supportive of a diagnosis of MCI (Table S2).[40,41] At a cutoff of 7 (maximum possible score 12), the sensitivity of the short version of the telephone MoCA for diagnosing MCI was 0.41 and specificity was 0.88.[40] At a cutoff value of 20 (maximum possible score 22), the sensitivity of the MoCA conducted on the telephone in German was 0.9 and specificity was 0.54.[41] Intraclass correlation coefficients for videoconference compared with in-person MoCA testing ranged from 0.59 to 0.99.[42–45]

Barriers and Facilitators to Virtual Cognitive Assessment and Resting

We derived 56 unique codes and seven major themes in our meta-synthesis of barriers and facilitators associated with virtual cognitive assessment and virtual cognitive testing (Figure 3 and File S3). We derived codes describing barriers and facilitators associated with access to (i.e., not all patients have the required hardware or technology, remote assessment is beneficial to the patient due to lack of travel and cost savings) and implementation of (i.e., clinician pronunciation is important for successful assessment, remote assessment depends heavily on sound and image quality) virtual cognitive assessment and cognitive testing (Figure 3). We identified patient symptoms or conditions affecting assessment, patient and care partner experiences, health care provider factors, cognitive test factors, technology factors, language factors, and presence of persons who could support virtual care as major themes. Sensory (i.e., visual or hearing) impairment was the most often voiced patient symptom or condition affecting assessment by patients and clinicians. Overall, patients reported a high degree of acceptance and comfort associated with virtual care. However, a lack of access to and familiarity with technology required for videoconference-based assessments caused frustration and fatigue for patients. To overcome this barrier, some studies had care partners or support staff setup necessary videoconferencing equipment.

Figure 3.

Synthesis of barriers and facilitators to virtual cognitive assessment and virtual cognitive testing

Comments

3090D553-9492-4563-8681-AD288FA52ACE

processing....