Diagnostic Accuracy of Virtual Cognitive Assessment and Testing

Systematic Review and Meta-analysis

Jennifer A. Watt MD PhD; Natasha E. Lane MD PhD; Areti Angeliki Veroniki PhD; Manav V. Vyas MBBS MSc; Chantal Williams MSc; Naveeta Ramkissoon MPH; Yuan Thompson PhD; Andrea C. Tricco PhD; Sharon E. Straus MD MSc; Zahra Goodarzi MD MSc

Disclosures

J Am Geriatr Soc. 2021;69(6):1429-1440.

Abstract and Introduction

Abstract

Background/Objectives: Virtual (i.e., telephone or videoconference) care was broadly implemented because of the COVID-19 pandemic. Our objectives were to compare the diagnostic accuracy of virtual to in-person cognitive assessments and tests and barriers to virtual cognitive assessment implementation.

Design: Systematic review and meta-analysis.

Setting: MEDLINE, EMBASE, CDSR, CENTRAL, PsycINFO, and gray literature (inception to April 1, 2020).

Participants and Interventions: Studies describing the accuracy or reliability of virtual compared with in-person cognitive assessments (i.e., reference standard) for diagnosing dementia or mild cognitive impairment (MCI), identifying virtual cognitive test cutoffs suggestive of dementia or MCI, or describing correlations between virtual and in-person cognitive test scores in adults.

Measurements: Reviewer pairs independently conducted study screening, data abstraction, and risk of bias appraisal.

Results: Our systematic review included 121 studies (15,832 patients). Two studies demonstrated that virtual cognitive assessments could diagnose dementia with good reliability compared with in-person cognitive assessments: weighted kappa 0.51 (95% confidence interval [CI] 0.41–0.62) and 0.63 (95% CI 0.4–0.9), respectively. Videoconference-based cognitive assessments were 100% sensitive and specific for diagnosing dementia compared with in-person cognitive assessments in a third study. No studies compared telephone with in-person cognitive assessment accuracy. The Telephone Interview for Cognitive Status (TICS; maximum score 41) and modified TICS (maximum score 50) were the only virtual cognitive tests compared with in-person cognitive assessments in >2 studies with extractable data for meta-analysis. The optimal TICS cutoff suggestive of dementia ranged from 22 to 33, but it was 28 or 30 when testing was conducted in English (10 studies; 1673 patients). Optimal modified TICS cutoffs suggestive of MCI ranged from 28 to 31 (3 studies; 525 patients). Sensory impairment was the most often voiced condition affecting assessment.

Conclusion: Although there is substantial evidence supporting virtual cognitive assessment and testing, we identified critical gaps in diagnostic certainty.

Introduction

In an era of physical distancing measures to prevent the spread of COVID-19, we are encouraged by health authorities to facilitate virtual medical encounters (i.e., telephone or videoconference) for older adults. As physical distancing restrictions relax, some patients may return to their doctor's office for in-person assessments. Others will receive ongoing care virtually: many believe the popularity of virtual medical care will continue.^[1] Cognitive assessments (e.g., based on Diagnostic and Statistical Manual of Mental Disorders [DSM] criteria for dementia and mild cognitive impairment [MCI]), which consist of identifying cognitive domain deficits (e.g., delayed recall, visuospatial skills) and associated functional impairments, have always been a critical component of in-person assessments for older adults. As part of a cognitive assessment, cognitive testing (e.g., Montreal Cognitive Assessment [MoCA], Telephone Interview for Cognitive Status [TICS]) can support a diagnosis of MCI or dementia through standardized identification of cognitive domain deficits and enable clinicians to develop holistic treatment plans that help older adults to complete basic and instrumental activities of daily living.^[2–5] Now, cognitive assessments and cognitive testing will continue to play an important role in virtual assessments, but our approach must be adapted to the virtual environment.

The recent rapid adoption of virtual cognitive assessments into clinical practice has created an immediate need to better understand the accuracy of virtual cognitive assessments and testing. If we are to continue providing evidence-informed care for older adults in the COVID-19 era, clinicians caring for older adults must become skilled at conducting virtual cognitive assessments and testing. We completed a systematic review of diagnostic accuracy studies to fill this critical knowledge gap. In our systematic review, we describe the accuracy and reliability of virtual compared with in-person cognitive assessments for diagnosing dementia or MCI, virtual cognitive test cutoffs suggestive of dementia or MCI compared with an in-person cognitive assessment, and correlations between virtual and in-person cognitive test scores. To enable rapid implementation of virtual cognitive assessment and testing into routine clinical practice, we synthesized barriers and facilitators to their use as described by study authors of included studies.

1 2 3 4

Next Section

Abstract and Introduction
Methods
Results
Discussion
References
Sidebar

Table 1. Characteristics of included studies
Study characteristic	Number of studies (%)
Mean age
≤60	16 (14.0)
61–69	23 (19.8)
70–79	62 (51.2)
≥80	11 (9.1)
Not reported	7 (0.6)
Proportion females
≥50%	69 (57.0)
<50%	38 (31.4)
Not reported	14 (11.6)
Degree of cognitive impairment
No cognitive impairment	12 (9.9)
Mixed (no cognitive impairment, MCI, dementia)	73 (60.3)
MCI/dementia	2 (1.7)
MCI	1 (0.8)
Dementia	6 (5)
Other	5 (4.1)
Not reported	22 (18.2)
Most commonly reported virtual cognitive tests
TICS or modified TICS	41 (33.9)
MMSE or modified MMSE	22 (18.2)
MoCA or modified MoCA	9 (7.4)
Language of cognitive assessment or cognitive testing
English	28 (23.1)
Other^a	36 (30.6)
Not reported	56 (46.3)
Modality of virtual cognitive testing
Telephone	85 (69.4)
Videoconference	37 (30.6)

Abbreviations: MCI, mild cognitive impairment; MMSE, Mini Mental State Examination; MoCA, Montreal Cognitive Assessment; TICS, telephone interview for cognitive status.
^aOther: Chinese (number [n] = 5), Dutch (n = 3), English or Spanish (n = 2), French (n = 2), Finnish (n = 1), German (n = 5), Hebrew (n = 1), Italian (n = 4), Japanese (n = 3), Korean (n = 2), Portuguese (n = 4), Spanish (n = 4), Turkish (n = 1).

Table 2. Diagnostic accuracy of TICS and modified TICS
Cutoff	Pooled sensitivity at cutoff (95% CI)	Pooled specificity at cutoff (95% CI)	Positive likelihood ratio	Negative likelihood ratio
DEMENTIA
TICS (maximum score of 41): 10 studies (1673 participants) ^{5, 19–27}
25	0.76 (0.56 to 0.90)	0.93 (0.79 to 0.99)	10.9	0.3
26*	0.80 (0.61 to 0.92)	0.90 (0.72 to 0.98)	8	0.2
27	0.84 (0.67 to 0.94)	0.85 (0.64 to 0.96)	5.6	0.2
28	0.87 (0.72 to 0.96)	0.80 (0.55 to 0.94)	4.4	0.2
29	0.90 (0.76 to 0.97)	0.72 (0.46 to 0.90)	3.2	0.1
30	0.92 (0.80 to 0.98)	0.64 (0.37 to 0.86)	2.6	0.1
Modified TICS (maximum score of 50): 3 studies (435 participants) ^{25, 28, 29}
25	0.80 (0.53 to 0.95)	0.93 (0.75 to 0.99)	11.4	0.2
26*	0.85 (0.61 to 0.96)	0.89 (0.68 to 0.98)	7.7	0.2
27	0.89 (0.68 to 0.98)	0.85 (0.61 to 0.96)	5.9	0.1
28	0.92 (0.74 to 0.99)	0.79 (0.53 to 0.94)	4.4	0.1
29	0.94 (0.79 to 0.99)	0.73 (0.44 to 0.92)	3.5	0.1
30	0.96 (0.84 to 0.99)	0.66 (0.36 to 0.88)	2.8	0.1
MILD COGNITIVE IMPAIRMENT
Modified TICS (maximum score of 50): 3 studies (525 participants) ^{25, 28, 30}
25	0.26 (0.12 to 0.46)	0.95 (0.88 to 0.99)	5.2	0.8
26	0.36 (0.18 to 0.58)	0.93 (0.82 to 0.98)	5.1	0.7
27	0.47 (0.26 to 0.69)	0.89 (0.74 to 0.96)	4.3	0.6
28	0.58 (0.35 to 0.78)	0.83 (0.66 to 0.94)	3.4	0.5
29	0.68 (0.48 to 0.86)	0.77 (0.56 to 0.90)	3	0.4
30*	0.77 (0.55 to 0.91)	0.68 (0.45 to 0.86)	2.4	0.3