Detecting Disengagement From HIV Care Before It Is Too Late

Development and Preliminary Validation of a Novel Index of Engagement in HIV Care

Mallory O. Johnson, PhD; Torsten B. Neilands, PhD; Kimberly A. Koester, PhD; Troy Wood, MA; John A. Sauceda, PhD; Samantha E. Dilworth, MS; Michael J. Mugavero, MD; Heidi M. Crane, MD; Rob J. Fredericksen, PhD; Kenneth H. Mayer, MD; William C. Mathews, MD; Richard D. Moore, MD; Sonia Napravnik, PhD; Katerina A. Christopoulos, MD, MPH


J Acquir Immune Defic Syndr. 2019;81(2):145-152. 

In This Article



We present the HIV Index of Engagement, or HIV Index for short, which is a 10-item, unidimensional scale that was developed using an iterative process of online consensus building, the Delphi process,[12] and focus groups with patients in 3 US cities. Items were tested through cognitive interviews and then validated with patients from 7 HIV clinics affiliated with the Centers for AIDS Research Network of Integrated Clinical Systems (CNICS).[13,14]

Item Development

The procedures for the Delphi and focus group methods are described in detail elsewhere.[12,15] The Delphi methodology comprised 4 iterative rounds of online surveys administered to 66 experts in 3 categories: HIV clinical care providers, engagement in HIV care researchers, and researchers focused on engagement in care and clinicians working in non-HIV fields. Twelve focus group discussions with patients in San Francisco, CA, Birmingham, AL, and Seattle, WA were conducted between Delphi rounds, with findings fed back into subsequent data collection. Inquiries started with broad, open-ended questions, such as "How would you describe patients who are well engaged in health care?" responses to which led to a distinction between historical components of engagement in care (ie, appointment attendance, adherence to treatment, and optimal clinical outcomes) and the patient-centered perceptual and experiential aspects of care. This, in turn, resulted in a working definition of engagement in care to capture the patient-centered facets to describe someone who is "fully engaged in care": "Engagement in care is the ongoing interaction of patients, their providers, and care settings that is characterized by a patient's sense of connection to and active participation in care." This structure promoted the generation of a large number of topics related to the broader construct of full engagement, which were subsequently ranked for importance by the Delphi panelists. The research team then drafted candidate index items. Of note is that one item, "My provider really understands me as a person," came directly from prior published work in the area of HIV treatment adherence.[16]

Analysis and discussion resulted in refinement, combination, and elimination of some items based on the following criteria: (1) Candidate items were retained if they were universally applicable to all patients; we removed items that were relevant only to subsets of patients, such as topics related to unmet childcare needs and the need for substance abuse treatment, (2) items were removed if they were operationalized as correlates of engagement, for example, substance use, psychiatric symptoms, or outcomes of engagement (eg, ART adherence, retention in care, and virologic control), and (3) twenty-five cognitive interviews with patients in 3 clinics were conducted to assess clarity and interpretation of 21 candidate items. Items that were not consistently understood by patients were revised or eliminated. This process resulted in 13 items for subsequent factor analysis and validation.

Validation Procedures

From April 2016 to March 2017, the 13 candidate items were added to the clinical assessment of patient-reported measures and outcomes (PROs)[17–19] completed as part of routine clinical care visits at 7 academic HIV clinics affiliated with the CNICS. The inclusion criteria for a patient to be enrolled in the CNICS cohort are at least 18 years old, HIV-positive, and having at least 2 HIV primary care visits at a CNICS site. The assessment includes brief self-administered assessments of depression, ART adherence, alcohol and drug use, and internalized HIV stigma, among other areas. The CNICS data repository links data from the clinical assessment, electronic health record (EHR), and other sources, including demographic characteristics, laboratory, medication, and visit data. A priori comparisons were planned between scores on the new HIV Index and PRO, and EHR variables hypothesized to correlate with engagement in care.


Index of Engagement in HIV Care. The 13-item version of the HIV Index was administered to patients completing PRO assessments at each of the 7 sites.

Background and Demographic Variables. Patient characteristics were extracted from the CNICS data repository and included age, gender, race, ethnicity, sexual orientation, and time since enrollment in the CNICS cohort (Table 1).

Hypothesized Correlates of Engagement in Care. Relevant data were evaluated for associations with domains expected to be negatively related to scores on the HIV Index. Symptoms of depression and anxiety were operationalized as continuous indicators measured using the Patient Health Questionnaire-9 (PHQ-9).[20,21] Alcohol and stimulant use were assessed using the AUDIT-C and the ASSIST,[22–24] respectively, with alcohol use/dependence/abuse measured on a continuous scale and stimulant use as any reported cocaine, crack, or methamphetamine use in the past 3 months, in the distant past, or no prior use. Internalized HIV stigma was assessed with a 4-item version of a validated 6-item scale.[25]

Hypothesized Outcomes of Engagement in Care. There were 3 main categories of variables that were hypothesized to be related to the HIV Index: ART adherence, retention in HIV care, and HIV viral load.

HIV Medication Adherence: We coded each patient as currently taking ART or not as per self-report. Because treatment guidelines strongly support that all patients with HIV initiate ART,[26] we hypothesized that not being on ART would be related to lower HIV Index scores. Second, self-reported adherence to HIV medications for those on ART was hypothesized to be positively associated with HIV Index scores and was assessed using 2 validated measures. The HIV Adherence Rating Scale asks patients to rate their ability to take HIV medications as prescribed over the past 30 days,[27] with response choices from excellent to very poor. The Visual Analog Scale (VAS) asks from 0 to 100, of the amount they were supposed to take, and the proportion of prescribed medications taken over the past 30 days.[28]

Retention in HIV Care: We hypothesized that better retention in care would be related to higher HIV Index scores. As an indicator of retention in care, we constructed the following variables from clinic attendance data during the 18 months before administration of the HIV Index: (1) 2 or more missed visits in the past 12 months, (2) the proportion of kept primary care visits in the 12 months before HIV Index administration, and (3) a visit constancy measure based on whether the patient attended at least one HIV care visit in each of the 2 six-month intervals starting 180 days before the HIV Index administration. Note that this operationalization necessitated the restriction of analyses focused on retention to patients who had been in care at the site for at least the reporting period for each retention variable so that all cases had the opportunity for appointment attendance during a comparable retention window.

HIV Viral Load: Virologic suppression is a key outcome of HIV clinical care and hypothesized to be related to HIV Index scores. We used EHR records to identify the HIV viral load results closest in time to the collection of HIV Index data. Because viral load is not assessed at each clinical care visit, we included those patients with a test result within 90 days before or after the HIV Index administration. We defined suppressed viral load as fewer than 200 copies per milliliter.

Data Analysis

SAS 9.4[29] was used to generate frequency tables to characterize the sample and responses to the 13 HIV Index items and to perform the reliability, predictive validity, and convergent validity analyses described below. The sample was randomly split into 2 approximate halves, a factor extraction (ie, exploratory) subsample and a factor structure testing (ie, validation) subsample, stratified by a recruitment site to maintain balance by the site. In the factor extraction subsample, items were initially screened using FACTOR 10.8.03[30] to determine the number of factors to retain via the Hull method[31] and to examine items' unidimensionality via the explained common variance (ECV) and mean of residual absolute loadings (MIREAL) statistics.[32] Corresponding item-level statistics, I-ECV and I-REAL, were used to identify individual nonunidimensional items, which were dropped from further analyses.

Following item screening, exploratory factor analyses (EFAs) were performed on the remaining items in the factor extraction subsample. Then, confirmatory factor analyses (CFAs) were performed on the validation subsample. Global model fit was assessed using the χ2 test of exact fit and the following well-studied measures of approximate fit: the root mean square error of approximation (RMSEA), Bentler's Comparative Fit Index (CFI), and the standardized root mean square residual (SRMR). Satisfactory model-data fit was determined by: (1) RMSEA ≤ 0.06 and SRMR ≤ 0.08, or (2) CFI ≥ 0.95 and SRMR ≤ 0.08.[33]

"Don't know" and "not applicable" responses were treated as missing data, resulting in 25% of cases having missing values. As a sensitivity analysis, the EFA and CFA were repeated using 50 data sets with missing values imputed via multiple imputation (MI).[34] Mplus version 8.1 was used to perform EFAs and CFAs.[35]

Internal reliability for the HIV Index for the factor extraction and validation data subsets was assessed by computing Cronbach's[36] coefficient alpha. Predictive validity analyses were then performed on the entire sample to assess the associations of the HIV Index scale score with HIV detectable viremia, HIV care appointment attendance (ie, retention), and HIV medication adherence–related measures (eg, being on ART and adherence quality). Binary logistic regression was used for dichotomous clinical and adherence outcomes, ordinal logistic regression was used for the ordinal adherence rating scale, and the Spearman rank-order correlation coefficient was used for the continuous non-normally distributed proportion of HIV care visits attended variable. For the analysis of the adherence rating scale, the proportional odds assumption was tested. For all logistic regression analyses, the HIV Index scale score was rescaled in SD units so that odds ratios (ORs) reflected changes in the odds of each level of the outcome relative to the next lowest level of the outcome per SD increase in the Index score.

Finally, convergent and discriminant validity were assessed by correlating the HIV Index scores with the existing measures described above. These correlations were calculated using the Spearman rank-order method. As a sensitivity analysis, the correlations were recomputed based on 50 multiply-imputed data sets.