Differences in Performance Characteristics Among Four High-Throughput Assays for the Detection of Antibodies Against SARS-CoV-2 Using a Common Set of Patient Samples

David M. Manthei, MD, PhD; Jason F. Whalen; Lee F. Schroeder, MD, PhD; Anthony M. Sinay, C(ASCP); Shih-Hon Li, MD, PhD; Riccardo Valdez, MD; Donald A. Giacherio, PhD; Carmen Gherasim, PhD

Disclosures

Am J Clin Pathol. 2021;155(2):267-279.

Abstract and Introduction

Abstract

Objectives: Serologic testing for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has experienced a changing landscape of available assays coupled with uncertainty surrounding performance characteristics. Studies are needed to directly compare multiple commercially available assays.

Methods: Residual serum samples were identified based on SARS-CoV-2 reverse transcription polymerase chain reaction (RT-PCR) testing, clinical test results, and collection dates. Serum samples were analyzed using assays from four different manufacturers: DiaSorin anti–SARS-CoV-2 S1/S2 IgG, EUROIMMUN anti–SARS-CoV-2 IgG ELISA, Roche Elecsys anti–SARS-CoV-2, and Siemens SARS-CoV-2 Total antibody assays.

Results: Samples from SARS-CoV-2 RT-PCR–positive patients became increasingly positive as time from symptom onset increased. For patients with latest sample 14 or more days after symptom onset, sensitivities reached 93.1% to 96.6%, 98.3%, and 96.6% for EUROIMMUN, Roche, and Siemens assays, respectively, which were superior to the DiaSorin assay at 87.7%. The specificity of Roche and Siemens assays was 100% and superior to DiaSorin and EUROIMMUN assays, which ranged from 96.1% to 97.0% and 86.3% to 96.4%, respectively.

Conclusions: Laboratories should be aware of the advantages and limitations of serology testing options for SARS-CoV-2. The specificity and sensitivity achieved by the Roche and Siemens assays would be acceptable for testing in lower-prevalence regions and have the potential of orthogonal testing advantages if used in combination.

Introduction

Clinical laboratory serologic testing for antibodies directed against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which causes coronavirus disease 2019 (COVID-19), has been beset by a series of problems. Serologic tests were initially allowed to be distributed by any manufacturer, although some claims were lacking in quality, leading to removal from the US Food and Drug Administration (FDA) "notification list" of assays with pending or approved emergency use authorization.^[1] As more manufacturers with established in vitro diagnostics history have released serology assays with larger premarket studies, the presumption is that more consistent and high-quality assays are available. Many SARS-CoV-2 serology assay manufacturers report impressive performance results, which are summarized by the FDA based on review of submitted data.^[2] However, given a paucity of real-world and consistent evaluations of these assays, the FDA has partnered with the National Institutes of Health and Centers for Disease Control and Prevention (CDC) to perform limited independent evaluations of some assays.^[3] Although there are multiple evaluations of SARS-CoV-2 serology assays,^[4–22] there is a relative paucity of robust evaluations of more recently released tests. As such, there is need for greater direct comparisons across multiple, different platforms. Many laboratories have little real-word information about performance limitations when choosing among assays.

The best use of SARS-CoV-2 serologic testing remains an open question.^[23] Regardless of whether clinical or epidemiologic use is planned, the predictive values are important to consider, particularly in areas with low pretest probability (eg, mass screening). Both the throughput of an assay and in particular its specificity are crucial parameters for clinical laboratory implementation if large-scale serology testing is desirable in a low-prevalence environment.

Approaches for anti–SARS-CoV-2 immunoassays involve differing antigens (eg, nucleocapsid vs spike protein and full-length vs subdomains) and/or immunoglobulin isotypes (eg, immunoglobulin G [IgG], immunoglobulin M [IgM], total). A preferred combination may evolve depending on the desired purpose of testing as our understanding of COVID-19 matures, including breadth and duration of antibody responses, neutralizing effects of antibodies, and antigen choices for vaccines. It is possible that appropriate assays for detection of prior viral infection may be different from those to confirm a vaccine response.

Given the uncertain and shifting landscape for clinical laboratories, we sought to compare a common set of serum samples across four commercial assays as part of our evaluation for potential implementation of high-throughput testing. These findings will help inform the strengths and limitations of each assay in a directly comparable manner.

1 2 3 4

Next Section

Table 1. Characteristics of Commercial Anti–SARS-CoV-2 Assays Used in This Comparative Study
Manufacturer	Assay	Analyte	Capture Antigen^a	Method	Analyzer	Output	Cutoff Values
Manufacturer	Assay	Analyte	Capture Antigen^a	Method	Analyzer	Output	Negative	Indeterminate	Positive
EUROIMMUN	Anti–SARS-CoV-2 IgG ELISA	IgG	S1	ELISA	BioTek 800 TS	Ratio to a single calibrator	<0.8	0.8 to <1.1	≥1.1
DiaSorin	LIAISON SARS-CoV-2 S1/S2 IgG	IgG	S1 and S2	CLIA	LIAISON XL	Index based on 2 calibrators	<15	NA	≥15
Roche	Elecsys Anti–SARS-CoV-2	Total immunoglobulins	Nucleocapsid	ECLIA	cobas e411	Index based on 2 calibrators	<1.0	NA	≥1.0
Siemens	SARS-CoV-2 Total (COV2T)	Total immunoglobulins	RBD	CLIA	Centaur XP	Index based on 2 calibrators	<1.0	NA	≥1.0

CLIA, chemiluminescence immunoassay; ECLIA, electrochemiluminescence immunoassay; ELISA, enzyme-linked immunosorbent assay; IgG, immunoglobulin G; NA, not applicable; RBD, receptor-binding domain; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.
^aS1 and S2 are subunits of the spike protein; the RBD is a domain within the S1 subunit.

Table 2. Characteristics of Patients With Serum Samples Used in This Comparative Study
Sample Type (Year Collected)	Samples (Unique Patients), No.	Sex, Male/Total, No.	Age, Range (Median) y	Days After Symptom Onset, Range (Median)	Hospital Admission, No.
Prepandemic presumptive negative (2016–2019)	74 (74)
Samples with demographics	31 (31)	17/31	12–79 (46)
Cross-reactivity panel (2020)	63 (63)
Seasonal coronavirus	13 (13)	6/13	2–83 (55)
HKU1	7
OC43	2
NL63	2
229E	2
Hepatitis B	10 (10)	5/10	22–72 (48)
Hepatitis C	10 (10)	710	20–74 (62)
Autoimmune	30 (30)	4/30	12–71 (40)
ANA	10
ENA	7
Anti-dsDNA	17
RF	6
Increased IgG	5
M-protein	1
Anti-EBV	1
Anti-HSV 1 and 2	1
Clinical RT-PCR negative (2020)	53 (53)	13/53	16–64 (40)	0–52^a (4)
Symptom onset >14 d	25 (25)	6/25	17–61 (41)	15–52 (32)
Symptom onset <14 d	28 (28)	7/28	16–64 (38)	0–4^a (0)
Clinical RT-PCR positive (2020)	131 (68)	29/68	23–65 (49)	5–59 (20)	35
Retrospective serology	36 (35)	10/35	23–65 (38)	18–59^b (41)	2
Prospective medical care	95 (33)	19/33	27–85 (63)	5–29 (14)	33

ANA, anti–nuclear antibody by immunofluorescence; dsDNA, double-stranded DNA; EBV, Epstein-Barr virus; ENA, anti–nuclear antibody by multiplex extractable nuclear antigen; HSV, herpes simplex virus; RF, rheumatoid factor; RT-PCR, reverse transcription polymerase chain reaction.
^aOne patient without symptoms.
^bRange excluding first sample on day 0 of a patient with two samples.

Table 3. Specificity of Assays ^a
Manufacturer	All Expected Negative Samples^b		Prepandemic and Cross-Reactivity		Clinical RT-PCR Negative^b
Manufacturer	Positive/Samples, No.	Specificity (95% CI)	Positive/Samples, No.	Specificity (95% CI)	Positive/Samples, No.	Specificity (95% CI)
DiaSorin	6/185	96.8 (93.1–98.8)	4/134	97.0 (92.5–99.2)	2/51	96.1 (86.5–99.5)
EUROIMMUN, indeterminate = positive	16/188	91.5 (86.6–95.1)	9/137	93.4 (87.9–97.0)	7/51	86.3 (73.7–94.3)
EUROIMMUN, indeterminate = negative	7/188	96.3 (92.5–98.5)	5/137	96.4 (91.7–98.8)	2/51	96.1 (86.5–99.5)
Roche	0/188	100 (98.1–100)	0/137	100 (97.3–100)	0/51	100 (93.0–100)
Siemens	0/188	100 (98.1–100)	0/137	100 (97.3–100)	0/51	100 (93.0–100)

CI, confidence interval; RT-PCR, reverse transcription polymerase chain reaction.
^aSpecificity is calculated for all samples expected to be negative and pertinent subgroups. For EUROIMMUN, calculations are provided for categorizing indeterminate results as positive or negative.
^bExcluding two samples that were positive on all assays and adjudicated as presumptive RT-PCR testing false negatives.

Table 4. Samples With Discordant Results or Immunosuppression ^a
Sample Type	DiaSorin	EUROIMMUN	Roche	Siemens	Age, y	Sex	Days After Symptom Onset	Other
Sample Type	Arbitrary Units (15.00)^b	Ratio (0.80–1.10)^b	Index (1.00)^b	Index (1.00)^b	Age, y	Sex	Days After Symptom Onset	Other
Negatives
Prepandemic	<3.80	1.68	0.09	<0.05	NA	NA	NA
Prepandemic	98.2	0.41	0.10	<0.05	NA	NA	NA
Prepandemic	36.7	0.96	0.08	<0.05	NA	NA	NA
Prepandemic	16.6	2.49	0.09	<0.05	NA	NA	NA
Prepandemic	10.9	1.55	0.10	<0.05	56	M	NA
Prepandemic	4.79	1.07	0.09	<0.05	30	F	NA	Also positive for rubella IgG
Prepandemic	8.60	4.29	0.10	<0.05	53	M	NA	Neurofibromatosis type 1 and adrenal adenoma
Cross-reactive	11.5	3.24	0.13	<0.05	11	F	NA	NL63 seasonal coronavirus
Cross-reactive	15.5	1.07	0.09	<0.05	22	F	NA	Hepatitis B
Cross-reactive	9.25	0.91	0.08	<0.05	36	F	NA	ANA speckled at 1:640
RT-PCR negative	<3.80	1.81	0.08	<0.05	16	M	NA	No symptoms, preprocedure test, and Noonan syndrome
No RT-PCR test	37.9	0.29	0.06	0.08	17	M	39	Foot rash of concern for possible post–COVID-19 sequelae
RT-PCR negative ^c	16.7	0.36	0.09	<0.05	49	F	4	Eosinophilic esophagitis, hypertension, liver mass
RT-PCR negative	<3.80	0.95	0.07	0.11	28	F	35
RT-PCR negative ^d	5.07	2.38	0.07	0.11	60	F	0	Pernicious anemia
RT-PCR negative	<3.80	1.01	0.07	0.11	49	F	34	Gilbert syndrome
RT-PCR negative	<3.80	0.89	0.06	<0.05	51	M	19
RT-PCR negative	8.20	0.99	0.06	0.15	37	M	16
RT-PCR negative	4.01	1.05	0.06	0.15	33	F	15
Positives
RT-PCR positive	9.71	1.02	9.15	1.01	37	F	32
RT-PCR positive	9.42	1.05	30.4	2.53	27	F	59
RT-PCR positive	38.4	0.43	4.83	0.29	50	F	22
Immunosuppression
RT-PCR positive	41	5.83	25.8	5.15	28	F	44	Systemic lupus erythematosus on hydroxychloroquine
RT-PCR positive	268	13.06	58.5	>10	63	F	48	Rheumatoid arthritis on hydroxychloroquine and tofacitinib (tofacitinib paused during symptoms)
RT-PCR positive	12.1	1.57	4.88	2.60	59	F	41	Unclear rheumatologic condition on hydroxychloroquine
RT-PCR positive	7.29	1.70	9.71	>10	60	M	11	Lung transplant on tacrolimus
RT-PCR positive	24.0	0.83	15.5	1.48	59	M	14	Kidney transplant on mycophenolate and tacrolimus
RT-PCR positive	90.5	8.11	43.2	>10			19	Kidney transplant on mycophenolate and tacrolimus

ANA, anti–nuclear antibody by immunofluorescence; COVID-19, coronavirus disease 2019; IgG, immunoglobulin G; RT-PCR, reverse transcription polymerase chain reaction.
^aDiscordant results are classified as nonconforming with expected results and/or other assays. Bold type indicates discordance, and italicized type indicates indeterminate results.
^bThreshold for positivity (and indeterminate range for EUROIMMUN).
^cRepeat testing by DiaSorin was 15.5 AU/mL. Testing 15 days later by EUROIMMUN was negative with ratio of 0.38.
^dTwenty days later tested by DiaSorin with negative results, <3.8 AU/mL.

Table 5. Sensitivity of Assays ^a
Sample Group	DiaSorin		EUROIMMUN, Indeterminate = Positive		EUROIMMUN, Indeterminate = Negative		Roche		Siemens
Sample Group	Positive/Samples, No.	Sensitivity (95% CI)	Positive/Samples, No.	Sensitivity (95% CI)	Positive/Samples, No.	Sensitivity (95% CI)	Positive/Samples, No.	Sensitivity (95% CI)	Positive/Samples, No.	Sensitivity (95% CI)
All RT-PCR+	87/124	70.2 (61.3–78.0)	104/131	79.4 (71.5–86.0)	97/131	74.1 (65.7–81.3)	109/129	84.5 (77.1–90.3)	103/131	78.6 (70.6–85.3)
All RT-PCR+ symptoms ≥10 d	83/114	72.8 (63.7–80.7)	98/121	81.0 (72.9–87.6)	92/121	76.0 (67.4–83.3)	103/119	86.6 (79.1–92.1)	97/121	80.2 (71.9–86.9)
All RT-PCR+ symptoms ≥12 d	75/98	76.5 (66.9–84.5)	88/104	84.6 (76.2–90.9)	82/104	78.9 (69.7–86.2)	92/103	89.3 (81.7–94.6)	87/104	83.7 (75.1–90.2)
All RT-PCR+ symptoms ≥14 d	68/82	82.9 (73.0–90.3)	79/84	94.1 (86.7–98.0)	74/84	88.1 (79.2–94.1)	81/83	97.6 (91.6–99.7)	78/84	92.9 (85.1–98.7)
All RT-PCR+ symptoms ≥16 d	56/65	86.2 (75.3–93.5)	63/65	96.9 (89.3–99.6)	60/65	92.3 (83.0–97.5)	64/65	98.5 (91.7–>99.5)	62/65	95.4 (87.1–99.0)
All RT-PCR+ symptoms ≥18 d	44/50	88.0 (75.7–95.5)	48/50	96.0 (86.3–99.5)	46/50	92.0 (80.8–97.8)	50/50	100 (92.9–100)	49/50	98.0 (89.4–>99.9)
Unique patients	56/65	86.2 (75.3–93.5)	64/68	94.1 (85.6–98.4)	62/68	91.2 (81.8–96.7)	63/67	94.0 (85.4–98.4)	63/68	92.7 (83.7–97.6)
Unique patients' symptoms ≥14 d on last sample	50/57	87.7 (76.3–94.9)	56/58	96.6 (88.1–99.6)	54/58	93.1 (83.3–98.1)	57/58	98.3 (90.8–>99.9)	56/58	96.6 (88.1–99.6)

CI, confidence interval; RT-PCR, reverse transcription polymerase chain reaction.
^aSensitivities of each assay are determined at varying thresholds for number of days after onset of symptoms. For EUROIMMUN, calculations are provided for categorizing indeterminate results as positive or negative.

Table 6. Estimated Positive Predictive Values at Varying Frequencies of COVID-19 Prevalence ^a
Prevalence	DiaSorin PPV (95% CI), %		EUROIMMUN PPV (95% CI), Indeterminate = Positive, %		EUROIMMUN PPV (95% CI), Indeterminate = Negative, %		Roche PPV (95% CI), %		Siemens PPV (95% CI), %
Prevalence	All Samples	≥14 Days After Symptom Onset	All Samples	≥14 Days After Symptom Onset	All Samples	≥14 Days After Symptom Onset	All Samples	≥14 Days After Symptom Onset	All Samples	≥14 Days After Symptom Onset
1	17.9 (10.1–40.6)	20.5 (11.8–44.6)	8.6 (5.8–14.1)	10.4 (6.8–16.2)	16.7 (9.8–36.2)	19.3 (11.3–37.9)	100 (36.2–100)	100 (39.9–100)	100 (34.6–100)	100 (38.2–100)
5	53.2 (37.0–78.0)	57.4 (41.1–80.7)	32.9 (24.2–46.1)	36.8 (27.5–50.1)	51.1 (36.2–74.7)	55.5 (39.8–76.1)	100 (74.7–100)	100 (77.6–100)	100 (73.4–100)	100 (76.3–100)
10	70.6 (55.4–88.2)	74.0 (59.6–89.8)	50.9 (40.2–64.3)	55.1 (44.5–68.0)	68.8 (54.5–86.2)	72.4 (58.3–87.0)	100 (86.2–100)	100 (88.0–100)	100 (85.3–100)	100 (87.2–100)
20	84.4 (73.6–94.4)	86.5 (76.8–95.2)	70.0 (60.2–80.2)	73.4 (64.3–82.7)	83.3 (73.0–93.4)	85.5 (75.9–93.8)	100 (93.4–100)	100 (94.3–100)	100 (92.9–100)	100 (93.9–100)

CI, confidence interval; PPV, positive predictive value.
^aPPVs are shown as the point estimate (%) and 95% confidence interval based on bootstrapped likelihood ratios.