Artificial Intelligence Improves the Accuracy in Histologic Classification of Breast Lesions

António Polónia, MD, PhD; Sofia Campelos, MD; Ana Ribeiro, MD; Ierece Aymore, MD; Daniel Pinto, MD; Magdalena Biskup-Fruzynska, MD; Ricardo Santana Veiga, MD; Rita Canas-Marques, MD; Guilherme Aresta, MEng; Teresa Araújo, MEng; Aurélio Campilho, PhD; Scotty Kwok, MSc; Paulo Aguiar, PhD; Catarina Eloy, MD, PhD

Disclosures

Am J Clin Pathol. 2021;155(4):527-536.

Conclusions

To our knowledge, this study represents the first time that machine learning algorithms in DP have been used to measure their impact in the classification accuracy of pathologists and pathology residents. We demonstrate that such CAD tools can increase the classification accuracy in the setting of breast lesions, providing the basis for its future clinical implementation with supervision.

1 2 3 4 5 6

Next Section

Table 1. Characterization of Tests A and B
Classes	Test A	Test B
Normal, No. (%)	25 (25)	31 (20.4)
Benign, No. (%)	25 (25)	65 (42.8)
Fibrocystic change	1	31
Inflammation	0	15
Columnar cell change	4	8
Microcalcification	0	5
Fibroadenoma	2	2
Ductal hyperplasia	1	3
Apocrine metaplasia	3	0
Fat necrosis	3	0
Atrophy	2	0
Intraductal papilloma	4	0
Sclerosing lesion	2	1
Adenosis	2	0
Secretory changes	1	0
Carcinoma in situ, No. (%)	25 (25)	33 (21.7)
Ductal	24	7
Lobular	1	26
Invasive carcinoma, No. (%)	25 (25)	23 (15.1)
Ductal	23	11
Lobular	1	2
Tubular	1	0
Mucinous	0	10
Total	100	152

Table 2. Summary of Tasks Developed in Different Tests and Phases
Tests and Phases	Tasks
Test A	100 H&E photographs
Test B	152 ROIs in 10 H&E WSIs
Phase 1^a	Classification of tests A and B with 4 classes: normal, benign, carcinoma in situ and invasive carcinoma
Phase 2^b	Same as in phase 1, knowing the classification of the algorithms
Phase 3 (test A only)	Same as in phase 2, knowing the accuracy of the algorithm and the observers and the types of errors between different classes

ROI, region of interest; WSI, whole-slide image.
^aIn cases of doubt between classes, observers should choose their favorite classification and provide the respective alternative.
^bIf the observer classification matched the algorithm classification, it could not be changed; if there was an alternative classification, there was the possibility of keeping or discarding the alternative classification. If the observer classification did not match the algorithm classification, observers could reclassify the photograph or ROI.

Table 3. Diagnostic Accuracy in Test A and B
	Phase	Test A	Test B
Algorithm A	1	0.87	NA
Algorithm B	1	NA	0.49
Pathologists (average)	1	0.83	0.83
	2	0.87	0.82
	3	0.90	NA
Residents (average)	1	0.77	0.89
	2	0.82	0.88
	3	0.87	NA
All observers (average)	1	0.80^a	0.86
	2	0.85^a,b	0.85
	3	0.88^b	NA

NA, not applicable.
^aWilcoxon, P < .001
^bWilcoxon, P = .001.

Table S1. Classification accuracy in test A
	Phase	Acc
Algorithm A	1	0.87
P1	1	0.86
	2	0.93
	3	0.92
P2	1	0.94
	2	0.94
	3	0.94
P3	1	0.73
	2	0.75
	3	0.75
P4	1	0.78
	2	0.87
	3	0.97
Pathologists (average)	1	0.83
	2	0.87
	3	0.90
R1	1	0.86
	2	0.86
	3	0.89
R2	1	0.72
	2	0.80
	3	0.88
R3	1	0.74
	2	0.81
	3	0.84
Residents (average)	1	0.77
	2	0.82
	3	0.87
All observers (average)	1	0.80
	2	0.85
	3	0.88

P: pathologist; R: resident; Acc: accuracy.

Table S2. Discordances with ground-truth in test A
Algorithm A	Phase	N vs B	N vs CIS	N vs IC	B vs CIS	B vs IC	CIS vs IC	Total
Algorithm A	1	4	2	0	2	4	1	13
Pathologists	1	30 (7.5%)	0 (0%)	0 (0%)	24 (6%)	13 (3.3%)	2 (0.5%)	69 (17.3%)
	2	24 (6%)	0 (0%)	0 (0%)	18 (4.5%)	9 (2.3%)	0 (0%)	51 (12.8%)
	3	23 (5.8%)	1 (0.3%)	0 (0%)	11 (2.8%)	7 (1.8%)	0 (0%)	42 (10.5%)
Residents	1	22 (7.3%)	2 (0.7%)	2 (0.7%)	26 (8.7%)	14 (4.7%)	2 (0.7%)	68 (22.7%)
	2	14 (4.7%)	2 (0.7%)	1 (0.3%)	22 (7.3%)	12 (4%)	2 (0.7%)	53 (17.7%)
	3	10 (3.3%)	3 (1%)	1 (0.3%)	13 (4.3%)	10 (3.3%)	2 (0.7%)	39 (13%)
All observers	1	52 (7.4%)	2 (0.3%)	2 (0.3%)	50 (7.1%)	27 (3.9%)	4 (0.6%)	137 (19.6%)
	2	38 (5.4%)	2 (0.3%)	1 (0.1%)	40 (5.7%)	21 (3%)	2 (0.3%)	104 (14.9%)
	3	33 (4.7%)	4 (0.6%)	1 (0.1%)	24 (3.4%)	17 (2.4%)	2 (0.3%)	81 (11.6%)

N: normal; B: benign; CIS: carcinoma in situ; IC: invasive carcinoma

Table S3. Differential diagnosis in test A
	Phase	N vs B	B vs CIS	B vs IC	CIS vs IC	Total
Pathologists	1	2 (0.5%)	20 (5%)	27 (6.8%)	24 (6%)	73 (18.3%)
	2	12 (3%)	24 (6%)	19 (4.8%)	7 (1.8%)	62 (15.5%)
	3	21 (5.3%)	26 (6.5%)	23 (5.8%)	7 (1.8%)	77 (19.3%)
Residents	1	0 (0%)	10 (3.3%)	6 (2%)	9 (3%)	25 (8.3%)
	2	4 (1.3%)	13 (4.3%)	7 (2.3%)	6 (2%)	30 (10%)
	3	8 (2.7%)	14 (4.7%)	9 (3%)	8 (2.7%)	39 (13%)
Allobservers	1	2 (0.3%)	30 (4.3%)	33 (4.7%)	33 (4.7%)	98 (14%)
	2	16 (2.3%)	37 (5.3%)	26 (3.7%)	13 (1.9%)	92 (13.1%)
	3	29 (4.1%)	40 (5.7%)	32 (4.6%)	15 (2.1%)	116 (16.6%)

N: normal; B: benign; CIS: carcinoma in situ; IC: invasive carcinoma

Table S4. Changes of classification in test A
	Phase	FC	AC
P1	2	7%	15%
P1	3	11%	14%
P2	2	6%	33%
P2	3	6%	33%
P3	2	2%	14%
P3	3	2%	24%
P4	2	12%	25%
P4	3	19%	24%
Pathologist (average)	2	6.8%	21.8%
Pathologist (average)	3	9.5%	23.8%
R1	2	0%	4%
R1	3	3%	4%
R2	2	8%	7%
R2	3	19%	22%
R3	2	9%	12%
R3	3	16%	14%
Resident (average)	2	5.7%	7.7%
Resident (average)	3	12.7%	13.3%
Allobservers (average)	2	6.3%	15.7%
Allobservers (average)	3	10.9%	19.3%

P: pathologist; R: resident; FC: favorite classification; AC: alternative classification.

Table S5. Classification accuracy in test B
	Phase	Acc
AlgorithmB	1	0.49
P1	1	0.93
P1	2	0.90
P2	1	0.85
P2	2	0.88
P3	1	0.70
P3	2	0.70
Pathologists (average)	1	0.83
Pathologists (average)	2	0.82
R1	1	0.92
R1	2	0.96
R2	1	0.93
R2	2	0.92
R3	1	0.82
R3	2	0.77
Residents (average)	1	0.89
Residents (average)	2	0.88
Allobservers (average)	1	0.86
Allobservers (average)	2	0.85

P: pathologist; R: resident; Acc: accuracy.

Table S6. Discordances with ground-truth in test B
AlgorithmB	Phase	N vs B	N vs CIS	N vs IC	B vs CIS	B vs IC	CIS vs IC	Total
AlgorithmB	1	23 (15.1%)	13 (8.6%)	1 (0.7%)	18 (11.8%)	22 (14.5%)	0 (0%)	77 (50.7%)
Pathologists	1	34 (7.5%)	0	0	44 (9.7%)	2 (0.4%)	0	80 (17.5%)
Pathologists	2	37 (8.1%)	0	0	42 (9.2%)	2 (0.4%)	0	81 (17.8%)
Residents	1	33 (7.2%)	0	0	13 (2.9%)	4 (0.9%)	0	50 (11.0%)
Residents	2	33 (7.2%)	0	0	17 (3.7%)	3 (0.7%)	0	53 (11.6%)
Allobservers	1	67 (7.4%)	0	0	57 (6.3%)	6 (0.7%)	0	130 (14.3%)
Allobservers	2	70 (7.7%)	0	0	59 (6.5%)	5 (0.6%)	0	134 (14.7%)

N: normal; B: benign; CIS: carcinoma in situ; IC: invasive carcinoma

Table S7. Differential diagnosis in test B
Algorithm B	Phase	N vs B	B vs CIS	B vs IC	CIS vs IC	Total
Algorithm B	1	57 (37.5%)	7 (4.6%)	12 (7.9%)	6 (3.9%)	82 (53.9%)
Pathologists	1	6 (1.3%)	13 (2.9%)	1 (0.2%)	0 (0%)	20 (4.4%)
Pathologists	2	14 (3.1%)	26 (5.7%)	1 (0.2%)	0 (0%)	41 (9.0%)
Residents	1	7 (1.5%)	8 (1.8%)	3 (0.7%)	8 (1.8%)	26 (5.7%)
Residents	2	34 (7.5%)	54 (11.8%)	3 (0.7%)	10 (2.2%)	101 (22.2%)
All observers	1	13 (1.4%)	21 (2.3%)	4 (0.4%)	8 (0.9%)	46 (5.0%)
All observers	2	48 (5.3%)	80 (8.8%)	4 (0.4%)	10 (1.1%)	142 (15.6%)

N: normal; B: benign; CIS: carcinoma in situ; IC: invasive carcinoma

Table S8. Changes of classification in test B
	Phase	FC	AC
P1	2	7.2%	17.6%
P2	2	3.9%	5.9%
P3	2	0%	2.6%
Pathologist (average)	2	3.7%	8.7%
R1	2	3.9%	7.2%
R2	2	2.0%	11.8%
R3	2	10.5%	37.3%
Resident (average)	2	5.5%	18.8%
All observers (average)	2	4.6%	13.7%