Watson vs Oncologists: 90% Agreed on Treatment (Eventually)

Nick Mulcahy

December 13, 2016

SAN ANTONIO – In the basement of the Manipal Comprehensive Cancer Center in Bengaluru, India, sits a mainframe computer that houses the Watson for Oncology artificial intelligence platform.

Each oncologist at the center has an iPad and laptop that can access Watson with the push of a button, said S. P. Somashekhar, MBBS, chairman of the center.

The button, which reads "Ask Watson," is embedded in each patient's electronic health record (EHR).

Watson integrates all of the information in an EHR, processes it, and returns a one-page treatment recommendation, with as many as 40 pages of backup that explains the advice, he said.

How long does the recommendation take? "Sixty seconds," Dr Somashekhar told Medscape Medical News at the San Antonio Breast Cancer Symposium (SABCS) 2016, where he presented a new study on Watson.

In a double-blinded validation study of 638 breast cancer cases at Manipal, which is one of the top cancer centers in India, Watson had a high level of agreement with the recommendations of the oncologists on the center's multidisciplinary tumor board.

Watson's treatment endorsements, which are presented as "for consideration" or "recommended," were concordant with the recommendations of the tumor board 90% of the time.

However, that sterling result came in the latter stages of the 3-year study after 175 "discordant cases" were reviewed a second time, adjusted by the oncologists, and passed through Watson again.

In the study's first pass, Watson and the tumor board only agreed 73% of the time.

The degree of concordance also depended on the type of breast cancer.

Watson's recommendations were concordant 79% of the time in nonmetastatic disease but only 46% of the time in metastatic cases, Dr Somashekhar told reporters at a meeting press conference, referring to data from both the study's first and second iterations.

In cases of triple-negative breast cancer, Watson agreed with the oncologists 68% of the time, but in HER2-negative cases, they agreed only 35% of the time, according to meeting press materials.

The tumor board doctors were much slower than Watson at making a recommendation. Initially it took an average of 20 minutes to capture and analyze data to generate recommendations when done manually; the mean time decreased to 12 minutes as the doctors improved performance.

C. Kent Osborne, MD, of Baylor College of Medicine in Houston, Texas, said that he could envision use of artificial intelligence in the cancer clinic.

Watson is "not going to put us out of business as physicians," he said with a smile at the press conference, for which he acted as moderator.

Dr Somashekhar emphasized that Watson "will always be consultative" and that it "cannot replace human clinical judgment and the essential patient-doctor relationship."

Other experts have embraced that vision.

In 2013, H. Jack West, MD, from the Swedish Cancer Center in Seattle, Washington, said Watson can help human beings to do the increasingly complex job of being an oncologist.

"A computer can incorporate a nearly infinite amount of new data coming out, whereas the human brain can't attend to as much and integrate it," he told Medscape Medical News at that time.

Dr West, who is not involved with Watson, saw two problems with its use in oncology: first, cancer is "defined by its variability"; and second, clinical decisions are often, in the end, matters of "judgment."

Even the most complex algorithms rely, to some degree, on "classic presentation," he said. Medical practice is less orderly, he noted. "The truth is that just about every cancer case is special and has some unique aspects to it, or at least there are so many extenuating factors that it's hard to be rigidly rule based," he said.

Ideally, Dr West would like to see a randomized controlled trial comparing clinical outcomes of patients treated with and without Watson.

Other clinicians have been skeptical about information technology initiatives in oncology, especially after the hype about how EHRs would cut costs and improve efficiencies, as reported by Medscape Medical News.

Watson made its first big splash in popular culture, when it appeared on the TV game show Jeopardy! and beat human opponents. It also has been used in cancer care in the United States, as reported by Medscape Medical News.

Watson for Oncology is being developed by IBM in conjunction with Memorial Sloan Kettering Cancer Center in New York City.

The system has synthesized an array of data gleaned from thousands of sources, including journal articles, national guidelines, individual hospital best practices, clinical trials, and textbooks. It can extract and access both structured and unstructured data from medical records through natural language processing and machine learning to provide cancer treatment options.

This study was funded by the Manipal Comprehensive Cancer Center. Dr Somashekhar has disclosed no relevant financial relationships. Dr Osborne has relationships with the pharmaceutical industry but none relevant to this area.

San Antonio Breast Cancer Symposium (SABCS) 2016. Abstract S6-07, presented December 9, 2016.

Follow Medscape senior journalist Nick Mulcahy on Twitter: @MulcahyNick

For more from Medscape Oncology, follow us on Twitter: @MedscapeOnc


Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.