Did IBM overhype Watson Health's AI promise?

In recent weeks, IBM has changed leadership at its Watson Health division and announced a new business strategy for deployment that relies on a hybrid cloud, not a  public- or private-cloud only model.

Over the past year, Watson Health – particularly Watson for Oncology – has come in for criticism that it has underperformed expectations.

(Watson for Oncology is IBM's commercial cognitive computing cloud platform that analyses large volumes of patient healthcare data and published medical studies to offer physicians cancer treatment options)

Laura Craft, a vice president of research for Gartner's Healthcare Strategy business, said it noted that IBM's Cognitive Computing Division did not do well in recent third-quarter results, "and that was largely driven by the healthcare component."

Craft also pointed to the recent leadership changes as indicative of internal problems.

In July, the healthcare news publication Stat published a report claiming "internal IBM documents" showed the Watson supercomputer often spit out erroneous cancer treatment advice and that company medical specialists and customers identified "multiple examples of unsafe and incorrect treatment recommendations," even as IBM was promoting its AI technology.

Stat cited several slide decks it had obtained from a presentation made by IBM Watson Health's deputy chief health officer in 2016.

The slides mostly blamed problems on the training of Watson by IBM engineers and doctors at the Memorial Sloan Kettering Cancer Center (MSKCC).

Separately, an article by the Wall Street Journal claimed Watson Health had not made progress in bringing AI to healthcare.

In August, John Kelly (then IBM's senior vice president of Cognitive Solutions and Research), fired back in a blog about the reports: "It is true, as the article reports, that we at IBM have placed a big bet on healthcare.

"We have done this for two reasons: 1) Most importantly, we know that AI can make a big difference in solving medical challenges and supporting the work of the healthcare industry, and 2) We see an enormous business opportunity in this area as the adoption of AI increases," he wrote.

"To suggest there has been no patient benefit," Kelly continued, "is to ignore both what we know The Wall Street Journal was told by a number of physicians around the world and these institutions' own public comments – which we believe speak for themselves."

Kelly pointed to five healthcare facilities and the Department of Veterans Affairs that offered physician testimonials on how Watson for Oncology, Watson for Clinical Trial Matching and Watson for Genomics had reduced physician time and effort, increased clinical trial participants or massively increased the volume of research data available to the AI engine.

Asked whether IBM's Watson for Oncology had relayed erroneous cancer treatment advice, Ed Barbini, IBM's vice president of external relations, flatly denied the charge.

While IBM faces declining revenue overall, and its recently released third-quarter earnings showed revenue from cognitive offerings was down six per cent from last year, Watson Health saw growth, according to Barbini.

He noted that IBM does not release numbers specific to Watson Health for "competitive reasons."

Barbini admitted that developing Watson Health and, specifically, Watson for Oncology is not an easy task, but it remains an important one.

"That's why IBM dove into it three years ago. Did you really think oncology would be mastered in three years?" Barbini said. "However, let's look at the facts. More than 230 hospitals are using one of our oncology tools.

"We've had 11 [software] updates over last year and half and we've doubled the number of patients we've reached to over 100,000 as of the end of the third quarter of this year."

Earlier this month, the head of Watson Health for the past three years, Deborah DiSanzo, stepped down and Kelly took over. DiSanzo is continuing to work with IBM Cognitive Solutions' strategy team, according to a company spokesperson.

IBM has been the most aggressive among technology companies using AI in support of evidence-based medicine for health care. But recent problems have reportedly led to the loss of some of its larger hospital clients.

Craft said the Watson for Oncology unit is garnering most of the bad press; she believes its because IBM's marketing department promised something they could not deliver.

Watson Health should have remained in an incubator stage far longer so more data could be ingested, enabling better evidence-based medicine treatment options.

IBM's "party line" has been to deny the reports and publish select development partner and client testimonials, Craft said.

IBM

"I think if I understand IBM's vision..., it was to really move toward personalised medicine. The ambition was to get to the state where they can target and correlate therapies and drugs to what the patient will respond best to," Craft said.

"I think from a medical research perspective, they are going to get there in 10 or 15 years when we've got better, more consistent, real-world data – all things I don't think Watson has the benefit of today."

As a result, IBM has failed deliver on the capabilities it marketed to healthcare providers. "They have disappointed clients and they've certainly created skepticism around the integrity of their more advanced products," Craft said.

It's not the tech, it's the time

Watson's AI technology is not the problem, Craft said; it simply hasn't had enough time or quality data input to become the personalised medicine engine IBM has pitched.

Cynthia Burghard, IDC's research director for Value-based Healthcare IT Transformation Strategies, said IBM "shot themselves in the foot" by choosing to aim Watson at something as complex as healthcare right out of the gate.

"Part of Watson for Health's challenges is they were very aggressive with marketing, which is kind of an IBM trait. And, then it came [to] delivering it and they chose oncology, they chose genomics – really tough nuts to crack," Burghard said.

IBM had hoped to offer Watson as a software product, Burghard said, where oncologists could simply plug in patient data and they would receive recommendations for treatment. "That's the commercial version of Watson Health they were hoping, but that's not been realised," she said.

Instead, IBM has had to work with the hospital clients to ensure Watson functions properly, Burghard explained.

Read more on the next page...