AI study by Google researchers reveals incredible jump in Med-PaLM 2's answering accuracy

Google researchers have showcased a study where Med-PaLM 2 was pitted against other contemporary AI platforms to check its performance in medical question-answering and it has outperformed significantly.

By: HT TECH
| Updated on: May 18 2023, 23:25 IST
Med-PaLM 2 outperforms platforms like GPT-Neo, PubMedBERT and others in medical question-answering, as per a new Google study.
Med-PaLM 2 outperforms platforms like GPT-Neo, PubMedBERT and others in medical question-answering, as per a new Google study. (Bloomberg)
Med-PaLM 2 outperforms platforms like GPT-Neo, PubMedBERT and others in medical question-answering, as per a new Google study.
Med-PaLM 2 outperforms platforms like GPT-Neo, PubMedBERT and others in medical question-answering, as per a new Google study. (Bloomberg)

Artificial intelligence has permeated every technology field today. Among them, one field which is particularly resilient to emerging technologies is the medical field. Due to dealing in extremely sensitive areas that can lead to life-and-death scenarios, the medical field has been apprehensive about deploying new medtech tools into general practice. However, AI has been knocking at its door for some time, and if a new study conducted by Google researchers is to be believed, Google's in-house Med-PaLM 2 is getting really high accuracy scores in medical question-answering (MedQA) and is in prime position to enable medical professionals in offering faster medical care to patients.

In effect, Med-PaLM 2, is a medical large language model (LLM) that is being trained to synthesise information from medical images. In fact, not just Google, other players too are working on generative AI in the healthcare industry, and among them is Sam Altman-led OpenAI's ChatGPT. And the competition is stiff. A study published in JAMA Internal Medicine said that ChatGPT delivered higher quality answers to questions than written responses from actual practitioners.

Now, on Wednesday, Google Health UK research lead Alan Karthikesalingam posted on Twitter, highlighting the accomplishment. He said, “So happy to share #MedPaLM2 - our team's evolution of Med-PaLM. A new state of art for medical question-answering! Med-PaLM 2 scores 86.5% on MedQA-USMLE, exceeding Med-PaLM's score by >19%, & 81.8% on PubMedQA”.

It should be noted that the MedQA-USMLE dataset is a multiple-choice questionnaire based on the USA's Medical License Exams. So, getting a high score essentially means that the AI could, in theory, get certified to practice medicine in the USA. PubMedQA is also a similar dataset. In the dataset test, Med-PaLM 2 has scored a high 86.5% as per the study conducted by the group. The study is currently available in a pre-print stage on arXiv. It should also be noted that the study has not been peer-reviewed or published in a journal so far.

Google AI scores big in Medical Licence Exam

Karthikesalingam stated in a series of tweets the high level of scrutiny taken in order to ensure that the results of the test were not a fluke or a misrepresentation of the AI platform's abilities. He said, “We believe in rigorous, careful evaluation. Physicians even preferred #MedPaLM2's long-form answers to answers from other real physicians along 8/9 axes of quality including medical accuracy (consensus w/medical opinion) and reasoning, with less likelihood of harm”.

“To highlight the real-world importance of nuanced evaluation we introduce a new dataset of "adversarial" questions designed specifically to probe LLM weaknesses including #HealthEquity,” he added.

It is unclear at the moment how much of an impact this new AI technology can have in the medical field but Google seems optimistic about the results. However, the study is just the beginning. In order for this technology to be adopted and used in real-life situations, it will have to undergo much stricter scrutiny to understand whether the AI can consistently and reliably help patients in their health care.

As it is, even Google chief Sundar Pichai, while speaking at the recently held Google I/O had highlighted how the company was working on this technology in a careful and responsible manner to ensure it did not go wrong.

Follow HT Tech for the latest tech news and reviews , also keep up with us on Twitter, Facebook, Google News, and Instagram. For our latest videos, subscribe to our YouTube channel.

First Published Date: 18 May, 23:08 IST
NEXT ARTICLE BEGINS