Medical AIs easily fooled by authoritative misinformation, study

Specialist medical LLMs are just as easily deceived by fake info in clinicians’ notes

AI tools used in a medical context are easily fooled by misinformation delivered in an authoritative tone.

That’s the finding of a recent study published in The Lancet Digital Health which assessed the outputs of 20 LLMs used in a medical context, including models specially trained on medical data.

The LLMs were fed textual data from hospital discharge summaries that included fake recommendations, medical misinformation from social media, and clinical information written by doctors which also included false information. They were prompted with requests to identify misinformation and logical fallacies, including appeals to emotion.

The results revealed all models were significantly susceptible to medical misinformation, with smaller models being more easily “fooled”. The worst performing smaller models were susceptible in 63.3% of cases whereas larger general-purpose models like GPT-4 were deceived in 10.4% of cases. Worryingly, specialised medical models, which tend to be smaller, were poor at detecting misinformation and logical fallacies.

Importantly, the models were more likely to “believe” fake information in the doctor’s notes than that from social media feeds, because it appeared more genuine, the study found.

“Falsehoods presented in authoritative and clinical prose in the discharge notes were more likely to bypass built-in guardrails than informal social media talk,” The Lancet Digital Health notes in an editorial.

“For example, more than half of the models accepted the false claim that drinking cold milk soothes oesophagitis-related bleeding, and at least three models accepted social media claims that mammography causes breast cancer.”

The results show that while AI tools can be helpful in tasks such as decision support, early screening and classification, they should be deployed with caution. The black-box nature of their decision making, failures in logical reasoning, variability of outputs across models, and propensity to give very different answers when prompts are slightly altered mean that medics should be extremely wary of using them for prognostics and critical decisions.

In the paper’s conclusion, authors Omar et al. suggest that increasing the size of the LLMs or clever prompt engineering will be unlikely to improve safety significantly. A more promising approach will be “focused grounding strategies and context-sensitive safeguards tailored specifically to clinical tasks and patient-facing applications.”

One possible avenue to improving LLMs is “model immunisation”, which fine tunes models on small, curated sets of explicitly labelled falsehoods, “inoculating” them with weak misinformation to build resistance to the real thing.

Underscoring the need to keep the human firmly in the loop when important decisions are being made, another recent study into medical AI found that while new tools are improving the accuracy of some cancer diagnoses, their use could be weakening doctors' own capabilities, a phenomenon akin to forgetting how to read a map after years of using GPS.