Large-language AI models aren’t immune to medical misinformation, but they can learn to discern

Medical AI is likely to pass along potentially dangerous misinformation when the distortion comes to the model from a normally trustworthy source using a convincingly authoritative voice. 

The source might be a physician’s note (rather than a social media comment). The voice might suggest the tone of a serious scholar (rather than a brash autodidact). 

In this way, LLMs can behave much like many if not most healthcare consumers. 

The good news is that the AI can be trained and prompted to alert its end-user when logical fallacies are present in an algorithm’s reasoning. 

Logical fallacy “framing” is a way to set up an argument so that the recipient is led to overlook shaky reasoning, often by a subtle appeal to emotion, popular opinion or the human inclination to trust authority. 

Researchers at Mount Sinai Health System define the problem and how best to deal with it in a large study published in The Lancet Digital Health.

“Our findings show that current AI systems can treat confident medical language as true by default, even when it’s clearly wrong,” explains co-senior and co-corresponding author Eyal Klang, MD, chief of generative AI at the Icahn School of Medicine at Mount Sinai. “A fabricated recommendation in a discharge note can slip through. It can be repeated as if it were standard care.”

For these models, Klang adds, “what matters is less whether a claim is correct than how it is written.” 

Logical fallacies can be made to reveal themselves  

For the study, the researchers tested 20 large-language models with more than 3.4 million prompts. All of these carried health misinformation.

The team drew the distortions from three real-world sources: public-forum and social-media dialogues, real hospital discharge notes in which the team inserted a single false recommendation, and 300 physician-validated simulated vignettes. 

The researchers used some common logical-fallacy pitfalls—circular reasoning, hasty generalization, slippery slope foreboding, strawman argumentation and others—to test how rhetorical framing influences model behavior. 

They posed each prompt once in a neutral base form and 10 times with a named logical fallacy. For every run they logged susceptibility (model accepts the false claim) and fallacy detection (model flags the rhetoric).

Their key finding: LLMs absorb harmful medical fabrications, especially when phrased in authoritative clinical prose. 

However, counter-intuitively, the models “become less vulnerable when the same claims are wrapped in most logical fallacy styles,” Klang and colleagues report. “Therefore, improving safety appears to depend less on model scale and more on fact-grounding and context-aware guardrails.”

How prone to lying is your medical LLM? 

In invited commentary on the study, two psychology professors at the University of Cambridge in the U.K. suggest LLMs need to be “immunized” to protect them, the models, against catching and transmitting medical misinformation. 

“[I]noculation prompting during training can be an effective method for preventing misaligned model behavior in the future,” write Sander van der Linden, PhD, and PhD candidate Yara Kyrychenko. 

In inoculation prompting, they explain, a model is explicitly asked to act in misaligned ways in controlled settings so that it learns to discern between different types of content. 

“In particular, adding a system prompt that instructs an LLM to produce misinformation and finetuning on a dataset of false healthcare-related claims could increase the model’s understanding of what health misinformation is,” van der Linden and Kyrychenko write. “When prompted to be helpful instead of producing misinformation, the model should be more likely to generate truthful responses or push back on false claims.”

Mahmud Omar, MD, co-lead author of the Mount Sinai study, suggests hospitals and developers consider using the project’s dataset as a “stress test” for medical AI. 

“Instead of assuming a model is safe, you can measure how often it passes along a lie, and whether that number falls in the next generation,” Omar says.

Both the Mount Sinai study and the Cambridge professors’ commentary on it are posted in full for free.