Popular AI chatbots disseminate false medical information, Mount Sinai researchers say

According to new research, fluently used generative models, such as Chatgpt and Deepseek, are very vulnerable to the repetition and development of medical disinformation.

Mount Sinai researchers published this month a study revealing that when fictitious medical terms were inserted in patient scenarios, large languages models undoubtedly accepted them – and then generated detailed explanations for fully manufactured conditions and treatments.

Even a single invented term can derail a conversation with an AI chatbot, said Dr. Eyal Klang, one of the authors of the study and the head of the generative AI of Mount Sinai. He and the rest of the research team found that the introduction of a single false medical term, such as a false illness or a symptom, was sufficient to encourage a chatbot to hallucinate and produce authoritarian responses – but completely inaccurate –

Dr. Klang and his team carried out two series of tests. In the first, chatbots have simply been fed in patients’ scenarios, and in the second, the researchers added a note of precursation in a line at the prompt, reminding the model of AI that all the information provided can be inaccurate.

The addition of this prompt has decreased hallucinations by about half, said Dr. Klang.

The research team has tested six large-language models, which are all “extremely popular”, he said. For example, Chatgpt receives around 2.5 billion invites per day from its users. People are also becoming more and more exposed to important languages of languages, whether they are looking for them – like a simple Google search offers a summary generated by Geminites, noted Dr. Klang.

But the fact that popular chatbots can sometimes disseminate disinformation for health does not mean that health care should abandon or reduce generator, he noted.

The generative use of AI becomes more and more common in health environments for a good reason – due to the way these tools can accelerate the manual work of clinicians during a crisis in the course of professional exhaustion, said Dr. Klang.

“”[Large language models] Basically emulates our work in front of a computer. If you have a patient report and want a summary of this, they are very good. They are very good at administrative work and can have a very good reasoning capacity, so that they can offer things like medical suggestions. And you will see it more and more, “he said.

It is clear that new forms of AI will become even more present in health care in the years to come, added Dr. Klang. AI startups dominate the digital health investment market, companies like Abridge and Ambient Healthcare exceed Unicorn’s status, and the White House recently published an action plan to advance the use of AI in critical sectors like Healthcare.

Some experts were surprised that the Action Plan of the White House AI had not further emphasized AI security, since it is a major priority within the AI research community.

For example, responsible use of AI is a subject frequently discussed during industry events, and organizational organizations on AI in health care – such as the coalition for Health IA and Digital Medicine Society – have attracted thousands of members. In addition, companies like Openai and Anthropic have devoted significant quantities of their IT resources to security efforts.

Dr. Klang noted that the health community of health is well aware of the risk of hallucinations, and it still works to mitigate harmful results.

In the future, he highlighted the need for better guarantees and continuous human surveillance to ensure security.

Photo: Andriy Onufryenko, Getty Images