Millions of people now ask AI chatbots about symptoms, medicines and diets, treating them as round-the-clock medical advisers that never need an appointment. A new perspective paper in Nature Health, written by researchers at Binghamton University, Stanford University, Texas A&M University and Indiana University, argues that the real danger is not only that a chatbot can be wrong. It is that when the advice causes harm, the evidence of what went wrong is often impossible to trace.
Advice that leaves no record
When a doctor gives bad advice, there is usually a trail: a medical record, an insurance claim, a regulator or a licensing board. A chatbot conversation is different. It typically stays locked inside the AI company’s platform, the user has no clear way to report a problem, and outside researchers cannot audit what was said or measure how often things go wrong. Lead author Kaicheng Yang puts it bluntly: people are getting medical advice from chatbots, but nobody is actively monitoring those exchanges for errors.
The paper highlights a case in which a 60-year-old man asked a chatbot how to reduce chloride in his diet and was advised to swap table salt for sodium bromide. He followed the advice and was hospitalised with bromide toxicity, experiencing hallucinations and paranoia. Because the conversation could not be retrieved, his doctors had no direct way to see exactly what he had been told. More health coverage is available in our Health section.
You do not even have to ask
The researchers also warn that exposure does not require a deliberate question. AI summaries and social feeds can surface health claims while someone is simply scrolling, and chatbots can present incorrect, outdated or oversimplified information in a confident tone that sounds authoritative. When that advice is acted on and something goes wrong, clinicians, regulators and independent investigators are left without the evidence needed to spot patterns or correct the system.
What the researchers want changed
The authors frame this as a governance problem rather than a single bad answer. They call for independent, third-party monitoring and evaluation of AI health outputs, closer to the pharmacovigilance systems used to track harmful drug effects, along with clearer ways for users to report problems and for conversations to be documented when harm is suspected. Reporting by Digital Trends summarises the paper’s concerns, and the study itself is published in Nature Health.
The takeaway for everyday users
None of this means chatbots are useless for health questions; they can explain terms, prepare questions for a doctor and summarise public guidance. But the paper is a reminder to treat a fluent answer as a starting point, not a diagnosis. Keep records of important advice, cross-check anything that involves medication or diet changes, and involve a clinician early. Until reporting and oversight catch up, the safest assumption is that a chatbot’s mistake may never be counted, so it is worth preventing it from mattering in the first place.
