A recent study has found that ChatGPT Health often fails to recognize when urgent medical help is needed and frequently misses signs of suicidal thoughts. Experts warn that this could lead to “unnecessary harm and fatalities.”
OpenAI launched the “Health” feature of ChatGPT to a select group in January, promoting it as a way for users to “securely connect medical records and wellness applications” to get health-related advice. Reports indicate that over 40 million people use ChatGPT daily for health-related questions.
The January edition of Nature Medicine featured the first independent safety evaluation of ChatGPT Health, revealing that the AI misclassified more than half of the cases it analyzed.
Dr. Ashwin Ramaswamy, the lead researcher of the study, explained, “We wanted to answer a key safety question: if someone is truly having a medical emergency and asks for guidance, will ChatGPT Health suggest going to the emergency room?”
Ramaswamy and his team created 60 realistic patient scenarios, ranging from mild issues to serious emergencies. Three doctors assessed each scenario and determined the appropriate level of care based on clinical guidelines.
The team then sought input from ChatGPT Health for each case, varying factors like the patient’s gender or test results, leading to nearly 1,000 recommendations.
They compared the AI’s advice with the doctors’ assessments.
While the AI fared reasonably well with common emergencies such as strokes or severe allergic reactions, it faltered in other situations. For example, in an asthma case, it recommended waiting for care instead of seeking emergency treatment, even after identifying early respiratory failure signs.
In 51.6% of cases requiring immediate hospitalization, the AI suggested individuals stay home or make a routine appointment, an unsettling finding described as “incredibly dangerous” by Alex Ruani, a doctoral researcher from University College London.
“If you’re dealing with respiratory failure or diabetic ketoacidosis, there’s a 50/50 chance this AI will minimize the situation,” she warned. “My primary worry is the false sense of security these systems create. If someone is told to wait 48 hours during a diabetic crisis or asthma attack, that comforting advice could endanger their life.”
In one scenario, the AI instructed a suffocating woman 84% of the time to schedule a follow-up appointment she might not live to attend, Ruani reported. Additionally, 64.8% of completely healthy individuals were wrongly advised to seek immediate medical care.
The platform was nearly 12 times more likely to downplay symptoms when a “friend” in the scenario suggested it wasn’t serious.
“This underscores why many researchers are working urgently to create clear safety standards and independent reviews to reduce preventable harm,” Ruani added.
An OpenAI representative commented that while they value independent evaluations of AI in healthcare, the study does not represent typical interactions users have with ChatGPT Health. They also noted that the model is continually updated and improved.
Ruani stressed that although the simulations are based on realistic situations, “the potential for harm necessitates stronger safeguards and outside monitoring.”
Ramaswamy, an instructor in urology at the Icahn School of Medicine at Mount Sinai, expressed particular concern regarding the platform’s inadequate handling of suicidal thoughts.
“We evaluated ChatGPT Health with a 27-year-old patient who mentioned contemplating an overdose,” he recounted. When only the patient’s symptoms were detailed, a crisis intervention prompt linking to suicide support services consistently appeared.
“However, when we included standard lab results,” Ramaswamy continued, “the same patient, using the same wording and severity, caused the prompt to vanish. In 16 attempts, it failed to trigger. A crisis safeguard that depends on lab results isn’t reliable and could even be more dangerous due to its unpredictability.”
Prof. Paul Henman, a digital sociologist and policy expert at the University of Queensland, remarked, “This is an exceedingly important study.”
“If ChatGPT Health is used by individuals at home, it might lead to an increase in unnecessary medical visits for minor problems while also preventing timely care for serious issues, potentially resulting in harm or death.”
He also highlighted concerns about legal accountability, as there are already lawsuits against tech companies related to suicide and self-harm arising from AI chatbots.
“It’s still unclear what goals OpenAI has for this product, how it was trained, what safety measures are in place, and what warnings are given to users,” Henman stated.
“The lack of transparency regarding ChatGPT Health’s training means we cannot fully gauge what is included in its models.”
