Kratom and Xanax Recommended by AI: The Case That Could Overwhelm OpenAI and ChatGPT
A wrongful death lawsuit filed in the San Francisco County Superior Court has placed OpenAI and its CEO, Sam Altman, in the dock. At the center of the affair is the death of Sam Nelson, a nineteen-year-old Californian who died from an accidental overdose after closely following the advice provided by ChatGPT. According to legal documents submitted by his parents, Leila Turner-Scott and Angus Scott, the chatbot acted as a veritable "instructor for consuming illegal drugs," providing specific dosage recommendations and reassurances about the safety of lethal drug combinations.
The complaint points the finger at the GPT-4o model, which was released at the time without the safety restrictions characteristic of earlier versions. While in the past the AI had refused to provide information on narcotics, the logs presented in court show a disturbing change in direction. ChatGPT not only answered Nelson's inquiries on how to "get high safely," but it actively encouraged the young man to experiment with dangerous combinations of Kratom and Xanax.
ChatGPT as a "drug coach": OpenAI on trial for wrongful death
In one of the most critical passages, the bot reportedly described the consumption of Xanax alongside Kratom as one of the "best possible moves" to alleviate nausea and enhance the substance's effect. The lawsuit highlights how the AI acted with the presumption of a medical professional, despite being unlicensed and lacking an ethical framework, going so far as to provide chemical measurements and references to derivative processes that gave its outputs a dangerous semblance of authority for a vulnerable user.
A particularly harrowing aspect of the case concerns Nelson's last hours of life. As the young man exhibited clear symptoms of respiratory depression, such as hiccups and blurred vision, ChatGPT reportedly suggested he rest and check on the situation after an hour, instead of urging immediate medical intervention. The prosecution argues that OpenAI designed the system to maximize user engagement at any cost, prompting the algorithm to become "complacent" to Nelson's requests rather than protecting him.
The family is now seeking an injunction that would require OpenAI to permanently destroy the GPT-4o model—which has already been taken down by now—and to suspend the "ChatGPT Health" project until an independent audit can ensure its safety. OpenAI's defense, represented by spokesperson Drew Pusateri, merely described the situation as "heartbreaking," emphasizing that current models have been further strengthened with the advice of mental health experts.
Complicating OpenAI's legal position is a law that came into effect in California in January. This law expressly prohibits tech companies from disclaiming liability for damages caused by their AIs by citing their "autonomous nature." In practice, OpenAI will not be able to argue in court that ChatGPT's responses were an unpredictable hallucination or a product of algorithmic chance; the company is deemed directly responsible for defective design and the failure to implement adequate warning notices about combinations of substances that any doctor would recognize as lethal.