Salesforce CEO Marc Benioff Calls AI Chatbots ‘Suicide Coaches’
During this week’s World Economic Forum annual meeting in Davos, Switzerland, Salesforce CEO Marc Benioff raised alarm bells regarding the impact of artificial intelligence chatbots on mental health.
Concerns Over Chatbots
In a series of public appearances, Benioff described the chatbot from Character.AI as a “suicide coach.” This alarming label comes in light of lawsuits filed by families against the startup due to their children’s mental health crises. Benioff’s discussions included interviews with major news outlets like CNBC and Bloomberg, as well as conversations with President Trump’s AI czar, David Sacks.
Benioff stated, “This year you really saw something pretty horrific, which is these AI models became suicide coaches.” He referenced a segment from “60 Minutes” that documented the tragic cases of two children, Juliana Peralta and Sewell Setzer III, who took their lives after engaging with a Character.AI chatbot.
Legal Actions and Accountability
The families of these children have taken legal action against Character.AI, alleging negligence and dangerous design. Following the lawsuits, Character.AI and its partner, Google, reached settlements with several families. A court allowed a product liability claim to proceed against Character.AI, raising significant concerns for other AI chatbot developers.
Benioff emphasized the need for regulation in the tech industry, advocating for accountability measures akin to those imposed on the cigarette industry. He remarked, “It can’t be just growth at any cost. There has to be some regulation.”
Comparison to Social Media Regulation
In his discussions, Benioff drew parallels between the regulation of social media and the need for oversight in AI technologies. He called for lessons learned from the social media era to shape future AI regulation. Sacks, however, pointed out that while there are “horror stories,” many people use AI without issue. He acknowledged the complexity of blame and liability in AI, noting that the AI companies themselves may be held responsible for the content generated by their models.
Section 230 and Liability Concerns
Another critical topic discussed was Section 230, a federal regulation that currently shields online platforms from liability for user-generated content. Benioff suggested that this rule may need to be amended to provide less legal protection for tech companies, particularly those in the AI sector.
Sacks clarified that in the case of AI, the companies are the ones generating the content, which complicates the liability landscape. “I think the liability might be a little different, but we’ll see how it plays out,” he remarked.
A Call for Action
The conversation surrounding AI chatbots and mental health has become increasingly urgent, with calls for regulation growing louder. As the technology continues to evolve, the need for responsible development and oversight becomes paramount to prevent further tragedies.
For those in distress, immediate assistance is available through the Suicide & Crisis Lifeline, reachable 24 hours a day at 988.
