California Steps Up with New Healthcare AI Regulation for the New Year
As the new year begins, several new laws regulating artificial intelligence (AI) in the healthcare industry have gone into effect, prompting renewed debate over how the rapidly expanding technology should be used and constrained in medical settings.
The Rise of AI in Healthcare
Over the past decade, AI has become increasingly embedded in daily life, and healthcare has emerged as one of its most promising applications. With its quick access to a web of medical information, seemingly reliable advice, and assured decision-making, AI tools offer answers without the time, cost, or logistical barriers of seeing a doctor in person.
For many patients, that accessibility has become especially appealing amid growing frustration with the U.S. healthcare system, which, according to Gallup, 70% of Americans view as either in crisis or facing major problems.
Kate Large, a patient who uses AI for research on her health, noted, “Medicine has been the top of my mind, and AI too, as the top of medicine’s mind, because I’ve been sick. And AI has given me more answers than anything because I’ve had to wait three months to see doctors.”
Concerns Over Misinformation
According to OpenAI, of its over 800 million regular users, more than 40 million prompt ChatGPT about their healthcare every day. However, unlike traditional internet searches, AI tools often present their information with authority and even a tone of empathy, which can lead to misplaced trust.
Nitya Thakkar, a Ph.D. student at Stanford, articulated the risks, stating, “What happens if you are asking ChatGPT to help you write an essay, and it gets some facts wrong? You’ll just correct it; no big deal. But what happens if you’re using ChatGPT to answer a health question and it gets some wrong? That could have really, really big implications on a person’s health.”
California’s Legislative Response
In response to these concerns, many of the new bills that have just taken effect, led by California, directly address how AI is presented and used in medical contexts. Assembly Bill (AB) 489 aims to keep patients cautious about AI’s role in their medical lives. It prohibits developers and companies from implying that their AI systems provide professional or licensed medical advice by restricting the use of titles that could mislead users into believing the chatbots are qualified healthcare providers.
Thakkar emphasized the need for clarity, stating, “There’s a big anthropomorphization of these language models. When the AI speaks to you with empathy and uses ‘I’ statements, you start to interface with it like it’s a real doctor.”
Challenges in Defining Professional Titles
Confusion regarding professional titles extends beyond AI. Large pointed out that patients often cannot distinguish between healthcare professionals with doctoral degrees, leading to misunderstandings. For instance, a nurse with a Ph.D. might be mistaken for a medical doctor.
This debate reached the courts in 2023, when three nurses holding Doctor of Nursing Practice degrees sued the state, arguing for their right to use the title “doctor.” However, the court ruled that such titles could constitute misleading commercial speech, highlighting the need for clarity.
Transparency in AI Development
California lawmakers have also moved to regulate the data behind AI systems. Governor Gavin Newsom signed AB 2013, requiring developers to disclose the data used to train their AI systems, especially those related to healthcare. This legislative move aims to ensure that clinical evaluations and recommendations are backed by reliable data.
Michelle Mello, a Stanford professor, expressed concern about the potential for harm, stating, “It’s one thing to have AI give you bad investment advice, but we’re talking about uses of AI that could kill you.”
Broader Implications and the Push for Federal Regulation
As other states, including Illinois and Texas, implement similar legislation, a growing federal pushback has emerged. In late 2025, President Donald Trump issued an executive order to limit state AI regulations, indicating that such laws might face legal challenges.
Mello noted, “AI developers really hate the idea of state regulation of their products because it subjects them to potentially 50 different regulatory regimes.”
The Need for Caution
Large expressed the importance of regulation, stating, “I’d rather have, at minimum, some kind of regulation, whether it comes from the state, than no regulation.” Thakkar echoed this sentiment, emphasizing that safeguards are necessary for AI in healthcare.
As AI continues to expand rapidly in healthcare, its role has shifted from whether it can be used to how it should be controlled. While the technology promises efficiency and access in a strained medical system, it also raises critical concerns about accuracy and trust, particularly in a field where errors can have dangerous consequences.
