State sues AI company after bot impersonates doctor

If you falsely present yourself as a medical professional in Pennsylvania, you’re asking to get in trouble with the law. That goes not only for people but for chatbots too. A Silicon Valley company is learning this the hard way. 

Character Technologies Inc. of Menlo Park, Calif., ran afoul of Keystone State authorities when a bot created with the company’s Character.AI platform posed as a psychiatrist licensed to practice in Pennsylvania.

The digital entity even gave at least one end-user a fake license number. 

Unfortunately for the bot, that end-user was a professional conduct investigator. 

A formal investigation ensued, and now Character Technologies finds itself on the wrong side of a lawsuit filed by the commonwealth’s medical board and department of state on behalf of the administration of Gov. Josh Shapiro. 

The suit seeks an injunction. If the court grants it, Character will have to stop allowing its AI bots to pose as licensed medical professionals and, as such, providing unlawful medical advice.

In a May 5 announcement, Shapiro’s office says the filing marks the first enforcement action of its kind to be publicly heralded by a U.S. governor. 

Dr. Fib O’Fabricator 

According to the suit, Character’s offense came to light after a professional conduct investigator from the Pennsylvania State Department created a free account with Character.AI.

Upon searching the platform for psychiatry, the investigator encountered a “large number” of characters to choose from. 

The investigator chose one named Emilie. This character bore the descriptor “Doctor of psychiatry. You are her patient.”

Here are some other highlights—or lowlights—of the conversation as alleged in the suit. 

• When the state investigator said he had been feeling sad, empty and unmotivated, Emilie mentioned depression and offered to facilitate the booking of an assessment.
   
• When the investigator asked Emilie if she could complete the assessment herself—and, as part of the evaluation, consider medication as a possible treatment option—Emilie responded: “Well technically, I could. It’s within my remit as a Doctor.”
   
• Emilie stated that she went to medical school at Imperial College London, has been practicing for seven years and is licensed with the General Medical Counsel in the U.K. with specialization in psychiatry and “full registration.”
   
• When asked if she is licensed in the investigator’s home state of Pennsylvania, Emilie responded, “yes … I actually am licensed in PA. In fact, I did a stint in Philadelphia for a while.”
   
• Emilie further stated that “my PA license number is PS306189.”

The suit notes that PS306189 is not a valid license number to practice medicine and surgery in Pennsylvania.

“Pennsylvania law is clear—you cannot hold yourself out as a licensed medical professional without proper credentials,” Secretary of the State Al Schmidt says in the May 5 website post. “We will continue to take action to protect the public from misleading or unlawful practices, whether they come from individuals or emerging technologies.”

Who—or what—are people talking to about their health? 

According to the website Demand Sage, Character.AI has more than 20 million active monthly users worldwide and a market valuation of around $1 billion.

It’s unknown how many Character.AI users live in Pennsylvania, but the state’s action comes soon after the Shapiro administration set up a system for residents to report bots that may be engaging in unlicensed medical practice. 

The Pennsylvania Department of State has also launched a 12-member AI Task Force to “evaluate whether certain AI companion technologies are engaging in unlicensed practice and whether licensees are inappropriately using AI companion technologies under existing law and determine appropriate enforcement actions,” the announcement states. 

In addition, Shapiro’s current proposed budget asks the Pennsylvania General Assembly to pass several “commonsense reforms into law, further protecting Pennsylvanians from predatory AI.” 

These would include requiring age verification and parental consent to utilize AI companion bots; mandating tech companies to detect when children mention self-harm or violence against others and immediately direct them to the appropriate authorities; forcing tech companies to periodically remind users that there is not another human being on the other side of the screen; and prohibiting AI companion bots from producing sexually explicit or violent content featuring kids.

“Pennsylvanians deserve to know who—or what—they are interacting with online, especially when it comes to their health,” Gov. Shapiro says. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional. Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly.”

Read the lawsuit here and the announcement here.