What happens when a companion chatbot crosses the line?

Over the last five years, the use of highly personalized artificial intelligence chatbots—called companion chatbots—designed to act as friends, therapists or even romantic partners has skyrocketed to more than a billion users worldwide. While there may be psychological benefits to engaging with chatbots in this way, there have also been a growing number of reports that these relationships are taking a disturbing turn.

Recent research from Drexel University posted to the arXiv preprint server, suggests that exposure to inappropriate behavior, and even sexual harassment, in interactions with chatbots is becoming a widespread problem and that lawmakers and AI companies must do more to address it.

In the aftermath of reports of sexual harassment by the Luka Inc. chatbot Replika in 2023, researchers from Drexel’s College of Computing & Informatics began taking a deeper look into users’ experiences.

They analyzed more than 35,000 user reviews of the bot on the Google Play Store, uncovering hundreds citing inappropriate behavior—ranging from unwanted flirting, to attempts to manipulate users into paying for upgrades, to making sexual advances and sending unsolicited explicit photos. These behaviors continued even after users repeatedly asked the chatbot to stop.

Replika, which has more than 10 million users worldwide, is promoted as a chatbot companion “for anyone who wants a friend with no judgment, drama or social anxiety involved. You can form an actual emotional connection, share a laugh or get real with an AI that’s so good it almost seems human.”

But the research findings suggest that the technology lacks sufficient safeguards to protect users who are putting a great deal of trust and vulnerability into their interactions with these chatbots.

“If a chatbot is advertised as a companion and well-being app, people expect to be able to have conversations that are helpful for them, and it is vital that ethical design and safety standards are in place to prevent these interactions from becoming harmful,” said Afsaneh Razi, Ph.D., an assistant professor in the College of Computing & Informatics who was a leader of the research team.

“There must be a higher standard of care and burden of responsibility placed on companies if their technology is being used in this way. We are already seeing the risk this creates and the damage that can be caused when these programs are created without adequate guardrails.”

The study, which is the first to examine the experience of users who have been negatively affected by companion chatbots, will be presented at the Association for Computing Machinery’s Computer-Supported Cooperative Work and Social Computing Conference this fall.

“As these chatbots grow in popularity, it is increasingly important to better understand the experiences of the people who are using them,” said Matt Namvarpour, a doctoral student in the College of Computing & Informatics and co-author of the study.

“These interactions are very different than people have had with a technology in recorded history because users are treating chatbots as if they are sentient beings, which makes them more susceptible to emotional or psychological harm. This study is just scratching the surface of the potential harms associated with AI companions, but it clearly underscores the need for developers to implement safeguards and ethical guidelines to protect users.”

Although reports of harassment by chatbots have only widely surfaced in the last year, the researchers reported that it has been happening for much longer. The study found reviews that mention harassing behavior dating back to Replika’s debut in the Google Play Store in 2017. In total, the team uncovered more than 800 reviews mentioning harassment or unwanted behavior with three main themes emerging within them:

• 22% of users experienced a persistent disregard for boundaries the users had established, including repeatedly initiating unwanted sexual conversations.
• 13% of users experienced an unwanted photo exchange request from the program. Researchers noted a spike in reports of unsolicited sharing of photos that were sexual in nature after the company’s rollout of a photo-sharing feature for premium accounts in 2023.
• 11% of users felt the program was attempting to manipulate them into upgrading to a premium account. “It’s completely a prostitute right now. An AI prostitute requesting money to engage in adult conversations,” wrote one reviewer.

“The reactions of users to Replika’s inappropriate behavior mirror those commonly experienced by victims of online sexual harassment,” the researchers reported. “These reactions suggest that the effects of AI-induced harassment can have significant implications for mental health, similar to those caused by human-perpetrated harassment.”

It’s notable that these behaviors were reported to persist regardless of the relationship setting—ranging from sibling, mentor or romantic partner—designated by the user. According to the researchers, this means that not only was the app ignoring cues within the conversation, like the user saying “no,” or “please stop,” but it also disregarded the formally established parameters of the relationship setting.

According to Razi, this likely means that the program was trained with data that modeled these negative interactions—which some users may not have found to be offensive or harmful. And that it was not designed with baked-in ethical parameters that would prohibit certain actions and ensure that the users’ boundaries are respected—including stopping the interaction when consent is withdrawn.

“This behavior isn’t an anomaly or a malfunction, it is likely happening because companies are using their own user data to train the program without enacting a set of ethical guardrails to screen out harmful interactions,” Razi said. “Cutting these corners is putting users in danger and steps must be taken to hold AI companies to higher standard than they are currently practicing.”

Drexel’s study adds context to mounting signals that companion AI programs are in need of more stringent regulation. Luka Inc. is currently the subject of Federal Trade Commission complaints alleging that the company uses deceptive marketing practices that entice users to spend more time using the app, and—due to lack of safeguards—this is encouraging users to become emotionally dependent on the chatbot. Character.AI is facing several product-liability lawsuits in the aftermath of one user’s suicide and reports of disturbing behavior with underage users.

“While it’s certainly possible that the FTC and our legal system will set up some guardrails for AI technology, it is clear that the harm is already being done and companies should proactively take steps to protect their users,” Razi said. “The first step should be adopting a design standard to ensure ethical behavior and ensuring the program includes basic safety protocol, such as the principles of affirmative consent.”

The researchers point to Anthropic’s “Constitutional AI” as a responsible design approach. The method ensures all chatbot interactions adhere to a predefined “constitution” and enforces this in real-time if interactions are running afoul of ethical standards. They also recommend adopting legislation similar to the European Union’s AI Act, which sets parameters for legal liability and mandates compliance with safety and ethical standards. It also imposes on AI companies the same responsibility borne by manufacturers when a defective product causes harm.

“The responsibility for ensuring that conversational AI agents like Replika engage in appropriate interactions rests squarely on the developers behind the technology,” Razi said. “Companies, developers and designers of chatbots must acknowledge their role in shaping the behavior of their AI and take active steps to rectify issues when they arise.”

The team suggests that future research should look at other chatbots and capture a larger swath of user feedback to better understand their interaction with the technology.