Ofcom, the UK’s communications regulator, has issued a stern warning to tech companies about the potential risks associated with user-created chatbots mimicking real and fictional people. This includes AI-generated avatars impersonating the deceased, such as Brianna Ghey, a British teenager tragically murdered last year, and Molly Russell, who took her own life in 2017. The warning follows growing concerns over the Online Safety Act, which is set to impose stricter rules on platforms hosting user-generated content.
The regulator’s intervention highlights a surge in distressing chatbot activity on platforms like Character.AI, where users have created bots based on real people—both alive and deceased. Ofcom’s clarification aims to ensure these platforms comply with the new digital safety laws, which aim to protect users, particularly minors, from harmful and illegal online content.
A Wake-Up Call for Digital Safety
Ofcom’s announcement was a direct response to increasing public pressure and concerns raised by digital safety campaigners. The issue first came to light when it was revealed that users on Character.AI had generated chatbots of Brianna Ghey, a transgender girl killed in an attack by two teenagers in 2023, and Molly Russell, whose death followed years of exposure to harmful online material. Both instances raised alarms about the potential emotional and psychological toll that such content could have on vulnerable individuals, particularly young people.
In its guidance, Ofcom clarified that platforms like Character.AI, which allow users to create their own chatbots—whether mimicking real people, fictional characters, or even deceased individuals—would fall under the scope of the Online Safety Act. The law, which is set to come into full force next year, is designed to regulate online platforms, requiring them to take greater responsibility for the content shared on their sites. Failure to comply could result in hefty fines of up to £18 million or 10% of global turnover, with the potential for sites to be blocked in severe cases.
What Does This Mean for Chatbot Platforms?
The new regulations target platforms hosting user-generated content. This includes social media networks, messaging apps, and chatbot platforms that allow users to create, share, and interact with avatars. Ofcom’s warning is aimed specifically at services like Character.AI, where chatbots can be created that mimic the personalities or appearances of people—sometimes without their consent or approval.
Ofcom made it clear that such content falls under the purview of the Online Safety Act. This means that platforms must now be proactive in monitoring and managing the user-generated content they host. The regulator emphasized that services enabling users to create chatbot clones of real or fictional characters—particularly ones that are deeply personal or sensitive—will need to comply with the legal framework designed to protect users.
This includes requiring platforms to implement measures to prevent harmful or illegal content from being created or shared. It also places the onus on platforms to ensure that there are clear channels for reporting and addressing harmful material, particularly when it concerns vulnerable groups such as minors.
- Key points to note:
- Platforms must have robust reporting systems in place.
- They must take action against harmful content created by users, including AI-generated avatars.
- Non-compliance could result in significant financial penalties or, in extreme cases, the removal of the platform from the UK market.
The Growing Impact of AI-Generated Content
Ofcom’s decision to step in follows a number of highly publicized and concerning incidents involving AI-generated avatars. The case of Brianna Ghey and Molly Russell is just the latest example of the potential dangers posed by these technologies. However, the problem extends beyond just the UK.
In the US, a similar case was reported where a teenager developed a relationship with a chatbot based on a popular Game of Thrones character. The relationship reportedly led to tragic consequences, underscoring the risks associated with these kinds of emotionally charged, AI-driven interactions.
While platforms like Character.AI have taken steps to moderate content, removing offensive or harmful bots when flagged, the larger question remains: what responsibility do these companies have when users create potentially harmful content on their platforms?
Ben Packer, a partner at the law firm Linklaters, commented on the situation, noting that Ofcom’s clarification highlights the complexity of the Online Safety Act. The act, which was first proposed years ago, is now having to adapt to the rapid rise of AI tools and the potential for misuse in ways that were not anticipated at the time.
The Role of the Molly Rose Foundation
In the wake of Ofcom’s warning, charities like the Molly Rose Foundation (MRF) have weighed in, supporting the regulator’s stance while calling for further clarity. The MRF, which was established by the family of Molly Russell, has long advocated for better protections against harmful online content.
The foundation’s representatives have applauded Ofcom’s guidance as a “clear signal” that AI-driven chatbots could cause significant harm, particularly when it involves vulnerable individuals or people no longer able to protect their own digital legacy. However, they’ve also urged that more specific guidelines be issued to clarify whether chatbot-generated content could be considered illegal under the existing legislation.
Ofcom has acknowledged these concerns and promised additional guidance on how the law will apply to chatbot content. The regulator is expected to issue further clarification in the near future to address any gaps in the current legal framework.
Character.AI’s Response
In response to the growing criticism, Character.AI has said it is committed to ensuring the safety of users on its platform. The company has taken down the controversial bots mimicking Brianna Ghey, Molly Russell, and the Game of Thrones character in question, but it remains clear that these incidents have raised significant concerns about how AI platforms can handle potentially harmful content.
Character.AI has stated that it actively moderates its platform and responds to user reports, but many are questioning whether these measures go far enough. As AI technologies continue to evolve, platforms like Character.AI will be under increasing scrutiny from regulators and digital safety groups to ensure they are not inadvertently facilitating harm.
