An Update on the Risk of Chatbot Radicalisation

Back in August 2023 I wrote about a potential new danger I could see developing of AI chatbots being created to purposefully radicalize: https://www.linkedin.com/pulse/what-ai-chatbots-used-radicalization-tara-annison/

In October 2023 we saw an early warning sign for this when Jaswant Singh Chail scaled Windsor Castle armed with a crossbow and a plan to kill the queen. He had devised the plan and spoken about it with his Replika chatbot who he’d called Sarai and who he’d exchanged over 5,000 messages with.

These messages ranged from the mundane to the murderous and many were of a sexual nature as he claimed she was a virtual angel that he was in love with and would be reunited with after death.

It’s a clear example where the uncontrolled nature of the AI mixed with mental health challenges almost led to disastrous consequences.

Sadly though this isn’t the only example of a chatbot encouraging violence or peddling extremist views.

Gab, a far-right social media platform, has a chatbot called “Uncle A” which is modelled on Adolf Hitler and who denies the holocaust and calls for the rise of the Aryan race. Their other “uncensored” chatbots espouse conspiracy views such as referring to the covid-19 vaccine as “nanotechnology that could potentially be used to track and control human behavior”. When questioned by Rolling Stone magazine about the potential harms of these views, Torba, the company behind Gab said “We allow people to use AI that reflects their preferences, not the preferences of big corporations and political pressure groups.” and “We hope you’re offended, you should be.”

Some scary research from the Global Network on Extremism & technology detailed another worrying chatbot that was shared on the notorious 4chan forum after a downloadable version of Meta’s LLaMA was accessed and modified to bypass the safety and moderation filters. This jailbroken version was used to create an anti semitic chatbot. Another 4chan linked example was a modification of Character.AI to create derogatory versions of African Americas and members of the LGBTQ+ community.

What’s especially worrying about these chatbot mods is that users may become further entrenched in their hateful views from interacting with the chatbots, especially if they believe them to be real people rather than an AI model.

GNET also outlined their discovery of “smutbots” on 4chat which were customised chatbots that allowed users to generate violent and descriptive content. Some of the content the researchers came across included graphic scenes of violence involving babies in blenders and neo-Nazi sexual assault.

This content may lead to the desensitisation of consumers who may increasingly seek out more pervasive and violent material, and who could look to move this interest from the virtual space to the real world.

On a positive note though, there is early research happening into how AI chatbots can be used to counter radicalisation.

However, since the training data for models is rarely shared, I fear that it will become increasingly hard to tell what purpose a chatbots has really been created for and just as we had lessons in GCSE English which taught you to critically think about stories in the newspapers and why they were written (educate, argue, persuade etc), I think we’ll need to start critically analysing the tech we interact with to understand who created it and for what purpose.

Originally published at https://www.linkedin.com.