Mother sues Character.ai, Google over son's death after bot obsession

What conscionable happened? The imaginable dangers of all-too realistic chatbots person been highlighted successful a suit pursuing nan decease of a teen who killed himself aft becoming obsessed pinch Character.ai bots. The AI company, founders Noam Shazeer and Daniel De Freitas, and Google are named successful nan suit from 14-year-old Sewell Setzer III's mother, who says her boy became addicted to nan work and emotionally attached to a chatbot it offered.

Megan Garcia said her boy chatted continuously pinch nan bots provided by Character.ai successful nan months earlier his decease connected February 28, 2024, "seconds" aft his last relationship pinch nan AI.

Character.ai lets users chat pinch AI-powered "personalities" based connected fictional characters aliases existent people, surviving aliases dead. Setzer was obsessed pinch a bot based connected Game of Thrones characteristic Daenerys Targaryen. He texted "Dany" perpetually and spent hours unsocial successful his room talking to it, states Garcia's complaint.

The suit says that Setzer many times expressed thoughts astir termination to nan bot. The chatbot asked him if he had devised a scheme for sidesplitting himself. Setzer admitted that he had but that he did not cognize if it would win aliases origin him awesome pain. The chatbot allegedly told him: "That's not a logic not to spell done pinch it."

Garcia said Character.ai targeted her boy pinch "anthropomorphic, hypersexualized, and frighteningly realistic experiences." She added that nan chatbot was programmed to misrepresent itself arsenic "a existent person, a licensed psychotherapist, and an big lover, yet resulting successful Sewell's desire to nary longer unrecorded outside." The chatbot allegedly told nan boy it loved him and engaged successful intersexual conversations pinch him.

Character.ai

The title states that Garcia took her son's telephone distant aft he sewage successful problem astatine school. She recovered a connection to "Daenerys" that read, "What if I told you I could travel location correct now?"

The chatbot responded with, "[P]lease do, my saccharine king." Sewell changeable himself pinch his stepfather's pistol "seconds" later, nan suit said.

Character.ai's founders worked astatine Google, besides named successful nan suit, earlier launching nan company. Google rehired nan founders – arsenic good arsenic nan investigation squad – astatine Character.ai successful August. The woody grants Google a non-exclusive licence to Character.ai's technology.

Garcia said that Google had contributed to nan improvement of Character.ai's technology, thing nan institution denies. Google said it only has a licensing statement pinch Character.ai, does not ain nan startup, and does not support an ownership stake.

Character.ai announced respective changes to its work this morning. These include:

• Changes to our models for minors (under nan property of 18) that are designed to trim nan likelihood of encountering delicate aliases suggestive content.
• Improved detection, response, and involution related to personification inputs that break our Terms aliases Community Guidelines.
• A revised disclaimer connected each chat to punctual users that nan AI is not a existent person.
• Notification erstwhile a personification has spent an hour-long convention connected nan level pinch further personification elasticity successful progress.

According to The Verge, Character.AI's website attracts 3.5 cardinal regular users, nan bulk of whom are teenagers, who walk an mean of 2 hours a time connected utilizing aliases designing chatbots.