What just happened? Another incident has taken place that illustrates the need to be careful what you tell AI. A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to “shoot up” the Sheriff’s office. After a human reviewer examined the statements, they were reported to police.
According to the arrest report, Carli Michelle Heller, of Bonita Springs, Florida, wrote on September 26 that she would attack the Sheriff’s office. She later said that she uses Anthropic’s chatbot like a “diary.”
Claude’s safety systems flagged the entry and it was escalated to a human reviewer. After deciding it was a credible threat, the reviewer reported it to law enforcement.
The company says it may share user information in limited emergencies if it believes disclosure is necessary to prevent death or serious physical injury.
Deputies identified Heller and visited her home. She was detained without incident before an LCSO intelligence detective took over the investigation.
Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Anthropic isn’t going to be taking any chances when it comes to anything it deems a potential threat. Last month, it was reported that OpenAI and Sam Altman are being sued by British Columbia over claims that the company could have prevented a mass shooting in the Canadian province.
The shooter, eighteen-year-old former pupil Jesse Van Rootselaar, had previously been flagged by OpenAI’s safety team for her conversations about gun violence, but OpenAI never alerted police because the conversations did not meet the threshold for legal referral.
In June, Florida also sued OpenAI and Altman, alleging that ChatGPT had contributed to real-world harms, including the 2025 Florida State University shooting.
The latest incident is another reminder to think before you enter something into a chatbot that could get you into trouble. It’s certainly not a private diary whose contents are for your eyes only.
Reports last month revealed that human contractors reviewing Microsoft Copilot’s image editor can see users’ prompts, uploaded photos and AI-generated edits. Documents show that some of those assignments contain sexual, disturbing or potentially illegal material, though the reviewers are not there to flag the content – only to assess whether the output is accurate.
Writing a threat is not illegal. Hell people vent all the time, threatening to do things they know damn well they never will. The law says transmitting a threat to another person is illegal because it can be used as intimidation or conspiracy. I think there’s a damn good argument that she didn’t knowingly transmit it to anyone. She thought she was writing in confidence to a non-person. And if it’s not her first time expressing outrage to AI, she would know it would only try to deescalate her.
Who knows how the case will go. I can see it going either way — probably depending on how much money she has. Laws are going to have to catch up to technology in this case, and it’s going to be bumpy. I definitely agree with erring on the side of caution. Private, local AI is far safer, but of course it’s less capable for now.
The article very much states that it is in fact illegal to write down a threat under Florida law.
I think this is a combination of the law being poorly written and the woman in question making a significantly questionable decision. Her actions have consequences (even if I think that law is problematic at best). There’s no reason for her to have had an expectation that Claude chats would be private (we’ve had repeated news reports that prove otherwise).
I don’t know that that’s my read here. The emphasis on the below quote from the article is mine.
It sounds like it’s not illegal to write the threat down, but it’s illegal to share it with others. If you do keep a private journal, or write your feelings onto slips of paper and burn them (idk if this is actually common, but it’s always been recommended to me after a breakup), that’s not illegal, even if you describe wanting to do something illegal. It’s the sort of hair splitting that wouldn’t normally be relevant, but could make a huge difference here.
Of course, she probably will be legally considered to have sent, posted, and transmitted it, so she’s SOL on that end, but she could credibly (imo) claim she was unaware that another person might view it.
She did transmit such a threat though. Which is against the law quoted in the article.
Yes but not in a way she would think another person may view it, most likely. Obviously IANAL but surely intent matters here?
If people actively realised someone can read all their chats, I’m sure these services would be far less popular.
And here’s the autism slop because I can’t think of the proper term for “actively realising” and do not have the energy to make it readable anymore:
What I mean is, I’m sure plenty of people know in the back of their mind that their data is not protected there, but it’s not the first, or second, or probably even tenth thing they think about when starting a chat.
Yeah half of the online gaming world would be in jail if this was consistently applied… intent must matter in a sane world.
https://vger.to/lemmy.world/comment/26171736
This is a link to my original comment on the subject. I did not say what you think I said in the comment you replied to or the original comment I linked. Her ignorance doesn’t absolve her of her actions or the consequences. Intent does matter to some extent but there have also been a lot of news reports about how data you share with corporate LLM’S isn’t private and so then the question becomes should she have known and not just if she did or didn’t know.
If someone else steals something and sells it to you or gifts it to you and you had no knowledge that it was stolen you can still be prosecuted under the law for receiving stolen goods. Your intent in the matter should be considered because there is a difference between knowingly receiving something you know is stolen and not reporting it, and just getting something without knowing it’s legal provenance and not reporting it. But you can still be charged either way.
In this case though an attempt was made to notify the user that her chats were not private in a EULA or TOS. They changed that TOS relatively recently (something like a year ago I think). At that time it looks like users would have been required to agree to an updated TOS with specific verbage about the sharing of chats that doesn’t look like it would have required users to read the entire TOS, just the agree popup. I linked the article for that below because while that was for allowing AI to train on user data, it should have been a clue that the chats were not private.
https://www.bgr.com/1953473/anthropic-train-claude-chats-how-opt-out/
https://www.anthropic.com/news/updates-to-our-consumer-terms
I appreciate the links! I still don’t think it’s a slam dunk either way, because even in Anthropic’s release, it never actually says that a human will be involved. “Training” could be anything, and while intent does not always matter, a threat is one of the few crimes where it really does.
Someone else (another user) has made a claim that she told Claude she had bought a weapon and that [police department] were on their “last chance”. It was this particular set of statements that caused Anthropic to call the police.
https://www.winknews.com/news/woman-arrested-after-ai-threat-against-lee-county-sheriffs-office-investigators/article_3d4c5915-7015-43c0-b86a-d7fa5eadf958.html
I don’t know if that was meant to reply to me, but I meant whether another human was involved in the training, not the threat. It’s definitely an imprudent thing to write, but I don’t know if a reasonable person would assume that another person would be able to read it. “Training” is a pretty nebulous concept, and the anthropic release doesn’t mention that humans are involved.
I’m not trying to say that this lady is making good decisions, but I probably wouldn’t vote that this was written so that another human might read it, if I were on the jury. Maybe my gut feeling’s off and everyone knows that training involves people (which I’m sure would come up during the trial), in which case I’d probably consider it a clear threat.
That’s why I said she was probably SOL on that side, but the other requirement for it to be considered an illegal threat is less cut and dry
With the intent that it may be seen by another person. That’s not the same. It’s not illegal to write a threat in your diary.
That being said the summary doesn’t include the fact that the following day she told Claude she had bought a gun and this was their “last chance,” and only then did Anthropic call the cops.
That seems exactly the sort of situation folks have been criticizing AI companies for not summoning help, and makes it seem far less likely that they called the cops over angry venting, but an actual intent to harm.