"Do you watch the news? Do you listen to the radio? Do you read anything about what's going on in the world?” one of the judges asked the lawyer, who submitted false testimony and witnesses after using ChatGPT.
“It’s little comfort to know that my stupidity is what brings us together this afternoon,” New Mexico attorney Stephen D. Aarons said in a hearing last month before five judges. He “assumed” ChatGPT would generate a “bulletproof summary of proceedings.” Instead, ChatGPT generated false testimony from “wholly fabricated witnesses,” according to an order of direct contempt from the New Mexico Supreme Court.
Aarons’ reasoning for trusting ChatGPT, he said in the August hearing, was that he’d heard of doctors using AI for medical research.
Aarons’ reasoning for trusting ChatGPT, he said in the August hearing, was that he’d heard of doctors using AI for medical research.
I don’t think the AI being used in medical research is ChatGPT (or any LLM for that matter), but if they are then that’s concerning… He’s still a dumbass though.
I had to check the date of the article to make sure it was recent. How a lawyer hasn’t heard of the several times other lawyers have had issues with ChatGPT I have no idea. There’s no way this guy didn’t perjure himself be feigning ignorance. I feel like the bar needs to send out a written notice requiring some kind of acknowledgment to all lawyers so they can no longer claim ignorance if this happens to them.
In a quote in the article, one of the judges even points this out to him rather harshly.
“So, Counsel, do you watch the news? Do you listen to the radio? Do you read anything about what’s going on in the world?” one of the judges asked Aarons. “Because the problem with lawyers relying on AI hallucinations is an above-the-fold story every single day.”
…
“So, either you buried your head in the sand. And that’s a choice to do that. An intentional choice to be uninformed,” the judge continued to Aarons. “Or, you took a gamble. And neither of those are consistent with the code of conduct. So I’m really struggling with your response, saying, ‘Well, I didn’t understand. I didn’t know about hallucinations.’ My 13-year-old nephew knows about hallucinations. My 75-year-old stepmother knows about hallucinations. So, either you made a decision not to be aware of what’s happening in the world in using this tool appropriately, which is a willful choice. Or you decided to roll the dice.”
The article also provides links to five other cases where lawyers used an LLM and filed AI hallucinations as fact. This bozo is clearly in the find out phase.
If you’re one of the people tasked with enforcing the law, then yes you are not expected to know it. If you are a mere citizen, then you are expected to know the law inside and out.
In theory, no. In practice, yes. I think it’s generally fine for people to be able to use discretion in a lot of cases. That discretion is not always evenly applied, but there are plenty of laws that the average person doesn’t know and has no reason to even think about. I’d rather live in a world where the justice system is able to give people leniency and the benefit of the doubt than otherwise. If one of your brake lights is out it’s possible you have no idea. Technically it’s illegal to be driving. Most people would agree that it’s a good thing that you can be given a warning or something instead of have to face the same penalty as a person knowingly driving with the same issue.
Yeah I heard of a case like this at least a year ago. I was wondering if we were reposting an old article, but no. That makes it worse, and explains the judge’s reaction.
AI is already being used for writing faux papers in journals, stem and medical ones. its just now they are getting caught. even using it for “medical scans” is highly suspect and likely wont see the light of day. a version of its being used in dental scans already by some.
Some people are smart enough to be a lawyer yet still really fucking stupid.
You don’t have to be smart to become a lawyer, lots of memorizing and some street smarts will do just fine
I don’t think the AI being used in medical research is ChatGPT (or any LLM for that matter), but if they are then that’s concerning… He’s still a dumbass though.
I had to check the date of the article to make sure it was recent. How a lawyer hasn’t heard of the several times other lawyers have had issues with ChatGPT I have no idea. There’s no way this guy didn’t perjure himself be feigning ignorance. I feel like the bar needs to send out a written notice requiring some kind of acknowledgment to all lawyers so they can no longer claim ignorance if this happens to them.
In a quote in the article, one of the judges even points this out to him rather harshly.
…
The article also provides links to five other cases where lawyers used an LLM and filed AI hallucinations as fact. This bozo is clearly in the find out phase.
You can claim ignorance for breaking the law?
If you’re one of the people tasked with enforcing the law, then yes you are not expected to know it. If you are a mere citizen, then you are expected to know the law inside and out.
In theory, no. In practice, yes. I think it’s generally fine for people to be able to use discretion in a lot of cases. That discretion is not always evenly applied, but there are plenty of laws that the average person doesn’t know and has no reason to even think about. I’d rather live in a world where the justice system is able to give people leniency and the benefit of the doubt than otherwise. If one of your brake lights is out it’s possible you have no idea. Technically it’s illegal to be driving. Most people would agree that it’s a good thing that you can be given a warning or something instead of have to face the same penalty as a person knowingly driving with the same issue.
You very explicitly cannot. The judge may take it into account when sentencing though.
Yeah I heard of a case like this at least a year ago. I was wondering if we were reposting an old article, but no. That makes it worse, and explains the judge’s reaction.
There are ways that LLMs are being used in science, AND a human mind needs to vet those findings for validity. smh.
Using AI for medical research probably doesn’t mean using a publicly available chat bot.
AI is already being used for writing faux papers in journals, stem and medical ones. its just now they are getting caught. even using it for “medical scans” is highly suspect and likely wont see the light of day. a version of its being used in dental scans already by some.