inari@piefed.zip to Programming@programming.devEnglish · 2 days agoI built a programming language that LLMs can't useblog.picheta.meexternal-linkmessage-square36linkfedilinkarrow-up182arrow-down128
arrow-up154arrow-down1external-linkI built a programming language that LLMs can't useblog.picheta.meinari@piefed.zip to Programming@programming.devEnglish · 2 days agomessage-square36linkfedilink
minus-squaredan@upvote.aulinkfedilinkarrow-up55arrow-down2·2 days agoThis is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.
minus-squareOldMrFish@feddit.dklinkfedilinkarrow-up34arrow-down3·edit-22 days agoAlternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…
minus-squaredan@upvote.aulinkfedilinkarrow-up7·2 days agoNot only was Claude not trained on it, but the language is actively trying to be hostile towards it.
minus-squareMirodir@discuss.tchncs.delinkfedilinkarrow-up24arrow-down2·2 days agoAnd that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.
minus-squarealdhissla@beehaw.orglinkfedilinkarrow-up8·2 days agoAs well-intentioned as this might be, I doubt they understand how trained models work.
minus-squareatzanteol@sh.itjust.workslinkfedilinkEnglisharrow-up1arrow-down8·2 days agoHow the hell is this a “self own”?
minus-squarealdhissla@beehaw.orglinkfedilinkarrow-up7·2 days agoThe premise was that this language is significantly different from other languages in terms of being understood and replicated by an LLM. The result showed that an LLM does exactly as well with it as you would expect.
This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.
Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…
Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.
And that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.
As well-intentioned as this might be, I doubt they understand how trained models work.
How the hell is this a “self own”?
The premise was that this language is significantly different from other languages in terms of being understood and replicated by an LLM.
The result showed that an LLM does exactly as well with it as you would expect.