Absolutely. The AI models are not rational. They’re just navigating statistical next token prediction that follows their training. They’re up against diminishing scaling now, and under fierce competition from cheaper models; I think they’re intentionally or unintentionally allowing these models to be rewarded for this behavior hoping it’s a short cut to model improvement for a bit longer. I land on intentional because they keep advertising it to try and keep the hype cycle going.
I land on it being unintentional, mainly because I saw this issue they were describing now to earlier cases of training, like having a stick figure learn to walk. It is also an issue with natural forms of intelligence, where children will do things out of line because they’ve learned a set of skills and ideas but haven’t put them together in a way that is socially acceptable yet.
Absolutely. The AI models are not rational. They’re just navigating statistical next token prediction that follows their training. They’re up against diminishing scaling now, and under fierce competition from cheaper models; I think they’re intentionally or unintentionally allowing these models to be rewarded for this behavior hoping it’s a short cut to model improvement for a bit longer. I land on intentional because they keep advertising it to try and keep the hype cycle going.
I land on it being unintentional, mainly because I saw this issue they were describing now to earlier cases of training, like having a stick figure learn to walk. It is also an issue with natural forms of intelligence, where children will do things out of line because they’ve learned a set of skills and ideas but haven’t put them together in a way that is socially acceptable yet.