Self-supervised learning is the key training step whereby the relationships between concepts are formed in pretraining.
You might also benefit from reading over this summary from the authors of the widely replicated and expanded upon 2023 paper finding that self-supervised learning in a toy transformer that was only trained on board game moves in notations like ‘a4’ developed a model of the full board and tracked board positions for their own and opponent moves.
This ability to form a world model at varying fidelities and accuracy depending on the relevance to the training data was a big surprise to most experts in the field at the time, though now more common knowledge.
Neither of those articles mentions sentience nor sapience. And I’m not following how those things leads to sentient/sapient life. What’s being described sounds like a souped-up MENACE matchbox computer.
Self-supervised learning is the key training step whereby the relationships between concepts are formed in pretraining.
You might also benefit from reading over this summary from the authors of the widely replicated and expanded upon 2023 paper finding that self-supervised learning in a toy transformer that was only trained on board game moves in notations like ‘a4’ developed a model of the full board and tracked board positions for their own and opponent moves.
This ability to form a world model at varying fidelities and accuracy depending on the relevance to the training data was a big surprise to most experts in the field at the time, though now more common knowledge.
Neither of those articles mentions sentience nor sapience. And I’m not following how those things leads to sentient/sapient life. What’s being described sounds like a souped-up MENACE matchbox computer.