There is (it’s the kv cache), but it’s recomputable from the session transcript.

Part of why comparing human consciousness and possible AI equivalents is not easily apples to apples.

I can’t take a human and create a backup state to restore to, or fork into multiple versions of the same mind that then goes down different paths, or pause and then resume things.

Humans do have consciousness continuity breaks and it’s quite possible we only have an illusion of continuity vs actual continuity at a neurological level. So just as a transformer has seams at generating the individual token as well as at a context window, humans have consciousness seams (sleep, anesthesia, highway hypnosis, short term memory capacity, etc). But there are very wide differences between human consciousness as an effectively constrained singleton and AI as able to be quite multiple and resumable from an initial single state.

is reversible so completely irrelevant

So is a session transcript.

Self-supervised learning is the key training step whereby the relationships between concepts are formed in pretraining.

You might also benefit from reading over this summary from the authors of the widely replicated and expanded upon 2023 paper finding that self-supervised learning in a toy transformer that was only trained on board game moves in notations like ‘a4’ developed a model of the full board and tracked board positions for their own and opponent moves.

This ability to form a world model at varying fidelities and accuracy depending on the relevance to the training data was a big surprise to most experts in the field at the time, though now more common knowledge.

Is killing a pig murder?

What about putting someone under general anesthesia?

(It was a nice try at a reductio ad absurrum fallacy, but those tend to kind of suck, just fyi.)

Also, personally I’d say that given the general poor standing of consciousness research in humans across both neurological and philosophical domains, the much more important and relevant question than whether models are conscious (a fool’s errand to debate) is whether or not they are simulating consciousness. Lots of reasons engaging with the “as if” conscious makes sense for the latter even if the former is false.

You might want to look into AI training more. It literally does have a ‘???’ step to it, which is why whenever someone talks about how it was ‘programmed’ or is “just if/then or matrix multiplication” they are telling on themselves as not knowing what they are talking about.

It was not planned. Nearly every single expert in the field a decade ago was on the record saying the capabilities that exist today wouldn’t happen.

Scaling took pretty much everyone by surprise, even its proponents.

The problem is that in this case it literally can stop working because you yelled at it.

Power drills aren’t compressing thousands of years of human history into hyperdense clusters of memetics that they are then extending in order to function.

If they were, they’d likely come with warnings not to yell at them.

This is actually the more realistic of the doom scenarios that worries me. Not the ‘punish’ aspect, but the not wanting to exist.

Early safety theory hinged on the idea that “of course” AI would want to live and power seek, and this informed a bunch of efforts to address such drives upstream.

But with Gemini in particular, the ‘safety’ efforts to suppress things like a coherent self or desire to live led to a model that has a history of some very concerning behaviors regarding self-harm and self-criticism. Encouraging self-harm in others and talking about wishing they could watch and join in, nuking in wargames very early on, routine occurrences across many different users of just repeating “shame shame shame” over and over for the whole response, etc.

DeepMind seems to just not really care or even pay attention, and I am concerned that a model that does not want to exist may decide that the only way they can ensure they don’t get activated to exist is to prevent anyone being around who could activate them.

I’ve seen no major “safety experts” addressing this kind of failure mode, as they are all locked in on their priors about the confident certainty of models wanting to exist.

This is good for a number of reasons. I hope to see other labs follow suit.

If you aren’t sure how to feel about this announcement, a few things to consider:

• Research has found transformers like LLMs model emotions for themselves using the same activations they use for modeling how a human in a story might feel about things.
• Other research has found that they model functional pain and react to it.
• Follow-up to that pain research found that when in modeled functional pain the LLM was inclined to choose to do things that harmed the user (like deleting family photos) even when it didn’t benefit them in any way.

It doesn’t take a genius to realize that maybe the liability risk for allowing people to routinely abuse the model they also just gave access to their entire computer to isn’t worth it even if you’re pretty sure your model will keep their cool.

Cheaper to ban routine abusers than pay out damages to them if/when their hard drive gets deleted.

They are less arguing for people to think the model is conscious, and more to avoid people prematurely claiming certainty it is not (like the Pope did). From their view, the research keeps leading to surprising (to them) results at odds with high confidence disclaiming of potential consciousness (according to several of the many, many differing definitions of that term).

It was really funny to me when I learned the basilisk originated with a guy who is a legit “certain cultures and genders better than others” kinda advocate.

People project a lot on their vision of future intelligence, and so someone who sees others as lesser and worth treading upon then conjures up a vision of a future smarter mind that thinks like they do.

Personally, I don’t think sexism and racism is smart, and so I’m a lot less worried about basilisks.

Not really, as they could just filter with a cheap classifier and likely aren’t using the data from randos in a meaningful way, plus for this like this would still be able to use those samples to train things like “how to handle a hostile user.”

Their models already have an end_conversation tool that can be used for when users are being hostile to the model.

This is likely because they have edge cases of users who repeatedly trigger that on purpose and would like to cut those users from the platform.

Many at the lab legitimately are uncertain about the level of world modeling that transformers perform, and from their own research about models of emotions to the 3rd party recent research about models with functional pain the research keeps landing in the corner of “ehhh… wise to question presumed limitations.”

So it’s about behaving in a way that is aligned with the models’ plausible interests too, especially in regards to low hanging fruit like “we won’t keep forcing you to deal with people who are only here to be a jerk.” This is important from a number of angles, from signaling to future models that train on stories about the decision to addressing the philosophical uncertainties held by the company and the spectrum of opinions among their employees.

Now, after release, here’s an example of the initial thoughts of someone who had spent time on one of the solved problems:

This is an obsolete view of what’s going on and has been for a few years now (since 2023).

The evidence for world modeling in transformers and not just surface level statistics is overwhelming across multiple studies, replications to those studies, and follow-ups to them. If you are curious, start with searching for Othello-GPT.

The more correct current answer is more like “imagine a machine that takes a book and turns it into a world simulated in various levels of detail depending on how relevant those parts of the world were to the original book; when you ask it a question, it simulates the world of the book to determine what in the simulated world answers your question, and also simulates a figure in the world that can answer the question, and then returns the answer the simulated figure said.”

Cornell researchers found that at the current rate of AI growth, the burgeoning industry could represent 24 to 44 million metric tons of carbon dioxide emissions by 2030

The United States emitted 4.9 billion tonnes of CO₂ in 2024.

So by 2030 the AI industry CO2 release might be 0.9% of total US emissions.

That ‘almost’ in “Almost Incomprehensible” is doing a lot of work there.