The explanations in the thread seem to me to be missing the middle or evading the heart of the problem. Zoomed out: an optimization target at level of personality. Zoomed in: a circuit diagram of layers. But those layers with billions of weights are pretty much Turing complete.
Unfortunately, I don’t think anyone has much idea how all those little learned computations are make up said personality. My suspicion is there isn’t going to be an *easy* way to explain what they’re doing. Of course, I’d be relieved to be wrong here!
This matters because the analogy in the thread between averaged faces and LLM outputs is broken in an important way. (Nearly) every picture of a face in the training data has a nose. When you look at the nose of an averaged face, it’s based very closely on the noses of all the faces that got averaged. However, despite the size of the training datasets for LLMs, the space of possible queries and topics of conversation is even vaster (it’s exponential in the prompt-window size, unlike the query space for the average faces which are just the size of the image).
As such, LLMs are forced to extrapolate hard. So, I’d expect that which particular generalizations they learned, hiding in those weights, to start to matter once users start poking them in unanticipated ways.
In short, if LLMs are like averaged faces, I think they’re faces that will readily fall apart into Shoggoths if someone looks at them from an unanticipated or uncommon angle.
Another disanalogy is in how GPT-4 writes novel quines without thinking out loud in the context window. It still needs to plan it, so the planning probably happens with layers updating the residual stream, the way it could’ve happened with thinking step by step, but using the inscrutable states of the network instead of tokens. Thinking step by step in tokens imitates humans from its training data, but who knows how the thinking step by step in the residual stream works.
Thus shoggoths might be the first to wake up, because models might already be training on this hypothetical alien deliberation in the residual stream, while human-imitating deliberation with generated tokens is still not being plugged back into the model as training data. This hypothesis also predicts future LLMs that are broadly trained the same as modern LLMs, still look non-agentic and situationally unaware like modern LLMs, but start succeeding in discussing advanced mathematics, because the necessary process of studying it (inventing and solving of exercises that are not already in the training set) might happen by alien deliberation within the residual stream during the training process, while SSL looks at episodes that involve related theory.
In Defense of the Shoggoth Analogy
In reply to: https://twitter.com/OwainEvans_UK/status/1636599127902662658
The explanations in the thread seem to me to be missing the middle or evading the heart of the problem. Zoomed out: an optimization target at level of personality. Zoomed in: a circuit diagram of layers. But those layers with billions of weights are pretty much Turing complete.
Unfortunately, I don’t think anyone has much idea how all those little learned computations are make up said personality. My suspicion is there isn’t going to be an *easy* way to explain what they’re doing. Of course, I’d be relieved to be wrong here!
This matters because the analogy in the thread between averaged faces and LLM outputs is broken in an important way. (Nearly) every picture of a face in the training data has a nose. When you look at the nose of an averaged face, it’s based very closely on the noses of all the faces that got averaged. However, despite the size of the training datasets for LLMs, the space of possible queries and topics of conversation is even vaster (it’s exponential in the prompt-window size, unlike the query space for the average faces which are just the size of the image).
As such, LLMs are forced to extrapolate hard. So, I’d expect that which particular generalizations they learned, hiding in those weights, to start to matter once users start poking them in unanticipated ways.
In short, if LLMs are like averaged faces, I think they’re faces that will readily fall apart into Shoggoths if someone looks at them from an unanticipated or uncommon angle.
Another disanalogy is in how GPT-4 writes novel quines without thinking out loud in the context window. It still needs to plan it, so the planning probably happens with layers updating the residual stream, the way it could’ve happened with thinking step by step, but using the inscrutable states of the network instead of tokens. Thinking step by step in tokens imitates humans from its training data, but who knows how the thinking step by step in the residual stream works.
Thus shoggoths might be the first to wake up, because models might already be training on this hypothetical alien deliberation in the residual stream, while human-imitating deliberation with generated tokens is still not being plugged back into the model as training data. This hypothesis also predicts future LLMs that are broadly trained the same as modern LLMs, still look non-agentic and situationally unaware like modern LLMs, but start succeeding in discussing advanced mathematics, because the necessary process of studying it (inventing and solving of exercises that are not already in the training set) might happen by alien deliberation within the residual stream during the training process, while SSL looks at episodes that involve related theory.