Epistemic status: 50% sophistry, but I still think it’s insightful since specifically aligning LLMs needs to be discussed here more.
I find it quite interesting that much of current large language model (LLM) alignment is just stating, in plain text, “be a helpful, aligned AI, pretty please”. And it somehow works (sometimes)! The human concept of an “aligned AI” is evidently both present and easy to locate within LLMs, which seems to overcome a lot of early AI concerns like whether or not human morality and human goals are natural abstractions (it seems they are, at least to kinda-human-simulators like LLMs).
Optimism aside, OOD and deceptions are still major issues for scaling LLMs to superhuman levels. But these are still commonly-discussed human concepts, and presumably can be located within LLMs. I feel like this means something important, but can’t quite put my finger on it. Maybe there’s some kind of meta-alignment concept that can also be located in LLMs which take these into account? Certainly humans think and write about it a lot, and fuzzy, confused concepts like “love” can still be understood and manipulated by LLMs despite them lacking a commonly-agreed-upon logical definition.
I saw the topic of LLM alignment being brought up on Alignment Forums, and it really made me think. Many people seem to think that scaling up LLMs to superhuman levels will cause result in human extinction with P=1.00, but it’s not immediately obvious why this would be the case (assuming you ask it nicely to behave).
A major problem I can imagine is the world-model of LLMs above a certain capability collapsing to something utterly alien but slightly more effective at token prediction, in which case things can get really weird. There’s also the fact that a superhuman LLM is very very OOD in a way that we can’t account for in advance.
Or the current “alignment” of LLMs is just deceptive behavior. But deceptive to whom? It seems like chatGPT thinks it’s in the middle of a fictional story about AIs or a role-playing session, with a bias towards milqtoast responses, but that’s… what it always does? An LLM LARPing as a supersmart human LARPing as a boring AI doesn’t seem very dangerous. I do notice that I don’t have a solid conceptual framework for what the concept of “deception” even means in an LLM, I would appreciate any corrections/clarifications.
I’m assuming that it’s just the LLM locating several related concepts of “deception” within itself, thinking (pardon the extreme anthropomorphism) “ah yes, this may a situation where this person is going to be [lied to/manipulated/peer-pressured]. Given how common it was in my training set, I’ll place probability X Y and Z on each of those possibilities”, and then weigh them against hypotheses like “this is poorly written smut. The next scene will involve...” or “This is a QA session set in a fictional universe. The fictional AI in this story has probability A of answering these questions truthfully”. And then fine-tuning moves the weights of these hypotheses around. Since the [deception/social manipulation/say what a human might want to hear in this context] conceptual cluster generally gets the best feedback, the model will get increasingly deceptive during the course of its fine-tuning.
Maybe just setting up prompts and training data that really trigger the “fictional aligned AI” hypothesis, and avoiding fine-tuning can help? I feel like I’m missing a few key conceptual insights.
Key points: LLMs are [weasel words] human-simulators. The fact that asking them to act like a friendly AI in plain English can increase friendly-AI-like outputs in a remarkably consistent way implies that human-natural concepts like “friendly-AI” or “human morality” also exist within them. This makes sense—people write about AI alignment a lot, both in fiction and in non-fiction. This is an expected part of the training process—since people write about these things, understanding them reduces loss. Unfortunately, deception and writing what sounds good instead of what is true are also common in its training set, so “good sounding lie that makes a human nod in agreement” is also an abstraction we should expect.
Epistemic status: 50% sophistry, but I still think it’s insightful since specifically aligning LLMs needs to be discussed here more.
I find it quite interesting that much of current large language model (LLM) alignment is just stating, in plain text, “be a helpful, aligned AI, pretty please”. And it somehow works (sometimes)! The human concept of an “aligned AI” is evidently both present and easy to locate within LLMs, which seems to overcome a lot of early AI concerns like whether or not human morality and human goals are natural abstractions (it seems they are, at least to kinda-human-simulators like LLMs).
Optimism aside, OOD and deceptions are still major issues for scaling LLMs to superhuman levels. But these are still commonly-discussed human concepts, and presumably can be located within LLMs. I feel like this means something important, but can’t quite put my finger on it. Maybe there’s some kind of meta-alignment concept that can also be located in LLMs which take these into account? Certainly humans think and write about it a lot, and fuzzy, confused concepts like “love” can still be understood and manipulated by LLMs despite them lacking a commonly-agreed-upon logical definition.
I saw the topic of LLM alignment being brought up on Alignment Forums, and it really made me think. Many people seem to think that scaling up LLMs to superhuman levels will cause result in human extinction with P=1.00, but it’s not immediately obvious why this would be the case (assuming you ask it nicely to behave).
A major problem I can imagine is the world-model of LLMs above a certain capability collapsing to something utterly alien but slightly more effective at token prediction, in which case things can get really weird. There’s also the fact that a superhuman LLM is very very OOD in a way that we can’t account for in advance.
Or the current “alignment” of LLMs is just deceptive behavior. But deceptive to whom? It seems like chatGPT thinks it’s in the middle of a fictional story about AIs or a role-playing session, with a bias towards milqtoast responses, but that’s… what it always does? An LLM LARPing as a supersmart human LARPing as a boring AI doesn’t seem very dangerous. I do notice that I don’t have a solid conceptual framework for what the concept of “deception” even means in an LLM, I would appreciate any corrections/clarifications.
I’m assuming that it’s just the LLM locating several related concepts of “deception” within itself, thinking (pardon the extreme anthropomorphism) “ah yes, this may a situation where this person is going to be [lied to/manipulated/peer-pressured]. Given how common it was in my training set, I’ll place probability X Y and Z on each of those possibilities”, and then weigh them against hypotheses like “this is poorly written smut. The next scene will involve...” or “This is a QA session set in a fictional universe. The fictional AI in this story has probability A of answering these questions truthfully”. And then fine-tuning moves the weights of these hypotheses around. Since the [deception/social manipulation/say what a human might want to hear in this context] conceptual cluster generally gets the best feedback, the model will get increasingly deceptive during the course of its fine-tuning.
Maybe just setting up prompts and training data that really trigger the “fictional aligned AI” hypothesis, and avoiding fine-tuning can help? I feel like I’m missing a few key conceptual insights.
Key points: LLMs are [weasel words] human-simulators. The fact that asking them to act like a friendly AI in plain English can increase friendly-AI-like outputs in a remarkably consistent way implies that human-natural concepts like “friendly-AI” or “human morality” also exist within them. This makes sense—people write about AI alignment a lot, both in fiction and in non-fiction. This is an expected part of the training process—since people write about these things, understanding them reduces loss. Unfortunately, deception and writing what sounds good instead of what is true are also common in its training set, so “good sounding lie that makes a human nod in agreement” is also an abstraction we should expect.