On training AI systems using human feedback: This is way better than nothing, and it’s great that OpenAI is doing it, but has the following issues:
Practical considerations: AI systems currently tend to require lots of examples and it’s expensive to get these if they all have to be provided by a human.
Some actions look good to a casual human observer, but are actually bad on closer inspection. The AI would be rewarded for finding and taking such actions.
If you’re training a neural network, then there are generically going to be lots of adversarial examples for that network. As the AI gets more and more powerful, we’d expect it to be able to generate more and more situations where its learned value function gives a high reward but a human would give a low reward. So it seems like we end up playing a game of adversarial example whack-a-mole for a long time, where we’re just patching hole after hole in this million-dimensional bucket with thousands of holes. Probably the AI manages to kill us before that process converges.
To make the above worse, there’s this idea of a sharp left turn, where a sufficiently intelligent AI can think of very weird plans that go far outside of the distribution of scenarios that it was trained on. We expect generalization to get worse in this regime, and we also expect an increased frequency of adversarial examples. (What would help a lot here is designing the AI to have an interpretable planning system, where we could run these plans forward and negatively reinforce the bad ones (and maybe all the weird ones, because of corrigibility reasons, though we’d have to be careful about how that’s formulated because we don’t want the AI trying to kill us because it thinks we’d produce a weird future).)
Once the AI is modelling reality in detail, its reward function is going to focus on how the rewards are actually being piped to the AI, rather than the human evaluator’s reaction, let alone of some underlying notion of goodness. If the human evaluators just press a button to reward the AI for doing a good thing, the AI will want to take control of that button and stick a brick on top of it.
On training models to assist in human evaluation and point out flaws in AI outputs: Doing this is probably somewhat better than not doing it, but I’m pretty skeptical that it provides much value:
The AI can try and fool the critic just like it would fool humans. It doesn’t even need a realistic world model for this, since using the critic to inform the training labels leaks information about the critic to the AI.
It’s therefore very important that the critic model generates all the strong and relevant criticisms of a particular AI output. Otherwise the AI could just route around the critic.
On some kinds of task, you’ll have an objective source of truth you can train your model on. The value of an objective source of truth is that we can use it to generate a list of all the criticisms the model should have made. This is important because we can update the weights of the critic model based on any criticisms it failed to make. On other kinds of task, which are the ones we’re primarily interested in, it will be very hard or impossible to get the ground truth list of criticisms. So we won’t be able to update the weights of the model that way when training. So in some sense, we’re trying to generalize this idea of “a strong a relevant criticism” between these different tasks of differing levels of difficulty.
This requirement of generating all criticisms seems very similar to the task of getting a generative model to cover all modes. I guess we’ve pretty much licked mode collapse by now, but “don’t collapse everything down to a single mode” and “make sure you’ve got good coverage of every single mode in existence” are different problems, and I think the second one is much harder.
On using AI systems, in particular large language models, to advance alignment research: This is not going to work.
LLMs are super impressive at generating text that is locally coherent for a much broader definition of “local” than was previously possible. They are also really impressive as a compressed version of humanity’s knowledge. They’re still known to be bad at math, at sticking to a coherent idea and at long chains of reasoning in general. These things all seem important for advancing AI alignment research. I don’t see how the current models could have much to offer here. If the thing is advancing alignment research by writing out text that contains valuable new alignment insights, then it’s already pretty much a human-level intelligence. We talk about AlphaTensor doing math research, but even AlphaTensor didn’t have to type up the paper at the end!
What could happen is that the model writes out a bunch of alignment-themed babble, and that inspires a human researcher into having an idea, but I don’t think that provides much acceleration. People also get inspired while going on a walk or taking a shower.
Maybe something that would work a bit better is to try training a reinforcement-learning agent that lives in a world where it has to solve the alignment problem in order to achieve its goals. Eg. in the simulated world, your learner is embodied in a big robot, and it there’s a door in the environment it can’t fit through, but it can program a little robot to go through the door and perform some tasks for it. And there’s enough hidden information and complexity behind the door that the little robot needs to have some built-in reasoning capability. There’s a lot of challenges here, though. Like how do you come up with a programming environment that’s simple enough that the AI can figure out how to use it, while still being complex enough that the little robot can do some non-trivial reasoning, and that the AI has a chance of discovering a new alignment technique? Could be it’s not possible at all until the AI is quite close to human-level.
On training AI systems using human feedback: This is way better than nothing, and it’s great that OpenAI is doing it, but has the following issues:
Practical considerations: AI systems currently tend to require lots of examples and it’s expensive to get these if they all have to be provided by a human.
Some actions look good to a casual human observer, but are actually bad on closer inspection. The AI would be rewarded for finding and taking such actions.
If you’re training a neural network, then there are generically going to be lots of adversarial examples for that network. As the AI gets more and more powerful, we’d expect it to be able to generate more and more situations where its learned value function gives a high reward but a human would give a low reward. So it seems like we end up playing a game of adversarial example whack-a-mole for a long time, where we’re just patching hole after hole in this million-dimensional bucket with thousands of holes. Probably the AI manages to kill us before that process converges.
To make the above worse, there’s this idea of a sharp left turn, where a sufficiently intelligent AI can think of very weird plans that go far outside of the distribution of scenarios that it was trained on. We expect generalization to get worse in this regime, and we also expect an increased frequency of adversarial examples. (What would help a lot here is designing the AI to have an interpretable planning system, where we could run these plans forward and negatively reinforce the bad ones (and maybe all the weird ones, because of corrigibility reasons, though we’d have to be careful about how that’s formulated because we don’t want the AI trying to kill us because it thinks we’d produce a weird future).)
Once the AI is modelling reality in detail, its reward function is going to focus on how the rewards are actually being piped to the AI, rather than the human evaluator’s reaction, let alone of some underlying notion of goodness. If the human evaluators just press a button to reward the AI for doing a good thing, the AI will want to take control of that button and stick a brick on top of it.
On training models to assist in human evaluation and point out flaws in AI outputs: Doing this is probably somewhat better than not doing it, but I’m pretty skeptical that it provides much value:
The AI can try and fool the critic just like it would fool humans. It doesn’t even need a realistic world model for this, since using the critic to inform the training labels leaks information about the critic to the AI.
It’s therefore very important that the critic model generates all the strong and relevant criticisms of a particular AI output. Otherwise the AI could just route around the critic.
On some kinds of task, you’ll have an objective source of truth you can train your model on. The value of an objective source of truth is that we can use it to generate a list of all the criticisms the model should have made. This is important because we can update the weights of the critic model based on any criticisms it failed to make. On other kinds of task, which are the ones we’re primarily interested in, it will be very hard or impossible to get the ground truth list of criticisms. So we won’t be able to update the weights of the model that way when training. So in some sense, we’re trying to generalize this idea of “a strong a relevant criticism” between these different tasks of differing levels of difficulty.
This requirement of generating all criticisms seems very similar to the task of getting a generative model to cover all modes. I guess we’ve pretty much licked mode collapse by now, but “don’t collapse everything down to a single mode” and “make sure you’ve got good coverage of every single mode in existence” are different problems, and I think the second one is much harder.
On using AI systems, in particular large language models, to advance alignment research: This is not going to work.
LLMs are super impressive at generating text that is locally coherent for a much broader definition of “local” than was previously possible. They are also really impressive as a compressed version of humanity’s knowledge. They’re still known to be bad at math, at sticking to a coherent idea and at long chains of reasoning in general. These things all seem important for advancing AI alignment research. I don’t see how the current models could have much to offer here. If the thing is advancing alignment research by writing out text that contains valuable new alignment insights, then it’s already pretty much a human-level intelligence. We talk about AlphaTensor doing math research, but even AlphaTensor didn’t have to type up the paper at the end!
What could happen is that the model writes out a bunch of alignment-themed babble, and that inspires a human researcher into having an idea, but I don’t think that provides much acceleration. People also get inspired while going on a walk or taking a shower.
Maybe something that would work a bit better is to try training a reinforcement-learning agent that lives in a world where it has to solve the alignment problem in order to achieve its goals. Eg. in the simulated world, your learner is embodied in a big robot, and it there’s a door in the environment it can’t fit through, but it can program a little robot to go through the door and perform some tasks for it. And there’s enough hidden information and complexity behind the door that the little robot needs to have some built-in reasoning capability. There’s a lot of challenges here, though. Like how do you come up with a programming environment that’s simple enough that the AI can figure out how to use it, while still being complex enough that the little robot can do some non-trivial reasoning, and that the AI has a chance of discovering a new alignment technique? Could be it’s not possible at all until the AI is quite close to human-level.