In the paper “Reward is Enough”, it is argued that all AI is really RL, and that loss is the reward. This means that a language model has a goal function to predict the next word in a text. By this reasoning, your human-level RL system should be equivalent to your GPT-n system.
That said, my intuition tells me there should be some fundamental difference. It always seemed to me that NLP is the light side of the force and RL is the dark side. Giving AI a numerical goal? That’s how you get paperclips. Giving AI the ability to understand all of human thought and wisdom? That sounds like a better idea.
To give a model of how things could go wrong in your hypothetical, suppose that the RL system was misaligned in such a way that, when you give it a goal function like “predict the next word”, it builds a model of the entire planet and all of human society, and then conquers the world to get as much computing power as possible, all because it wants to be 99.9999% sure rather than 99.99% sure that it will predict the next word correctly. A GPT-n system is more chill, it wants to get the next word correct but it’s not a goal, more like an instinct.
However, I think you’re likely to be tempted to put a layer of RL on top of your GPT-n so it can act like an agent, and then we’re back where we started.
I suspect the difference is mostly in what training opportunities are available, not what type of system is used internally.
In principle, a strong NLP AI might learn some behaviour that manipulates humans. It’s just that in practice it is more difficult for it to do so, because in almost all of the training phase there is no interaction at all. The input is decoupled from its output, so there is no training signal to improve any ability to manipulate the input.
In reality there are some side-channels that are interactive, such as selection of fine-tuning training based on human evaluation. A sufficiently powerful system might be able to learn enough from that to manipulate the world, but it seems much less likely than some other type of system with more interactive learning doing it first.
In the paper “Reward is Enough”, it is argued that all AI is really RL, and that loss is the reward. This means that a language model has a goal function to predict the next word in a text. By this reasoning, your human-level RL system should be equivalent to your GPT-n system.
That said, my intuition tells me there should be some fundamental difference. It always seemed to me that NLP is the light side of the force and RL is the dark side. Giving AI a numerical goal? That’s how you get paperclips. Giving AI the ability to understand all of human thought and wisdom? That sounds like a better idea.
To give a model of how things could go wrong in your hypothetical, suppose that the RL system was misaligned in such a way that, when you give it a goal function like “predict the next word”, it builds a model of the entire planet and all of human society, and then conquers the world to get as much computing power as possible, all because it wants to be 99.9999% sure rather than 99.99% sure that it will predict the next word correctly. A GPT-n system is more chill, it wants to get the next word correct but it’s not a goal, more like an instinct.
However, I think you’re likely to be tempted to put a layer of RL on top of your GPT-n so it can act like an agent, and then we’re back where we started.
I suspect the difference is mostly in what training opportunities are available, not what type of system is used internally.
In principle, a strong NLP AI might learn some behaviour that manipulates humans. It’s just that in practice it is more difficult for it to do so, because in almost all of the training phase there is no interaction at all. The input is decoupled from its output, so there is no training signal to improve any ability to manipulate the input.
In reality there are some side-channels that are interactive, such as selection of fine-tuning training based on human evaluation. A sufficiently powerful system might be able to learn enough from that to manipulate the world, but it seems much less likely than some other type of system with more interactive learning doing it first.