Possibly you’d want to rule out (c) with your stipulation that the tests are “robust”? But I’m not sure you can get tests that robust.
That sounds right. I was thinking about an infinitely robust misalignment-oracle to clarify my thinking, but I agree that we’ll need to be very careful with any real-world-tests.
If I imagine writing code and using the misalignment-oracle on it, I think I mostly agree with Nate’s point. If we have the code and compute to train a superhuman version of GPT-2, and the oracle tells us that any agent coming out from that training process is likely to be misaligned, we haven’t learned much new, and it’s not clear how to design a safe agent from there.
I imagine a misalignment-oracle to be more useful if we use it during the training process, though. Concretely, it seems like a misalignment-oracle would be extremely useful to achieve inner alignment in IDA: as soon as the AI becomes misaligned, we can either rewind the training process and figure out what we did wrong, or directly use the oracle as a training signal that severely punish any step that makes the agent misaligned. Coupled with the ability to iterate on designs, since we won’t accidentally blow up the world on the way, I’d guess that something like this is more likely to work than not. This idea is extremely sensitive to (c), though.
That sounds right. I was thinking about an infinitely robust misalignment-oracle to clarify my thinking, but I agree that we’ll need to be very careful with any real-world-tests.
If I imagine writing code and using the misalignment-oracle on it, I think I mostly agree with Nate’s point. If we have the code and compute to train a superhuman version of GPT-2, and the oracle tells us that any agent coming out from that training process is likely to be misaligned, we haven’t learned much new, and it’s not clear how to design a safe agent from there.
I imagine a misalignment-oracle to be more useful if we use it during the training process, though. Concretely, it seems like a misalignment-oracle would be extremely useful to achieve inner alignment in IDA: as soon as the AI becomes misaligned, we can either rewind the training process and figure out what we did wrong, or directly use the oracle as a training signal that severely punish any step that makes the agent misaligned. Coupled with the ability to iterate on designs, since we won’t accidentally blow up the world on the way, I’d guess that something like this is more likely to work than not. This idea is extremely sensitive to (c), though.