Well, it might be the case that a system is aligned but is mistakenly running an exploitable decision theory. I think the idea is we would prefer to have things set up so that failures are contained, ie if your AI is running an exploitable decision theory, that problem doesn’t cascade into even worse problems.
At least for the neural net approach, I don’t think “mistakenly running an exploitable decision theory” is a plausible risk—there isn’t going to be a “decision theory” module that we get right or wrong. I feel like this sort of worry is mistaking the abstractions in our map for the territory. I think it is more likely that there will be a bunch of messy reasoning that doesn’t neatly correspond to a single decision theory (just as with humans).
But let’s take the least convenient world, in which that messy reasoning could lead to mistakes that we might call “having the wrong decision theory”. I think often I would count this as a failure of alignment. Presumably humans want the AI system to be cautious about taking huge impactful decisions in novel situations, and any reasonably intelligent AI system should know this, so giving up the universe to an acausal threatener without consulting humans would count as the AI knowably not doing what we want, aka a failure of alignment. So I continue to stand by my claim.
(In your parlance, I might say “humans have preferences over the AI system’s decision theory, so if the AI system uses a flawed decision theory that counts as a failure of alignment”. But the reason that it makes sense to carve up the world in this way is because there won’t be an explicit decision theory module.)
Fwiw, I expect MIRI people are usually on board with this, e.g. this seems to have a similar flavor (though it doesn’t literally say the same thing).
At least for the neural net approach, I don’t think “mistakenly running an exploitable decision theory” is a plausible risk—there isn’t going to be a “decision theory” module that we get right or wrong. I feel like this sort of worry is mistaking the abstractions in our map for the territory. I think it is more likely that there will be a bunch of messy reasoning that doesn’t neatly correspond to a single decision theory (just as with humans).
But let’s take the least convenient world, in which that messy reasoning could lead to mistakes that we might call “having the wrong decision theory”. I think often I would count this as a failure of alignment. Presumably humans want the AI system to be cautious about taking huge impactful decisions in novel situations, and any reasonably intelligent AI system should know this, so giving up the universe to an acausal threatener without consulting humans would count as the AI knowably not doing what we want, aka a failure of alignment. So I continue to stand by my claim.
(In your parlance, I might say “humans have preferences over the AI system’s decision theory, so if the AI system uses a flawed decision theory that counts as a failure of alignment”. But the reason that it makes sense to carve up the world in this way is because there won’t be an explicit decision theory module.)
Fwiw, I expect MIRI people are usually on board with this, e.g. this seems to have a similar flavor (though it doesn’t literally say the same thing).