It sounds like you’re proposing a system that is vulnerable to the Fully Updated Deference problem, and where if it has a flaw in how it models your preferences, it can very plausibly go against your words. I don’t think that’s corrigible.
In the specific example, just because one is confused about what they want doesn’t mean the AI will be (or should be). It seems like you think the AGI should not “take a guess” at the preferences of the principal, but it should listen to what the principal says. Where is the qualitative line between the two? In your system, if I write in my diary that I want the AI to do something, should it not listen to that? Certainly the diary entry is strong evidence about what I want, which it seems is how you’re thinking about commands. Suppose the AGI can read my innermost desires using nanomachines, and set up the world according to those desires. Is it corrigible? Notably, if that machine is confident that it knows better than me (which is plausible), it won’t stop if I tell it to shut down, because shutting down is a bad way to produce MaxUtility. (See the point in my document, above, where I discuss Queen Alice being totally disempowered by sufficiently good “servants”.)
My model of Seth says “It’s fine if the AGI does what I want and not what I say, as long as it’s correct about what I want.” But regardless of whether that’s true, I think it’s important not to confuse that system with one that’s corrigible.
I don’t understand your proposal if it doesn’t boil down to “do what the principal wants” or “do what the principal says” (correctly interpreted and/or carefully verified). This makes me worried that what you have in mind is not that simple and coherent and therefore relatively easy to define or train into an AGI.
This (maybe misunderstanding) of your corrigibility=figure out what I want is why I currently prefer the instruction-following route to corrigibility. I don’t want the AGI to guess at what I want any more than necessary. This has downsides, too; back to those at the end.
I do think what your model of me says, but I think it’s only narrowly true and probably not very useful that
It’s fine if the AGI does what I want and not what I say, as long as it’s correct about what I want.
I think this is true for exactly the right definition of “what I want”, but conveying that to an AGI is nontrivial, and re-introduces the difficulty of value learning. That’s mixed with the danger that it’s incorrect about what I want. That is, it could be right about what I want in one sense, but not the sense I wanted to convey to it (E.G., it decides I’d really rather be put into an experience machine where I’m the celebrated hero of the world, rather than make the real world good for everyone like I’d hoped to get).
Maybe I’ve misunderstood your thesis, but I did read it pretty carefully, so there might be something to learn from how I’ve misunderstood. All of your examples I remember correspond to “doing what the principal wants” by a pretty common interpetation of that phrase.
Instruction-following puts a lot of the difficulty back on the human(s) in charge. This is potentially very bad, but I think humans will probably choose this route anyway. You’ve pointed out some ways that following instructions could be a danger (although I think your genie examples aren’t the most relevant for a modest takeoff speed). But I think unless something changes, humans are likely to prefer keeping the power and the responsibility to trying to put more of the project into the AGIs alignment. That’s another reason I’m spending my time thinking through this route to corrigibility instead of the one you propose.
Although again, I might be missing something about your scheme.
I just went back and reread 2. Corrigibility Intuition (after writing the above, which I won’t try to revise). Everything there still looks like a flavor of “do what I want”. My model of Max says “corrigibility is more like ‘do your best to be correctable’”. It seems like correctable means correctable toward what the principal wants. So I wonder if your formulation reduces to “do what I want, with an emphasis on following instructions and being aware that you might be wrong about what I want”. That sounds very much like the Do What I Mean And Check formulation of my instruction-following approach to corrigibility.
Thanks for engaging. I think this is productive.
Just to pop back to the top level briefly, I’m focusing on instruction-following because I think it will work well and be the more likely pick for a nascent language-model agent AGI, from below human level to somewhat above it. If RL is heavily involved in creating that agent, that might shift the balance and make your form of corrigibility more attractive (and still vastly more attractive than attempting value alignment in any broader way). I think working through both of these is worthwhile, because those are the two most likely forms of first AGI, and the two most likely actual alignment targets.
I definitely haven’t wrapped my head around all of the pitfalls with either method, but I continue to think that this type of alignment target makes good outcomes much more likely, at least as far as we’ve gotten with the analysis so far.
I think this type of alignment target is also important because the strongest and most used arguments for alignment difficulty don’t apply to them. So when we’re debating slowing down AGI, proponents of going forward will be talking about these approaches. If the alignment community hasn’t thought through them carefully, there will be no valid counterargument. I’d still prefer that we slow AGI even though I think these methods give us a decent chance of succeeding at technical alignment. So that’s one more reason I find this topic worthwhile.
This has gotten pretty discursive, so don’t worry about responding to all of it.
It sounds like you’re proposing a system that is vulnerable to the Fully Updated Deference problem, and where if it has a flaw in how it models your preferences, it can very plausibly go against your words. I don’t think that’s corrigible.
In the specific example, just because one is confused about what they want doesn’t mean the AI will be (or should be). It seems like you think the AGI should not “take a guess” at the preferences of the principal, but it should listen to what the principal says. Where is the qualitative line between the two? In your system, if I write in my diary that I want the AI to do something, should it not listen to that? Certainly the diary entry is strong evidence about what I want, which it seems is how you’re thinking about commands. Suppose the AGI can read my innermost desires using nanomachines, and set up the world according to those desires. Is it corrigible? Notably, if that machine is confident that it knows better than me (which is plausible), it won’t stop if I tell it to shut down, because shutting down is a bad way to produce MaxUtility. (See the point in my document, above, where I discuss Queen Alice being totally disempowered by sufficiently good “servants”.)
My model of Seth says “It’s fine if the AGI does what I want and not what I say, as long as it’s correct about what I want.” But regardless of whether that’s true, I think it’s important not to confuse that system with one that’s corrigible.
This seems productive.
I don’t understand your proposal if it doesn’t boil down to “do what the principal wants” or “do what the principal says” (correctly interpreted and/or carefully verified). This makes me worried that what you have in mind is not that simple and coherent and therefore relatively easy to define or train into an AGI.
This (maybe misunderstanding) of your corrigibility=figure out what I want is why I currently prefer the instruction-following route to corrigibility. I don’t want the AGI to guess at what I want any more than necessary. This has downsides, too; back to those at the end.
I do think what your model of me says, but I think it’s only narrowly true and probably not very useful that
I think this is true for exactly the right definition of “what I want”, but conveying that to an AGI is nontrivial, and re-introduces the difficulty of value learning. That’s mixed with the danger that it’s incorrect about what I want. That is, it could be right about what I want in one sense, but not the sense I wanted to convey to it (E.G., it decides I’d really rather be put into an experience machine where I’m the celebrated hero of the world, rather than make the real world good for everyone like I’d hoped to get).
Maybe I’ve misunderstood your thesis, but I did read it pretty carefully, so there might be something to learn from how I’ve misunderstood. All of your examples I remember correspond to “doing what the principal wants” by a pretty common interpetation of that phrase.
Instruction-following puts a lot of the difficulty back on the human(s) in charge. This is potentially very bad, but I think humans will probably choose this route anyway. You’ve pointed out some ways that following instructions could be a danger (although I think your genie examples aren’t the most relevant for a modest takeoff speed). But I think unless something changes, humans are likely to prefer keeping the power and the responsibility to trying to put more of the project into the AGIs alignment. That’s another reason I’m spending my time thinking through this route to corrigibility instead of the one you propose.
Although again, I might be missing something about your scheme.
I just went back and reread 2. Corrigibility Intuition (after writing the above, which I won’t try to revise). Everything there still looks like a flavor of “do what I want”. My model of Max says “corrigibility is more like ‘do your best to be correctable’”. It seems like correctable means correctable toward what the principal wants. So I wonder if your formulation reduces to “do what I want, with an emphasis on following instructions and being aware that you might be wrong about what I want”. That sounds very much like the Do What I Mean And Check formulation of my instruction-following approach to corrigibility.
Thanks for engaging. I think this is productive.
Just to pop back to the top level briefly, I’m focusing on instruction-following because I think it will work well and be the more likely pick for a nascent language-model agent AGI, from below human level to somewhat above it. If RL is heavily involved in creating that agent, that might shift the balance and make your form of corrigibility more attractive (and still vastly more attractive than attempting value alignment in any broader way). I think working through both of these is worthwhile, because those are the two most likely forms of first AGI, and the two most likely actual alignment targets.
I definitely haven’t wrapped my head around all of the pitfalls with either method, but I continue to think that this type of alignment target makes good outcomes much more likely, at least as far as we’ve gotten with the analysis so far.
I think this type of alignment target is also important because the strongest and most used arguments for alignment difficulty don’t apply to them. So when we’re debating slowing down AGI, proponents of going forward will be talking about these approaches. If the alignment community hasn’t thought through them carefully, there will be no valid counterargument. I’d still prefer that we slow AGI even though I think these methods give us a decent chance of succeeding at technical alignment. So that’s one more reason I find this topic worthwhile.
This has gotten pretty discursive, so don’t worry about responding to all of it.