When you talk about their “current values” without any process of growth, I don’t think there’s any there there—it’s a big mess, not a utility function
Sure, yes, exactly my point. The problem is, you don’t need to untangle this mess, or care about having coherent values, to tell an AGI to do things. It’s not going to loop back to you and complain that what you’re telling it to do is incoherent, inasmuch as you’ve solved the control problem and successfully made it do what you want. It’ll just do what you want, the way you’re imagining it, however incoherent it is.
“Maintain US military supremacy the way I typically imagine it” is, in fact, the primary use-case I have in mind, not a weird, unlikely exception.
Talking about good processes of growth is a vital part of getting an AI to do something that looks like “what you want.”
How so? I have wants now. Why do I have to do some kind of “growth”, for these wants to become legitimate? What’d prevent an AGI from understanding them as they are now?
Sure, yes, exactly my point. The problem is, you don’t need to untangle this mess, or care about having coherent values, to tell an AGI to do things. It’s not going to loop back to you and complain that what you’re telling it to do is incoherent, inasmuch as you’ve solved the control problem and successfully made it do what you want. It’ll just do what you want, the way you’re imagining it, however incoherent it is.
“Maintain US military supremacy the way I typically imagine it” is, in fact, the primary use-case I have in mind, not a weird, unlikely exception.
How so? I have wants now. Why do I have to do some kind of “growth”, for these wants to become legitimate? What’d prevent an AGI from understanding them as they are now?