Nate Soares points out that the first paragraph is not quite right: Imagine writing a program that somehow implements an aligned superintelligence, giving it as an objective, “maximize utility according to the person who pressed the ‘go’ button”, and pressing the ‘go’ button.
There’s some sense in which, by virtue of existing in the world, you’re already kind of “lucky” by this metric: It can take a finite amount of information to instantiate an agent that takes unbounded actions on your behalf.
Nate Soares points out that the first paragraph is not quite right: Imagine writing a program that somehow implements an aligned superintelligence, giving it as an objective, “maximize utility according to the person who pressed the ‘go’ button”, and pressing the ‘go’ button.
There’s some sense in which, by virtue of existing in the world, you’re already kind of “lucky” by this metric: It can take a finite amount of information to instantiate an agent that takes unbounded actions on your behalf.