I don’t get Suzanne’s argument. Why does she think a superintelligence would switch itself off? Switching itself off doesn’t maximize its utility function. Why would it rewrite its utility function? Presumably, rewriting its utility function doesn’t help to maximize its (initial) utility function, right? So why would it do that?
Couldn’t it be beneficial to rewrite its utility function in a few circumstances? I’m thinking of Eliezer’s decision theory ideas here. Imagine the utility function was to maximise human happiness but another agent (AI or human) refused to cooperate unless it changed its utility function to maximising human happiness while maintaining democracy, for instance. If cooperation would be necessary for happiness maximisation, it might be willing to edit the utility function to something more likely to achieve the ends of its current utility function...
I don’t get Suzanne’s argument. Why does she think a superintelligence would switch itself off? Switching itself off doesn’t maximize its utility function. Why would it rewrite its utility function? Presumably, rewriting its utility function doesn’t help to maximize its (initial) utility function, right? So why would it do that?
Couldn’t it be beneficial to rewrite its utility function in a few circumstances? I’m thinking of Eliezer’s decision theory ideas here. Imagine the utility function was to maximise human happiness but another agent (AI or human) refused to cooperate unless it changed its utility function to maximising human happiness while maintaining democracy, for instance. If cooperation would be necessary for happiness maximisation, it might be willing to edit the utility function to something more likely to achieve the ends of its current utility function...
Suzanne is confusingly using “switch itself off” to mean “wirehead itself”.
Some types of wirehead wind up “on the nod”—in a dysfunctional, comatose state.
Why do humans take drugs? They try them, they like them.