That option runs into the problem that you’ve just let Omega extort money by threatening to create a person, torture it, and then destroy it. That seems problematic in other ways.
Everything Omega does is horribly problematic because Omega is at best an UFAI. I’ve never seen “preemptively neutralize Omega completely” offered as an option in any of the hypothetical scenarios even though that’s obviously the very best choice.
Is it really in anyone’s best interest to ever cooperate with Omega given that Omega seems intent on continuing a belligerent campaign of threats against humanity? “I’ll give you a choice between $1,000,000 or $1,000 today, but tomorrow I might threaten to slap you or throw you in hell. Oh, btw, I just simulated you against your will 2^100 times to maintain my perfect record on one/two-boxing.”
I may be overly tired and that may sound like hyperbole, but I do think that any rational agent encountering a far more powerful agent known to be at least not-friendly should think long and hard about the probability that the powerful agent can be trusted with even seemingly innocuous situations. There may be no way to Win. Some form of defection or defiance of the powerful agent may yield more dignity utilons than playing along with any of the choices offered by Omega. Survival machines may not value dignity and self-determination, but many humans value them quite highly.
That option runs into the problem that you’ve just let Omega extort money by threatening to create a person, torture it, and then destroy it. That seems problematic in other ways.
Everything Omega does is horribly problematic because Omega is at best an UFAI. I’ve never seen “preemptively neutralize Omega completely” offered as an option in any of the hypothetical scenarios even though that’s obviously the very best choice.
Is it really in anyone’s best interest to ever cooperate with Omega given that Omega seems intent on continuing a belligerent campaign of threats against humanity? “I’ll give you a choice between $1,000,000 or $1,000 today, but tomorrow I might threaten to slap you or throw you in hell. Oh, btw, I just simulated you against your will 2^100 times to maintain my perfect record on one/two-boxing.”
I may be overly tired and that may sound like hyperbole, but I do think that any rational agent encountering a far more powerful agent known to be at least not-friendly should think long and hard about the probability that the powerful agent can be trusted with even seemingly innocuous situations. There may be no way to Win. Some form of defection or defiance of the powerful agent may yield more dignity utilons than playing along with any of the choices offered by Omega. Survival machines may not value dignity and self-determination, but many humans value them quite highly.
I’m using this comment to test the −5 karma rule. Just ignore it.