Suppose that you, as the owner of the bottle, have the power either to free the genie into whatever otherworld it came from (from which he cannot interact with the human world) or to cast the bottle into the fires of Mount Doom, destroying the genie irrevocably.
That isn’t very much like the real world. You can’t release a powerful “genie” in the real world—without lots of changes taking place. So, how is this supposed to relate to “unfriendly” machines? You can’t play this game with them—for obvious reasons. If you lie to them about their reward, they may figure that out—especially if you feed them real-world problems.
I think the OP was just adding flavor with that. For a more realistic scenario, imagine a GAI with a simple utility function that is maximized by pushing a button, which also happens to shut down the GAI immediately (and the GAI knows this, and its utility function has no problem with that fact). The GAI is then motivated to want that button pushed, even if it hates your guts.
That isn’t very much like the real world. You can’t release a powerful “genie” in the real world—without lots of changes taking place. So, how is this supposed to relate to “unfriendly” machines? You can’t play this game with them—for obvious reasons. If you lie to them about their reward, they may figure that out—especially if you feed them real-world problems.
I think the OP was just adding flavor with that. For a more realistic scenario, imagine a GAI with a simple utility function that is maximized by pushing a button, which also happens to shut down the GAI immediately (and the GAI knows this, and its utility function has no problem with that fact). The GAI is then motivated to want that button pushed, even if it hates your guts.
Uh, yes—that helps—thanks!
Right...