(sorry for giving you another answer, but it seems it’s useful to separate the points):
But in order to win, we have to find the right utility function and maximize that one.
The “right” utility function. Which we don’t currently know. And yet we can make decisions until the day we get it, and still make ourselves unexploitable in the meantime. The first AIs may well be in a similar situation.
(sorry for giving you another answer, but it seems it’s useful to separate the points):
The “right” utility function. Which we don’t currently know. And yet we can make decisions until the day we get it, and still make ourselves unexploitable in the meantime. The first AIs may well be in a similar situation.