It’s forceful and unambiguous because Stockfish’s victory over the other player terminates the other player’s goals, whatever those may be: no matter what your goals during the game may be, you can’t pursue it once the game is over (and you’ve lost). Available joules are zero-sum in the same way that playing a chess game is zero-sum.
I am rather doubtful that humanity has any (ahem) terminal goals which a hypothetical trade of all future joules/life would maximize, but if you think humanity does have a short-term goal or value akin to the ‘3 pawns capture’ achievement, which we could pursue effectively while allowing superintelligences to take over and would choose to do so both ex ante & ex post, despite the possible consequences, you should definitely say what it is, because capturing 3 pawns is certainly not a compelling analogy of a goal worth pursuing at the cost of further losing the game.
To me this looks like circular reasoning: this example supports my conceptual framework because I interpret the example according to the conceptual framework.
Instead, I notice that Stockfish in particular has some salient characteristics that go against the predictions of the conceptual framework:
It is indeed superhuman
It is not the case that once Stockfish ends the game that’s it. I can rewind Stockfish. I can even make one version of Stockfish play against another. I can make Stockfish play a chess variant. Stockfish doesn’t annihilate my physical body when it defeats me
It is extremely well aligned with my values. I mostly use it to analyze games I’ve played against other people my level
If Stockfish wants to win the game and I want an orthogonal goal, like capturing its pawns, this is very feasible
Now, does this even matter for considering whether a superintelligence would trade, wouldn’t trade? Not that much, it’s a weak consideration. But insofar as it’s a consideration, does it really convince someone who doesn’t already but the frame? Not to me.
It’s forceful and unambiguous because Stockfish’s victory over the other player terminates the other player’s goals, whatever those may be: no matter what your goals during the game may be, you can’t pursue it once the game is over (and you’ve lost). Available joules are zero-sum in the same way that playing a chess game is zero-sum.
The analogy only goes through if you really double down on the ‘goal’ of capturing some pawns as intrinsically valuable, so even the subsequent defeat is irrelevant. At which point, you’re just unironically making the New Yorker cartoon joke: “Yes, the planet got destroyed. But for a beautiful moment in time we created a lot of value for [capturing 3 pawns].”.
I am rather doubtful that humanity has any (ahem) terminal goals which a hypothetical trade of all future joules/life would maximize, but if you think humanity does have a short-term goal or value akin to the ‘3 pawns capture’ achievement, which we could pursue effectively while allowing superintelligences to take over and would choose to do so both ex ante & ex post, despite the possible consequences, you should definitely say what it is, because capturing 3 pawns is certainly not a compelling analogy of a goal worth pursuing at the cost of further losing the game.
To me this looks like circular reasoning: this example supports my conceptual framework because I interpret the example according to the conceptual framework.
Instead, I notice that Stockfish in particular has some salient characteristics that go against the predictions of the conceptual framework:
It is indeed superhuman
It is not the case that once Stockfish ends the game that’s it. I can rewind Stockfish. I can even make one version of Stockfish play against another. I can make Stockfish play a chess variant. Stockfish doesn’t annihilate my physical body when it defeats me
It is extremely well aligned with my values. I mostly use it to analyze games I’ve played against other people my level
If Stockfish wants to win the game and I want an orthogonal goal, like capturing its pawns, this is very feasible
Now, does this even matter for considering whether a superintelligence would trade, wouldn’t trade? Not that much, it’s a weak consideration. But insofar as it’s a consideration, does it really convince someone who doesn’t already but the frame? Not to me.