There is no equally simple version of Stockfish that is still supreme at winning at chess, but will easygoingly let you take a pawn or too. You can imagine a version of Stockfish which does that—a chessplayer which, if it’s sure it can win anyways, will start letting you have a pawn or two—but it’s not simpler to build. By default, Stockfish tenaciously fighting for every pawn (unless you are falling into some worse sacrificial trap), is implicit in its generic general search through chess outcomes.
is saying that when you make a [thing that achieves very impressive things / strongly steers the world], it probably [in general sucks up all the convergent instrumental resources] because that’s simpler than [sucking up all the convergent instrumental resources except in certain cases unrelated to its terminal goals].
Humanity getting a sliver of the Sun’s energy for the next million years, would be a noticeable waste of convergent instrumental resources from the AI’s perspective. Humanity getting a sliver of the Sun’s energy while the nanobots are infecting our bloodstream, in order that we won’t panic, and then later sucking up all the Sun’s energy, is just good tactics; letting you sac your bishop for a pawn for no reason is analogous.
You totally can rewrite Stockfish so that it genuinely lets you win material, but is still unbeatable. You just check: is the evalulation >+20 for Stockfish right now, and will it stay >+15 if I sac this pawn for no benefit? If so, sac the pawn for no benefit. This would work. The point is it’s more complicated, and you have to know something about how Stockfish works, and it’s only stable because Stockfish doesn’t have robust self-improvement optimization channels.
The paragraph you quoted
is saying that when you make a [thing that achieves very impressive things / strongly steers the world], it probably [in general sucks up all the convergent instrumental resources] because that’s simpler than [sucking up all the convergent instrumental resources except in certain cases unrelated to its terminal goals].
Humanity getting a sliver of the Sun’s energy for the next million years, would be a noticeable waste of convergent instrumental resources from the AI’s perspective. Humanity getting a sliver of the Sun’s energy while the nanobots are infecting our bloodstream, in order that we won’t panic, and then later sucking up all the Sun’s energy, is just good tactics; letting you sac your bishop for a pawn for no reason is analogous.
You totally can rewrite Stockfish so that it genuinely lets you win material, but is still unbeatable. You just check: is the evalulation >+20 for Stockfish right now, and will it stay >+15 if I sac this pawn for no benefit? If so, sac the pawn for no benefit. This would work. The point is it’s more complicated, and you have to know something about how Stockfish works, and it’s only stable because Stockfish doesn’t have robust self-improvement optimization channels.