That’s their purported terminal value, yes. But if we just had magical boxes that we could put problems into and get the solutions… we’d end up feeling quite listless. (Well, until somebody noticed that you could munchkin that to solve all of the problems ever and build utopia, perhaps even eutopia if you could phrase your question right, because these boxes are capable of violating the Second Law of Thermodynamics and more – but that doesn’t apply in the Pebblesorters’ case.)
Unlike them, our terminal value seems to include seeking the feeling that we’re personally contributing. (A magic box that understood our terminal values and would tell us how to solve our problems in order to maximize our values would probably phrase its answer with some open parts in a way that still made us feel like we had agency in executing the answer.)
That’s their purported terminal value, yes. But if we just had magical boxes that we could put problems into and get the solutions… we’d end up feeling quite listless. (Well, until somebody noticed that you could munchkin that to solve all of the problems ever and build utopia, perhaps even eutopia if you could phrase your question right, because these boxes are capable of violating the Second Law of Thermodynamics and more – but that doesn’t apply in the Pebblesorters’ case.)
Unlike them, our terminal value seems to include seeking the feeling that we’re personally contributing. (A magic box that understood our terminal values and would tell us how to solve our problems in order to maximize our values would probably phrase its answer with some open parts in a way that still made us feel like we had agency in executing the answer.)