I would say we want an ASI to view world-state-optimization from the perspective of a game developer. Not only should it create predictive models of what goals humans wish to achieve (from both stated and revealed preferences), but it should also learn to predict what difficulty level each human wants to experience in pursuit of those goals.
Then the ASI could aim to adjust the world into states where humans can achieve any goal they can think of when they apply a level of effort that would leave them satisfied in the accomplishment.
Humans don’t want everything handed to us for free, but we also don’t generally enjoy struggling for basic survival (unless we do). There’s a reason we pursue things like competitive sports and video games, even as we denounce the sort of warfare and power struggles that built those competitive instincts in the ancestral environment.
A safe world of abundance that still feels like we’ve fought for our achievements seems to fit what most people would consider “fun”. It’s what children expect in their family environment growing up, it’s what we expect from the games we create, and it’s what we should expect from a future where ASI alignment has been solved.
I would say we want an ASI to view world-state-optimization from the perspective of a game developer. Not only should it create predictive models of what goals humans wish to achieve (from both stated and revealed preferences), but it should also learn to predict what difficulty level each human wants to experience in pursuit of those goals.
Then the ASI could aim to adjust the world into states where humans can achieve any goal they can think of when they apply a level of effort that would leave them satisfied in the accomplishment.
Humans don’t want everything handed to us for free, but we also don’t generally enjoy struggling for basic survival (unless we do). There’s a reason we pursue things like competitive sports and video games, even as we denounce the sort of warfare and power struggles that built those competitive instincts in the ancestral environment.
A safe world of abundance that still feels like we’ve fought for our achievements seems to fit what most people would consider “fun”. It’s what children expect in their family environment growing up, it’s what we expect from the games we create, and it’s what we should expect from a future where ASI alignment has been solved.