While I very much would love a new frontier era (I work at a rocket launch startup), and would absolutely be on board with Culture utopia, I see no practical means to ensure that any of these worlds come about without:
Developing proper aligned AGI and making a pivotal turn, i.e creating a Good™ culture mind that takes over the world (fat chance!)
Preventing the development of AI entirely
I do not see a world where AGI exists and follows human orders that does not result in a boot, stomping on a face, forever—societal change in dystopian or totalitarian environments is largely produced via revolution, which becomes nonviable when means of coordination can be effectively controlled and suppressed at scale.
First world countries only enjoy the standard of living they do because, to some degree, the ways to make tons of money are aligned with the well being of society (large diversified investment funds optimize for overall economic well being). Break this connection and things will slide quickly.
Yes, AI will probably create some permanent autocracies. But I think democratic order and responsiveness to societal preferences can survive where it already exists, if a significantly large selectorate of representatives or citizens creates and updates the values for alignment.
Fighting AI development is not only swimming against the tide of capitalist competition between companies, but also competition between democratic and autocratic nations. Difficult, if not impossible.
While I very much would love a new frontier era (I work at a rocket launch startup), and would absolutely be on board with Culture utopia, I see no practical means to ensure that any of these worlds come about without:
Developing proper aligned AGI and making a pivotal turn, i.e creating a Good™ culture mind that takes over the world (fat chance!)
Preventing the development of AI entirely
I do not see a world where AGI exists and follows human orders that does not result in a boot, stomping on a face, forever—societal change in dystopian or totalitarian environments is largely produced via revolution, which becomes nonviable when means of coordination can be effectively controlled and suppressed at scale.
First world countries only enjoy the standard of living they do because, to some degree, the ways to make tons of money are aligned with the well being of society (large diversified investment funds optimize for overall economic well being). Break this connection and things will slide quickly.
Yes, AI will probably create some permanent autocracies. But I think democratic order and responsiveness to societal preferences can survive where it already exists, if a significantly large selectorate of representatives or citizens creates and updates the values for alignment.
Fighting AI development is not only swimming against the tide of capitalist competition between companies, but also competition between democratic and autocratic nations. Difficult, if not impossible.