Half of the world, and the half that’s ahead in the AGI race right now, has been doing very well with centralized power for the last couple of centuries.
It seems to me that, to whatever degree that that’s true, it’s because the “centralized” power is relatively decentralized (and relatively tightly constrained). There’s a formal power structure, but it has a lot of rules and a lot of players and some number of the ever-popular checks and balances. It’s relatively hard even for the Big Kahuna, or even the Big Kahuna and a few cronies, to make the whole power structure do much of anything, and that’s an intentional part of the design.
If you create an ASI that can and will implement somebody’s personal intent, you’re not trusting the power structure; you’re trusting that person. And if you try to make it more “constitutional” or more about collective intent, you suddenly run into a bunch of complex issues that look more and more like the ones you’d get with “value alignment”[1].
I’m also not sure that anybody actually is doing so well with centralized power structures that it’s very comfortable to trust those power structures with “one mistake and we’re all screwed forever” levels of power. It’s not like anybody in the half of the world you’re talking about has been infallible.
It seems to me that, to whatever degree that that’s true, it’s because the “centralized” power is relatively decentralized (and relatively tightly constrained). There’s a formal power structure, but it has a lot of rules and a lot of players and some number of the ever-popular checks and balances. It’s relatively hard even for the Big Kahuna, or even the Big Kahuna and a few cronies, to make the whole power structure do much of anything, and that’s an intentional part of the design.
If you create an ASI that can and will implement somebody’s personal intent, you’re not trusting the power structure; you’re trusting that person. And if you try to make it more “constitutional” or more about collective intent, you suddenly run into a bunch of complex issues that look more and more like the ones you’d get with “value alignment”[1].
I’m also not sure that anybody actually is doing so well with centralized power structures that it’s very comfortable to trust those power structures with “one mistake and we’re all screwed forever” levels of power. It’s not like anybody in the half of the world you’re talking about has been infallible.
I still can’t bring myself to use the word “aligned” without some kind of distancing, hence the quotes.