Whilst interesting, this analysis doesn’t seem to quite hit the nail on the head for me.
Power Distribution: First-movers with advanced AI could gain permanent military, economic, and/or political dominance.
This framing both merges multiple issues and almost assumes a particular solution (that of power distribution).
Instead, I propose that this problem be broken into:
a) Distributive justice: Figuring out how to fairly resolve conflicting interests
b) Stewardship: Ensuring that no-one can seize control of any ASI’s and that such power isn’t transferred to a malicious or irresponsible actor
c) Trustworthiness: Designing the overall system (both human and technological components) in such a way that different parties have rational reasons to trust that conflicting interests will be resolved fairly and that proper stewardship will be maintained over the system
d) Buy-in: Gaining support from different actors for a particular system to be implemented. This may involve departing from any distributive ideal
Of course, broadly distributing power can be used to address any of these issues, but we shouldn’t assume that it is necessarily the best solution.
Economics Transition: When AI generates all wealth, humans have no leverage to ensure they are treated well… It’s still unclear how we get to a world where humans have any economic power if all the jobs are automated by advanced AI.
This seems like a strange framing to me. Maybe I’m reading too much into your wording, but it seems to almost assume that the goal is to maintain a broad distribution of “economic” power through the AGI transition. Whilst this would be one way of ensuring the broad distribution of benefits, it hardly seems like the only, or even most promising route. Why should we assume that the world will have anything like a traditional economy after AGI?
Additionally, alignment can refer to either “intent alignment” or “alignment with human values”[1]. Your analysis seems to assume the former, I’d suggest flagging this explicitly if that’s what you mean. Where this most directly matters is the extent to which we are telling these machines what to do vs. autonomously making their own decisions, which affects the importance of solving problems manually.
It will be the goverment(s) who decides how AGI is used, not a benevolent coalition of utilitarian rationalists.
Somebody is going to make AGI and thereby control it (in the likely event it’s intent-aligned—see below). And the goverment that asserts control over that company is probably going to seize effective control of that project as soon as they realize its potential.
National security critical technologies are the the domain of the goverment and always have been. And AGI is the most security-relevant technology in history. Finally, politicians often don’t understand new technologies, but the national security apparatus is not composed entirely of idiots.
On the economic side: We’re likely to see a somewhat slow takeoff on the current trajectory. That’s enough time for everyone to starve if they’re all out of work before an ASI can just make technologies that make food and housing out of nothing—if its controllers want it to.
It will be the goverment(s) who decides how AGI is used, not a benevolent coalition of utilitarian rationalists.
Even so, the government still needs to weigh up opposing concerns, maintain ownership of the AGI, set up the system in such a way that they have trust in it and gain some degree of buy-in from society for the plan[1].
Whilst interesting, this analysis doesn’t seem to quite hit the nail on the head for me.
This framing both merges multiple issues and almost assumes a particular solution (that of power distribution).
Instead, I propose that this problem be broken into:
a) Distributive justice: Figuring out how to fairly resolve conflicting interests
b) Stewardship: Ensuring that no-one can seize control of any ASI’s and that such power isn’t transferred to a malicious or irresponsible actor
c) Trustworthiness: Designing the overall system (both human and technological components) in such a way that different parties have rational reasons to trust that conflicting interests will be resolved fairly and that proper stewardship will be maintained over the system
d) Buy-in: Gaining support from different actors for a particular system to be implemented. This may involve departing from any distributive ideal
Of course, broadly distributing power can be used to address any of these issues, but we shouldn’t assume that it is necessarily the best solution.
This seems like a strange framing to me. Maybe I’m reading too much into your wording, but it seems to almost assume that the goal is to maintain a broad distribution of “economic” power through the AGI transition. Whilst this would be one way of ensuring the broad distribution of benefits, it hardly seems like the only, or even most promising route. Why should we assume that the world will have anything like a traditional economy after AGI?
Additionally, alignment can refer to either “intent alignment” or “alignment with human values”[1]. Your analysis seems to assume the former, I’d suggest flagging this explicitly if that’s what you mean. Where this most directly matters is the extent to which we are telling these machines what to do vs. autonomously making their own decisions, which affects the importance of solving problems manually.
Whatever that means
This seems unrealistically idealistic to me.
It will be the goverment(s) who decides how AGI is used, not a benevolent coalition of utilitarian rationalists.
Somebody is going to make AGI and thereby control it (in the likely event it’s intent-aligned—see below). And the goverment that asserts control over that company is probably going to seize effective control of that project as soon as they realize its potential.
National security critical technologies are the the domain of the goverment and always have been. And AGI is the most security-relevant technology in history. Finally, politicians often don’t understand new technologies, but the national security apparatus is not composed entirely of idiots.
On the economic side: We’re likely to see a somewhat slow takeoff on the current trajectory. That’s enough time for everyone to starve if they’re all out of work before an ASI can just make technologies that make food and housing out of nothing—if its controllers want it to.
Thanks for the care and possible nod to not Conflating value alignment and intent alignment! The poster seems to be assuming intent alignment, which I think is very likely right because Instruction-following AGI is easier and more likely than value aligned AGI
See my other comment with links to related discussions.
Even so, the government still needs to weigh up opposing concerns, maintain ownership of the AGI, set up the system in such a way that they have trust in it and gain some degree of buy-in from society for the plan[1].
Unless their plan is to use the AGI to enforce their will