The deployment problem is part of societal response to me, not separate.
[...] Eg race dynamics, regulation (including ability to cooperate with competitors), societal pressure on leaders, investment in watchdogs (human and machine), safety testing norms, whether things get open sourced, infohazards.
“The deployment problem is hard and weird” comes from a mix of claims about AI (AGI is extremely dangerous, you don’t need a planet-sized computer to run it, software and hardware can and will improve and proliferate by default, etc.) and about society (“if you give a decent number of people the ability to wield dangerous AGI tech, at least one or them will choose to use it”).
The social claims matter — two people who disagree about how readily Larry Page and/or Mark Zuckerberg would put the world at risk might as a result disagree about whether a Good AGI Project has median 8 months vs. 12 months to do a pivotal act.
When I say “AGI ruin rests on strong claims about the alignment problem and deployment problem, not about society”, I mean that the claims you need to make about society in order to think the alignment and deployment problems are that hard and weird, are weak claims (e.g. “if fifty random large AI companies had the ability to use dangerous AGI, at least one would use it”), and that the other claims about society required for high p(doom) are weak too (e.g. “humanity isn’t a super-agent that consistently scales up its rationality and effort in proportion to a problem’s importance, difficulty, and weirdness”).
Arguably the difficulty of the alignment problem itself also depends in part on claims about society. E.g., the difficulty of alignment depends on the difficulty of the task we’re aligning, which depends on “what sort of task is needed to end the acute x-risk period?”, which depends again on things like “will random humans destroy the world if you hand them world-destroying AGI?”.
The thing I was trying to communicate (probably poorly) isn’t “Alignment, Deployment, and Society partitions the space of topics”, but rather:
High p(doom) rests on strong claims about AI/compute/etc. and quite weak claims about humanity/society.
The most relevant claims (~all the strong ones, and an important subset of the weak ones) are mostly claims about the difficulty, novelty, and weirdness of the alignment and deployment problems.
Dustin Moskovitz comments on Twitter:
“The deployment problem is hard and weird” comes from a mix of claims about AI (AGI is extremely dangerous, you don’t need a planet-sized computer to run it, software and hardware can and will improve and proliferate by default, etc.) and about society (“if you give a decent number of people the ability to wield dangerous AGI tech, at least one or them will choose to use it”).
The social claims matter — two people who disagree about how readily Larry Page and/or Mark Zuckerberg would put the world at risk might as a result disagree about whether a Good AGI Project has median 8 months vs. 12 months to do a pivotal act.
When I say “AGI ruin rests on strong claims about the alignment problem and deployment problem, not about society”, I mean that the claims you need to make about society in order to think the alignment and deployment problems are that hard and weird, are weak claims (e.g. “if fifty random large AI companies had the ability to use dangerous AGI, at least one would use it”), and that the other claims about society required for high p(doom) are weak too (e.g. “humanity isn’t a super-agent that consistently scales up its rationality and effort in proportion to a problem’s importance, difficulty, and weirdness”).
Arguably the difficulty of the alignment problem itself also depends in part on claims about society. E.g., the difficulty of alignment depends on the difficulty of the task we’re aligning, which depends on “what sort of task is needed to end the acute x-risk period?”, which depends again on things like “will random humans destroy the world if you hand them world-destroying AGI?”.
The thing I was trying to communicate (probably poorly) isn’t “Alignment, Deployment, and Society partitions the space of topics”, but rather:
High p(doom) rests on strong claims about AI/compute/etc. and quite weak claims about humanity/society.
The most relevant claims (~all the strong ones, and an important subset of the weak ones) are mostly claims about the difficulty, novelty, and weirdness of the alignment and deployment problems.