I can see why, if someone thought FOOM was highly likely, they could be pessimistic about governance as a path to safety.
It’s worse than that because foom is so powerful the difference between “no government restricts AI meaningfully” and “9 out of 10 power blocs able to build AI restrict it” is small. Foom for a 90 day takeover implies a doubling time under a week, if all power blocs were equal in starting resources, the “”90 percent ” regulation case vs the “no regulations ” case is 4 doublings or about 4 weeks.
One governance solution proposed to handle this is “nuke em”, but 7 day doubling times imply other things, like some method of building infrastructure that doesn’t need humans current cities and factories and specialists, because by definition humans are not that fast at building anything. Just shipping parts around takes days.
It would be like trying to stop machine cancer. Nukes just buy time.
I personally don’t think the above is possible starting from current technology, I am just trying to take the scenario seriously. (If it’s possible at all I think you would need to bootstrap there through many intermediate stages of technology that take unavoidable amounts of time)
It’s worse than that because foom is so powerful the difference between “no government restricts AI meaningfully” and “9 out of 10 power blocs able to build AI restrict it” is small. Foom for a 90 day takeover implies a doubling time under a week, if all power blocs were equal in starting resources, the “”90 percent ” regulation case vs the “no regulations ” case is 4 doublings or about 4 weeks.
One governance solution proposed to handle this is “nuke em”, but 7 day doubling times imply other things, like some method of building infrastructure that doesn’t need humans current cities and factories and specialists, because by definition humans are not that fast at building anything. Just shipping parts around takes days.
It would be like trying to stop machine cancer. Nukes just buy time.
I personally don’t think the above is possible starting from current technology, I am just trying to take the scenario seriously. (If it’s possible at all I think you would need to bootstrap there through many intermediate stages of technology that take unavoidable amounts of time)