I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it.
I basically agree with this.
A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.
In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it’s too easy for people to defect from governance on AI.
I’m not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.
I don’t think it’s hopeless, just that the plans would be rather differently shaped.
IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I’d focus on is on making the world’s offense-defense balance much better than it is now, which means we’d have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.
I basically agree with this.
A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.
In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it’s too easy for people to defect from governance on AI.
I’m not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.
IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I’d focus on is on making the world’s offense-defense balance much better than it is now, which means we’d have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.