The realistic answer for decision making purposes (not epistemic purposes) is mostly hope that offense-defense balances are good enough to prevent the end of the world, combined with accepting somewhat larger risk of misalignment to prevent open source from being bad.
To be kind of blunt, any scenario where AI progress is algorithm dominated and a world where basically everyone can train superintelligence without nations being able to control it and a world where timelines are short is a world where governance of AI is more or less useless, and alignment becomes a little more dicey, so we should mostly ignore such worlds for utility purposes (they are valid for prediction/epistemic purposes).
But I’m approx 80% confident that that’s the world we’re in! I don’t want to just give up in 80% of cases!
I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it. I don’t think it’s hopeless, just that the plans would be rather differently shaped.
I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it.
I basically agree with this.
A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.
In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it’s too easy for people to defect from governance on AI.
I’m not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.
I don’t think it’s hopeless, just that the plans would be rather differently shaped.
IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I’d focus on is on making the world’s offense-defense balance much better than it is now, which means we’d have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.
The realistic answer for decision making purposes (not epistemic purposes) is mostly hope that offense-defense balances are good enough to prevent the end of the world, combined with accepting somewhat larger risk of misalignment to prevent open source from being bad.
To be kind of blunt, any scenario where AI progress is algorithm dominated and a world where basically everyone can train superintelligence without nations being able to control it and a world where timelines are short is a world where governance of AI is more or less useless, and alignment becomes a little more dicey, so we should mostly ignore such worlds for utility purposes (they are valid for prediction/epistemic purposes).
But I’m approx 80% confident that that’s the world we’re in! I don’t want to just give up in 80% of cases!
I think we need to try to figure out some empirical measures for how much like this the world is, and at least come up with plans for what to do about it. I don’t think it’s hopeless, just that the plans would be rather differently shaped.
I basically agree with this.
A crux here is that I believe that AI governance is basically worthless if you cannot control the ability to create powerful AIs, because they will soon be able to create states within a state, and thus resist any governance of AI plan you might wish to implement.
In other words, you need something close to a monopoly on violence in order for a government to continue existing, and under scenarios where AIs get very powerful very quickly because of algorithms, there is no good way to control the distribution of AI power, and it’s too easy for people to defect from governance on AI.
I’m not hopeless that such a world can survive. However, I do think AI governance breaks hard if we are in a future where AI power primarily is determined by algorithms, since I basically consider algorithmic advances mostly impossible to control.
IMO, a plan for short timelines where algorithmic secrets are very powerful I would pivot hard towards technical safety/alignment, and would more or less abandon all but the most minimal governance/control plans, and the other big thing I’d focus on is on making the world’s offense-defense balance much better than it is now, which means we’d have to invest more in making bio-tech less risky, with the caveat that we cannot assume the government will successfully suppress bio-threats from civilian actors.