I agree with most of these claims. However, I disagree about the level of intelligence required to take over the world, which makes me overall much more scared of AI/doomy than it seems like you are. I think there is at least a 20% chance that a superintelligence with +12 SD capabilities across all relevant domains (esp. planning and social manipulation) could take over the world.
I specifically said a human with +12 SD g factor. I didn’t actually consider what a superintelligence that was at that level on all domains would mean, but I don’t think it would matter because of objection 4: by the time superhuman agents arrive, we would already have numerous superhuman non agentic AI, including systems specialised for planning/tactics/strategy.
You’d need to make particular claims about how a superhuman agent performs in a world of humans amplified by superhuman non agents. It’s very not obvious to me that they can win any ensuing cognitive arms race.
I am sceptical that a superhuman agent /agency would easily attain decisive cognitive superiority to the rest of civilisation.
Hmm… I guess I’m skeptical that we can train very specialized “planning” systems? Making superhuman plans of the sort that could counter those of an agentic superintelligence seems like it requires both a very accurate and domain-general model of the world as well as a search algorithm to figure out which plans actually accomplish a given goal given your model of the world. This seems extremely close in design space to a more general agent. While I think we could have narrow systems which outperform the misaligned superintelligence in other domains such as coding or social manipulation, general long-term planning seems likely to me to be the most important skill involved in taking over the world or countering an attempt to do so.
In the intervening period, I’ve updated towards your position, though I still think it is risky to build systems with capabilities that open ended which are that close to agents in design space
I specifically said a human with +12 SD g factor. I didn’t actually consider what a superintelligence that was at that level on all domains would mean, but I don’t think it would matter because of objection 4: by the time superhuman agents arrive, we would already have numerous superhuman non agentic AI, including systems specialised for planning/tactics/strategy.
You’d need to make particular claims about how a superhuman agent performs in a world of humans amplified by superhuman non agents. It’s very not obvious to me that they can win any ensuing cognitive arms race.
I am sceptical that a superhuman agent /agency would easily attain decisive cognitive superiority to the rest of civilisation.
Hmm… I guess I’m skeptical that we can train very specialized “planning” systems? Making superhuman plans of the sort that could counter those of an agentic superintelligence seems like it requires both a very accurate and domain-general model of the world as well as a search algorithm to figure out which plans actually accomplish a given goal given your model of the world. This seems extremely close in design space to a more general agent. While I think we could have narrow systems which outperform the misaligned superintelligence in other domains such as coding or social manipulation, general long-term planning seems likely to me to be the most important skill involved in taking over the world or countering an attempt to do so.
Well, simulator type systems like GPT-3 do not become agents if amplified to superhuman cognition.
Simulators could be used to generate/evaluate superhuman plans without being agents with independent objectives of their own.
In the intervening period, I’ve updated towards your position, though I still think it is risky to build systems with capabilities that open ended which are that close to agents in design space