AGI is dangerous if it pursues an unaligned goal more competently than humans. [...] It’s proposing “AGI won’t work”.
I’d say it’s proposing something like “minds including AGIs generally aren’t agentic enough to reliably exert significant power on the world”, with an implicit assumption like “minds that look like they have done that have mostly just gotten lucky or benefited from something like lots of built-up cultural heuristics that are only useful in a specific context and would break down in a sufficiently novel situation”.
I agree that even if this was the case, it wouldn’t eliminate the argument for AI risk; even allowing that, AIs could still become more competent than us and eventually, some of them could get lucky too. My impression of the original discussion was that the argument wasn’t meant as an argument against all AI risk, but rather just against hard takeoff-type scenarios depicting a single AI that takes over the world by being supremely intelligent and agentic.
I’d say it’s proposing something like “minds including AGIs generally aren’t agentic enough to reliably exert significant power on the world”, with an implicit assumption like “minds that look like they have done that have mostly just gotten lucky or benefited from something like lots of built-up cultural heuristics that are only useful in a specific context and would break down in a sufficiently novel situation”.
I agree that even if this was the case, it wouldn’t eliminate the argument for AI risk; even allowing that, AIs could still become more competent than us and eventually, some of them could get lucky too. My impression of the original discussion was that the argument wasn’t meant as an argument against all AI risk, but rather just against hard takeoff-type scenarios depicting a single AI that takes over the world by being supremely intelligent and agentic.