The reference was mostly a reply to “a paperclipper can’t really be intelligent”. It can be intelligent in the sense relevant for AI risk.
I guess the current contenders for AGI are unlikely to become paperclippers, perhaps not even RL reduces to squiggle maximization. I think simple goals still give an important class of AIs, because such goals might be easier to preserve through recursive self-improvement, making AIs that pursue them afford faster FOOMing. AIs with complicated values might instead need to hold off on self-improvement much longer to ensure alignment, which makes them vulnerable to being overtaken by the FOOMing paperclippers. This motivates strong coordination that would prevent initial construction of paperclippers anywhere in the world.
The reference was mostly a reply to “a paperclipper can’t really be intelligent”. It can be intelligent in the sense relevant for AI risk.
I guess the current contenders for AGI are unlikely to become paperclippers, perhaps not even RL reduces to squiggle maximization. I think simple goals still give an important class of AIs, because such goals might be easier to preserve through recursive self-improvement, making AIs that pursue them afford faster FOOMing. AIs with complicated values might instead need to hold off on self-improvement much longer to ensure alignment, which makes them vulnerable to being overtaken by the FOOMing paperclippers. This motivates strong coordination that would prevent initial construction of paperclippers anywhere in the world.