I mostly agree with what you say here—which is why I said the criticisms were exaggerated, not totally wrong—but I do think the classic arguments are still better than you portray them. In particular, I don’t remember coming away from Superintelligence (I read it when it first came out) thinking that we’d have an AI system capable of optimizing any goal and we’d need to figure out what goal to put into it. Instead I thought that we’d be building AI through some sort of iterative process where we look at existing systems, come up with tweaks, build a new and better system, etc. and that if we kept with the default strategy (which is to select for and aim for systems with the most impressive capabilities/intelligence, and not care about their alignment—just look at literally every AI system made in the lab so far! Is AlphaGo trained to be benevolent? Is AlphaStar? Is GPT? Etc.) then probably doom.
It’s true that when people are building systems not for purposes of research, but for purposes of economic application—e.g. Alexa, Google Search, facebook’s recommendation algorithm—then they seem to put at least some effort into making the systems aligned as well as intelligent. However history also tells us that not very much effort is put in, by default, and that these systems would totally kill us all if they were smarter. Moreover, usually systems appear in research-land first before they appear in economic-application-land. This is what I remember myself thinking in 2014, and I still think it now. I think the burden of proof has totally not been met; we still don’t have good reason to think the outcome will probably be non-doom in the absence of more AI safety effort.
It’s possible my memory is wrong though. I should reread the relevant passages.
When I wrote that I was mostly taking what Ben Garfinkel said about the ‘classic arguments’ at face value, but I do recall that there used to be a lot of loose talk about putting values into an AGI after building it.
I mostly agree with what you say here—which is why I said the criticisms were exaggerated, not totally wrong—but I do think the classic arguments are still better than you portray them. In particular, I don’t remember coming away from Superintelligence (I read it when it first came out) thinking that we’d have an AI system capable of optimizing any goal and we’d need to figure out what goal to put into it. Instead I thought that we’d be building AI through some sort of iterative process where we look at existing systems, come up with tweaks, build a new and better system, etc. and that if we kept with the default strategy (which is to select for and aim for systems with the most impressive capabilities/intelligence, and not care about their alignment—just look at literally every AI system made in the lab so far! Is AlphaGo trained to be benevolent? Is AlphaStar? Is GPT? Etc.) then probably doom.
It’s true that when people are building systems not for purposes of research, but for purposes of economic application—e.g. Alexa, Google Search, facebook’s recommendation algorithm—then they seem to put at least some effort into making the systems aligned as well as intelligent. However history also tells us that not very much effort is put in, by default, and that these systems would totally kill us all if they were smarter. Moreover, usually systems appear in research-land first before they appear in economic-application-land. This is what I remember myself thinking in 2014, and I still think it now. I think the burden of proof has totally not been met; we still don’t have good reason to think the outcome will probably be non-doom in the absence of more AI safety effort.
It’s possible my memory is wrong though. I should reread the relevant passages.
When I wrote that I was mostly taking what Ben Garfinkel said about the ‘classic arguments’ at face value, but I do recall that there used to be a lot of loose talk about putting values into an AGI after building it.