I think the answer to ‘where is Eliezer getting this from’ can be found in the genesis of the paperclip maximizer scenario. There’s an older post on LW talking about ‘three types of genie’ and another on someone using a ‘utility pump’ (or maybe it’s one and the same post?), where Eliezer starts from the premise that we create an artifical intelligence to ‘make something specific happen for us’, with the predictable outcome that the AI finds a clever solution which maximizes for the demanded output, one that naturally has nothing to do with what we ‘really wanted from it’. If asked to produce smiles, it will manufacture molecular smiley faces, and it will do its best to prevent us from executing this splendid plan.
This scenario, to me, seems much more realistic and likely to occur in the near-term than an AGI with full self-reflective capacities either spontaneously materializing or being created by us (where would we even start on that one)? AI, more than anything else, is a kind of transhumanist dream, a deus ex machina that will grant all good wishes and make the world into the place they (read:people who imagine themselves as benevolent philosopher kings) want it to be ー so they’ll build a utility maximizer and give it a very painstakingly thought-through list of instructions, and the genie will inevitably find a loophole that lets it follow those instructions to the letter, with no regard for its spirit.
It’s not the only kind of AI that we could build, but it will likely be the first, and, if so, it will almost certainly also be the last.
I think the answer to ‘where is Eliezer getting this from’ can be found in the genesis of the paperclip maximizer scenario. There’s an older post on LW talking about ‘three types of genie’ and another on someone using a ‘utility pump’ (or maybe it’s one and the same post?), where Eliezer starts from the premise that we create an artifical intelligence to ‘make something specific happen for us’, with the predictable outcome that the AI finds a clever solution which maximizes for the demanded output, one that naturally has nothing to do with what we ‘really wanted from it’. If asked to produce smiles, it will manufacture molecular smiley faces, and it will do its best to prevent us from executing this splendid plan.
This scenario, to me, seems much more realistic and likely to occur in the near-term than an AGI with full self-reflective capacities either spontaneously materializing or being created by us (where would we even start on that one)?
AI, more than anything else, is a kind of transhumanist dream, a deus ex machina that will grant all good wishes and make the world into the place they (read:people who imagine themselves as benevolent philosopher kings) want it to be ー so they’ll build a utility maximizer and give it a very painstakingly thought-through list of instructions, and the genie will inevitably find a loophole that lets it follow those instructions to the letter, with no regard for its spirit.
It’s not the only kind of AI that we could build, but it will likely be the first, and, if so, it will almost certainly also be the last.