For my own part, I neither find it likely that an arbitrarily selected superintelligence will be “super-moral” given the ordinary connotations of that term, nor that it will be immoral given the ordinary connotations of that term. I do expect it to be amoral by my standards.
That it’s an AI is irrelevant; I conclude much the same thing about arbitrarily selected superintelligent NIs. (Of course, if I artificially limit my selection space to superintelligent humans, my predictions change.)
FWIW, an “arbitrarily selected superintelligence” is not what I meant at all. I was talking about the superintelligences we are likely to see—which will surely not be “arbitrarily selected”.
While thinking about “arbitrarily selected superintelligences” might make superintelligence seem scary, the concept has relatively little to do with reality. It is like discussing arbitrarily selected computer programs. Fun for philosophers—maybe—but not much use for computer scientists or anyone interested in how computer programs actually behave in the real world.
I’ll certainly agree that human-created superintelligences are more likely to be moral in human terms than, say, dolphin-created superintelligences or alien superintelligences.
If I (for example) restrict myself to the class of superintelligences built by computer programmers, it seems reasonable to assume their creators will operate substantively like the computer programmers I’ve worked with (and known at places like MIT’s AI Lab). That assumption leads me to conclude that insofar as they have a morality at all, that morality will be constructed as a kind of test harness around the underlying decision procedure, under the theory that the important problem is making the right decisions given a set of goals. That leads me to expect the morality to be whatever turns out to be easiest to encode and not obviously evil. I’m not sure what the result of that is, but I’d be surprised if I recognized it as moral.
If I instead restrict myself to the class of superintelligences constructed by intelligence augmentation of humans, say, I expect the resulting superintelligence to work out a maximally consistent extension of human moral structures. I expect the result to be recognizably moral as long as we unpack that morality using terms like “systems sufficiently like me” rather than terms like “human beings.” Given how humans treat systems as much unlike us as unaugmented humans are unlike superintelligent humans, I’m not looking forward to that either.
So… I dunno. I’m reluctant to make any especially confident statement about the morality of human-created superintelligences, but I certainly don’t consider “super-moral” some kind of default condition that we’re more likely to end up in than we are to miss.
For my own part, I neither find it likely that an arbitrarily selected superintelligence will be “super-moral” given the ordinary connotations of that term, nor that it will be immoral given the ordinary connotations of that term. I do expect it to be amoral by my standards.
That it’s an AI is irrelevant; I conclude much the same thing about arbitrarily selected superintelligent NIs. (Of course, if I artificially limit my selection space to superintelligent humans, my predictions change.)
FWIW, an “arbitrarily selected superintelligence” is not what I meant at all. I was talking about the superintelligences we are likely to see—which will surely not be “arbitrarily selected”.
While thinking about “arbitrarily selected superintelligences” might make superintelligence seem scary, the concept has relatively little to do with reality. It is like discussing arbitrarily selected computer programs. Fun for philosophers—maybe—but not much use for computer scientists or anyone interested in how computer programs actually behave in the real world.
I’ll certainly agree that human-created superintelligences are more likely to be moral in human terms than, say, dolphin-created superintelligences or alien superintelligences.
If I (for example) restrict myself to the class of superintelligences built by computer programmers, it seems reasonable to assume their creators will operate substantively like the computer programmers I’ve worked with (and known at places like MIT’s AI Lab). That assumption leads me to conclude that insofar as they have a morality at all, that morality will be constructed as a kind of test harness around the underlying decision procedure, under the theory that the important problem is making the right decisions given a set of goals. That leads me to expect the morality to be whatever turns out to be easiest to encode and not obviously evil. I’m not sure what the result of that is, but I’d be surprised if I recognized it as moral.
If I instead restrict myself to the class of superintelligences constructed by intelligence augmentation of humans, say, I expect the resulting superintelligence to work out a maximally consistent extension of human moral structures. I expect the result to be recognizably moral as long as we unpack that morality using terms like “systems sufficiently like me” rather than terms like “human beings.” Given how humans treat systems as much unlike us as unaugmented humans are unlike superintelligent humans, I’m not looking forward to that either.
So… I dunno. I’m reluctant to make any especially confident statement about the morality of human-created superintelligences, but I certainly don’t consider “super-moral” some kind of default condition that we’re more likely to end up in than we are to miss.