That’s a really fascinating question. I don’t know that there’d be a “standard” answer to this—were the questions taken up, they’d be subject to hot debate.
Are we specifying that this ultrapowerful superintelligence has mind-reading power, or the closest non-magical equivalent in the form of access to every mental state that an arbitrary individual human has, even stuff that now gets lumped under the label “qualia”/ability to perfectly simulate the neurobiology of such an individual?
If so, then two approaches seem defensible to me. First: let’s assume there is an answer out there to moral questions, in a form that is accessible to a superintelligence, and let’s just assume the hard problem away, viz., that the questioners know how to tell the superintelligence where to look (or the superintelligence can figure it out itself).
We might not be able to produce a well-formed specification of what is to be computed when we’re talking about moral questions (it’s easy to think that any attempt to do so would rig the answer in advance—for example, if you ask it for universal principles, you’re going to get something different from what you’d get if you left the universality variable free...). But if the superintelligence could simulate our mental processes such that it could tell what it is that we want (for some appropriate values of we, like the person asking or the whole of humanity if there was any consensus—which I doubt), then in principle it could simply answer that by declaring what the truth of the matter is with respect to that which it has determined that we desire.
That assumes the superintelligence has access to moral truth, but once we do that, I think the standard arguments against “guardianship” (e.g. the first few chapters of Robert Dahl, Democracy and its Critics) fail, in that if they’re true—if people are really better off deciding for themselves (the standard argument), and making people better off is what is morally correct, then we can expect the superintelligence to return “you figure it out.” And then the answer to “friendly to who” or “so you get to decide what’s friendly” is simply to point to the fact that the superintelligence has access to moral truth.
The more interesting question perhaps is what should happen if the superintelligence doesn’t have access to moral truth (either because there is no such thing in the ordinary sense, or because it exists but is unobservable). I assume here that being responsive to reasons is an appropriate way to address moral questions (if not, all bets are off). Then the superintelligence loses one major advantage over ordinary human reasoning (access to the truth on the question), but not the other (while humans are responsive to reasons in a limited and inconsistent sense, the supercomputer is ideally responsive to reasons). For this situation, I think the second defensible outcome would be that the superintelligence should simulate ideal democracy. That is, it should simulate all the minds in the world, and put them into an unlimited discussion with one another, as if they were bayesians with infinite time. The answers it would come up with would be the equivalent to the most legitimate conceivable human decisional process, but better...
I’m pretty sure this is a situation that hasn’t come under sustained discussion in the literature as such (in superintelligence terms—though it has come up in discussions of benevolent dictators and the value of democracy), so I’m talking out my ass a little here, but drawing on familiar themes. Still, the argument defending these two notions—especially the second—isn’t a blog comment, it’s a series of long articles or more.
That’s a really fascinating question. I don’t know that there’d be a “standard” answer to this—were the questions taken up, they’d be subject to hot debate.
Are we specifying that this ultrapowerful superintelligence has mind-reading power, or the closest non-magical equivalent in the form of access to every mental state that an arbitrary individual human has, even stuff that now gets lumped under the label “qualia”/ability to perfectly simulate the neurobiology of such an individual?
If so, then two approaches seem defensible to me. First: let’s assume there is an answer out there to moral questions, in a form that is accessible to a superintelligence, and let’s just assume the hard problem away, viz., that the questioners know how to tell the superintelligence where to look (or the superintelligence can figure it out itself).
We might not be able to produce a well-formed specification of what is to be computed when we’re talking about moral questions (it’s easy to think that any attempt to do so would rig the answer in advance—for example, if you ask it for universal principles, you’re going to get something different from what you’d get if you left the universality variable free...). But if the superintelligence could simulate our mental processes such that it could tell what it is that we want (for some appropriate values of we, like the person asking or the whole of humanity if there was any consensus—which I doubt), then in principle it could simply answer that by declaring what the truth of the matter is with respect to that which it has determined that we desire.
That assumes the superintelligence has access to moral truth, but once we do that, I think the standard arguments against “guardianship” (e.g. the first few chapters of Robert Dahl, Democracy and its Critics) fail, in that if they’re true—if people are really better off deciding for themselves (the standard argument), and making people better off is what is morally correct, then we can expect the superintelligence to return “you figure it out.” And then the answer to “friendly to who” or “so you get to decide what’s friendly” is simply to point to the fact that the superintelligence has access to moral truth.
The more interesting question perhaps is what should happen if the superintelligence doesn’t have access to moral truth (either because there is no such thing in the ordinary sense, or because it exists but is unobservable). I assume here that being responsive to reasons is an appropriate way to address moral questions (if not, all bets are off). Then the superintelligence loses one major advantage over ordinary human reasoning (access to the truth on the question), but not the other (while humans are responsive to reasons in a limited and inconsistent sense, the supercomputer is ideally responsive to reasons). For this situation, I think the second defensible outcome would be that the superintelligence should simulate ideal democracy. That is, it should simulate all the minds in the world, and put them into an unlimited discussion with one another, as if they were bayesians with infinite time. The answers it would come up with would be the equivalent to the most legitimate conceivable human decisional process, but better...
I’m pretty sure this is a situation that hasn’t come under sustained discussion in the literature as such (in superintelligence terms—though it has come up in discussions of benevolent dictators and the value of democracy), so I’m talking out my ass a little here, but drawing on familiar themes. Still, the argument defending these two notions—especially the second—isn’t a blog comment, it’s a series of long articles or more.