Agreed that superhuman intelligence seems like the kind of thing that could be a very powerful lever. What gets me is that we don’t seem to know how orthogonal or non-orthogonal intelligence and empathy are to one another.[1] If we were capable of creating a superhumanly intelligent AI and we were to be able to give it superhuman empathy, I might be inclined to trust ceding over a large amount of power and control to that system (or set of systems whatever). But a sociopathic superhuman intelligence? Definitely not ceding power over to that system.
The question then becomes to me, how confident are we that we are not creating dangerously sociopathic AI?
If I were to take a stab, I would say they were almost entirely orthogonal, as we have perfectly intelligent yet sociopathic humans walking around today who lack any sort of empathy. Giving any of these people superhuman ability and control would seem like an obviously terrible idea to me.
Agreed that superhuman intelligence seems like the kind of thing that could be a very powerful lever. What gets me is that we don’t seem to know how orthogonal or non-orthogonal intelligence and empathy are to one another.[1] If we were capable of creating a superhumanly intelligent AI and we were to be able to give it superhuman empathy, I might be inclined to trust ceding over a large amount of power and control to that system (or set of systems whatever). But a sociopathic superhuman intelligence? Definitely not ceding power over to that system.
The question then becomes to me, how confident are we that we are not creating dangerously sociopathic AI?
If I were to take a stab, I would say they were almost entirely orthogonal, as we have perfectly intelligent yet sociopathic humans walking around today who lack any sort of empathy. Giving any of these people superhuman ability and control would seem like an obviously terrible idea to me.