Which individual? The might be some decision theory which promotes the interests of Joe Soap, against the interests of society, but there is no way i would call it morality.
Ah, I may have been unclear there.
To go into more detail, then; you appear to be suggesting that optimal morality can be approached as a society-wide optimisation problem; in the current situations, these moral strictures produce a more optimal society than those, and this optimisation problem can be solved with sufficient computational resources and information.
But now, let us consider an individual example. Let us say that I find a wallet full of money on the ground. There is no owner in sight. The optimal choice for the society as a whole is that I return the money to the original owner; the optimal choice for the individual making the decision is to keep the money and use it towards my aims, whatever those are. (I can be pretty sure that the man to whom I return the money will be putting it towards his aims, not mine, and if I’m sufficiently convinced that my aims are better for society than his then I can even rationalise this action).
By my current moral structures, I would have to return the money to its original owner. But I can easily see a superintelligent AI giving serious consideration to the possibility that it can do more good for the original owner with the money than the original owner could.
Its motivational system. We’re already assuming it’s motivated to make the deduction, we need to assume it’s motivated to implement.
This, right here, is the hard problem of Friendly AI. How do we make it motivated to implement? And, more importantly, how do we know that it is motivated to implement what we think it’s motivated to implement?
I am not bypassing the need for a goal driven AI to have appropriate goals, I am by passing the need for a detailed and accurate account of human ethics to be preprogrammed.
You’re suggesting that it can figure out the complicated day-to-day minutae and the difficult edge cases on its own, given a suitable algorithm for optimising morality.
My experience in software design suggests that that algorithm needs to be really, really good. And extremely thoroughly checked, from every possible angle, by a lot of people.
I’m not denying that such an algorithm potentially exists. I can just think of far, far too many ways for it to go very badly wrong.
I am not sayngn it necessarily does not. I am saying it does not necessarily.
...point taken. It may or may not share those values.
But then we must at least give serious consideration to the worst-case scenario.
Ah, I may have been unclear there.
To go into more detail, then; you appear to be suggesting that optimal morality can be approached as a society-wide optimisation problem; in the current situations, these moral strictures produce a more optimal society than those, and this optimisation problem can be solved with sufficient computational resources and information.
But now, let us consider an individual example. Let us say that I find a wallet full of money on the ground. There is no owner in sight. The optimal choice for the society as a whole is that I return the money to the original owner; the optimal choice for the individual making the decision is to keep the money and use it towards my aims, whatever those are. (I can be pretty sure that the man to whom I return the money will be putting it towards his aims, not mine, and if I’m sufficiently convinced that my aims are better for society than his then I can even rationalise this action).
By my current moral structures, I would have to return the money to its original owner. But I can easily see a superintelligent AI giving serious consideration to the possibility that it can do more good for the original owner with the money than the original owner could.
This, right here, is the hard problem of Friendly AI. How do we make it motivated to implement? And, more importantly, how do we know that it is motivated to implement what we think it’s motivated to implement?
You’re suggesting that it can figure out the complicated day-to-day minutae and the difficult edge cases on its own, given a suitable algorithm for optimising morality.
My experience in software design suggests that that algorithm needs to be really, really good. And extremely thoroughly checked, from every possible angle, by a lot of people.
I’m not denying that such an algorithm potentially exists. I can just think of far, far too many ways for it to go very badly wrong.
...point taken. It may or may not share those values.
But then we must at least give serious consideration to the worst-case scenario.