Ethics is a top-level problem that, even when “solved”, can only be applied adequately once you have been able to define mathematically what constitutes a human being or values. If you can’t even define some of the subject matter of ethics, e.g. pain and consciousness, then how will an AI know what constitutes a valid input of its friendliness function?
If you believe that an AI is able to figure all that out on its own, why won’t it be able to do the same with ethics? And if not, then why do you think ethics is of particular importance to friendly AI research if there are so many other basic problems to be solved before?
And if not, then why do you think ethics is of particular importance to friendly AI research if there are so many other basic problems to be solved before?
The way I see it, all these problems are interrelated, and it’s hard to say which ones can or should be solved first. I think it is reasonable to pursue multiple approaches simultaneously. Yes it seems hard to solve ethics without first understanding pain and consciousness, but it is possible that the correct theory of ethics does not use “pain” or “consciousness”. Perhaps it only depends on “desire”. And if ethics is related to consciousness, say, it can be useful to develop them simultaneously so that we can check whether our theory of consciousness is compatible with our theory of ethics.
After a while, smart machines will probably know what a human is better than individual humans do—due to all the training cases we can easily feed them.
“Is this behaviour ethical” generally seems to be a trickier categorisation problem—humans disagree about it more, it is more a matter of degree—etc.
The whole: “can’t we just let the S-I-M figure it out?” business seems kind of paralysing. Should we stop working on math or physics because the S-I-M will figure it out? No—because we need to use that stuff in the mean time.
I should add:
Ethics is a top-level problem that, even when “solved”, can only be applied adequately once you have been able to define mathematically what constitutes a human being or values. If you can’t even define some of the subject matter of ethics, e.g. pain and consciousness, then how will an AI know what constitutes a valid input of its friendliness function?
If you believe that an AI is able to figure all that out on its own, why won’t it be able to do the same with ethics? And if not, then why do you think ethics is of particular importance to friendly AI research if there are so many other basic problems to be solved before?
The way I see it, all these problems are interrelated, and it’s hard to say which ones can or should be solved first. I think it is reasonable to pursue multiple approaches simultaneously. Yes it seems hard to solve ethics without first understanding pain and consciousness, but it is possible that the correct theory of ethics does not use “pain” or “consciousness”. Perhaps it only depends on “desire”. And if ethics is related to consciousness, say, it can be useful to develop them simultaneously so that we can check whether our theory of consciousness is compatible with our theory of ethics.
After a while, smart machines will probably know what a human is better than individual humans do—due to all the training cases we can easily feed them.
“Is this behaviour ethical” generally seems to be a trickier categorisation problem—humans disagree about it more, it is more a matter of degree—etc.
The whole: “can’t we just let the S-I-M figure it out?” business seems kind of paralysing. Should we stop working on math or physics because the S-I-M will figure it out? No—because we need to use that stuff in the mean time.