@Eliezer—I think Shane is right. “Good” abstractions do exist, and are independent of the observer. The value of an abstraction relates to its ability to allow you to predict the future. For example, “mass” is a good abstraction, because when coupled with a physical law it allows you to make good predictions.
If we assume a superintelligent AI, we have to assume that the AI has the ability to discover abstractions. Human happiness is one such abstraction. Understanding the abstraction “happiness” allows one to predict certain events related to human activity. Thus a superintelligent AI will necessarily develop the concept of happiness in order to allow it to predict human events, in much the same way that it will develop a concept of mass in order to predict physical events.
Plato had a concept of “forms”. Forms are ideal shapes or abstractions: every dog is an imperfect instantiation of the “dog” form that exists only in our brains. If we can accept the existence of a “dog” form or a “house” form or a “face” form, then it is not difficult to believe in the existence of a “good” form. Plato called this the Form of the Good. If we assume an AI that can develop its own forms, then it should be able to discover the Form of the Good.
Whether or not the AI finds the abstraction of human happiness to be pertinent, and whether it considers increasing it to be worthwhile sacrificing other possible benefits for, are unpredictable, unless we have succeeded in achieving EY’s goal of pre-destining the AI to be Friendly.
@Eliezer—I think Shane is right. “Good” abstractions do exist, and are independent of the observer. The value of an abstraction relates to its ability to allow you to predict the future. For example, “mass” is a good abstraction, because when coupled with a physical law it allows you to make good predictions.
If we assume a superintelligent AI, we have to assume that the AI has the ability to discover abstractions. Human happiness is one such abstraction. Understanding the abstraction “happiness” allows one to predict certain events related to human activity. Thus a superintelligent AI will necessarily develop the concept of happiness in order to allow it to predict human events, in much the same way that it will develop a concept of mass in order to predict physical events.
Plato had a concept of “forms”. Forms are ideal shapes or abstractions: every dog is an imperfect instantiation of the “dog” form that exists only in our brains. If we can accept the existence of a “dog” form or a “house” form or a “face” form, then it is not difficult to believe in the existence of a “good” form. Plato called this the Form of the Good. If we assume an AI that can develop its own forms, then it should be able to discover the Form of the Good.
http://en.wikipedia.org/wiki/Form_of_the_Good
Whether or not the AI finds the abstraction of human happiness to be pertinent, and whether it considers increasing it to be worthwhile sacrificing other possible benefits for, are unpredictable, unless we have succeeded in achieving EY’s goal of pre-destining the AI to be Friendly.