This was Eliezer’s point: how could you ever recognize which ones are good and which ones are evil? How could you even recognize a process for recognizing objective good and evil?
I have only one suggestion so far, which is that if you find yourself in a situation which satisfies all five of the conditions I just listed, obeying the Mugger initiates an indefinitely-long causal chain that is good rather than evil. I consider, “You might as well assume it is good,” to be equivalent to, “It is good.” Now that I have an example I can try to generalize it, which is best done after the scenario has been expressed mathematically. That is my plan of research. So for example I am going to characterize mathematically the notion of a possible world in which an agent can become confident of a “negative fact” about its environment. An example of a negative fact is, I will probably not be able to refine further my model of the Mugger using any evidence except what the Mugger tells me. Then I will try to determine whether our reality is an example of a possible world that allows agents to become confident of negative facts. I will try to devise a way to compute an answer to the question of how to trade off the two goals of obeying the Mugger and refining my model of reality.
A moral system must contain some postulates. I have retracted my claim that one can derive ought from is and apologize for advancing it. Above I give a list four postulates I consider unobjectionable—the list whose last item is Occam’s razor. I do not claim that you and I will come to agree on the fundamental moral postulates if we knew more, thought faster, were more the people we wished we were, had grown up farther together. I do not claim that we have or can discover a procedure that allows two rational humans always to cooperate. I do not claim that this is the summer of love. I reserve the right to continue to advocate for my fundamental moral postulates even if it causes conflict.
This was Eliezer’s point: how could you ever recognize which ones are good and which ones are evil? How could you even recognize a process for recognizing objective good and evil?
I have only one suggestion so far, which is that if you find yourself in a situation which satisfies all five of the conditions I just listed, obeying the Mugger initiates an indefinitely-long causal chain that is good rather than evil. I consider, “You might as well assume it is good,” to be equivalent to, “It is good.” Now that I have an example I can try to generalize it, which is best done after the scenario has been expressed mathematically. That is my plan of research. So for example I am going to characterize mathematically the notion of a possible world in which an agent can become confident of a “negative fact” about its environment. An example of a negative fact is, I will probably not be able to refine further my model of the Mugger using any evidence except what the Mugger tells me. Then I will try to determine whether our reality is an example of a possible world that allows agents to become confident of negative facts. I will try to devise a way to compute an answer to the question of how to trade off the two goals of obeying the Mugger and refining my model of reality.
A moral system must contain some postulates. I have retracted my claim that one can derive ought from is and apologize for advancing it. Above I give a list four postulates I consider unobjectionable—the list whose last item is Occam’s razor. I do not claim that you and I will come to agree on the fundamental moral postulates if we knew more, thought faster, were more the people we wished we were, had grown up farther together. I do not claim that we have or can discover a procedure that allows two rational humans always to cooperate. I do not claim that this is the summer of love. I reserve the right to continue to advocate for my fundamental moral postulates even if it causes conflict.