The fact that your post was upvoted so much makes me take it seriously; I want to understand it better. Currently I see your post as merely a general skeptical worry. Sure, maybe we should never be very confident in our FAI-predictions, but to the extent that we are confident, we can allow that confidence to influence our other beliefs and decisions, and we should be confident in some things to some extent at least (the alternative, complete and paralyzing skepticism, is absurd) Could you explain more what you meant, or explain what you think my mistake is in the above reasoning?
Of course, Bayesians want to be Mr. degrees-of-belief Carneades, not Mr. know-nothing Berkeley. Far be it from me to suggest that we ought to stop making models. It just worried me that you were so willing to adjust your behavior based on inherently untrustworthy predictions.
An acquaintance of mine liked to claim that superhuman intelligence was one of Superman’s powers. The idea immediately struck me as contradictory: nothing Superman does will ever be indicative of superhuman intelligence as long as the scriptwriter is human. My point here is the same: Your model of an ideal FAI will fall short of accurately simulating the ideal just as much as your own mind falls short of being ideal.
It is easier to determine whether you are doing “better” than your current self than it is to determine how well you line up with a perceived ideal being. So perhaps the lesson to take away is to try to just be better rather than be perfect.
It is easier to determine whether you are doing “better” than your current self than it is to determine how well you line up with a perceived ideal being.
Really? That doesn’t seem obvious to me. Could you justify that claim?
An “ideal” being is many layers of “better” than you are, whereas something that is simply better is only one layer better. To get to ideal, you would have to imagine someone better, then imagine what that person would consider better, and so forth, until you hit a state where there are no further improvements to be made.
In the picture you just drew, the ideal being is derived from a series of better beings, thus it is (trivially) easier to imagine a better being than to imagine an ideal being.
I see it differently: The ideal being maximizes all good qualities, whereas imperfect beings have differing levels of the various good qualities. Thus to compare a non-ideal being to an ideal being, we only need to recognize how the ideal being does better than the non-ideal being in each good quality. But to compare two non-ideal beings, we need to evaluate trade-offs between their various attributes (unless one is strictly greater than the other)
Thinking about it more, I am not happy with either of the above models. One question that arises is: Does the same reasoning extend to other cases as well? i.e. are we better off thinking about incremental improvements than about the ideal society? Are we better off thinking about incremental improvements than about the ideal chess algorithm?
I think in some cases maybe we are, but in some cases we aren’t—ideals are useful sometimes. I’d go farther to say that some aspects of many ideals must be arrived at by iterating, but other aspects can be concluded more directly. An uninteresting conclusion, but one that supports my overall point: I wasn’t claiming that I knew everything about the ideal FAI, just that I had justified high confidence in some things.
AFAICT you are not an ideal FAI, so your model of what an ideal FAI would do is always suspect.
The fact that your post was upvoted so much makes me take it seriously; I want to understand it better. Currently I see your post as merely a general skeptical worry. Sure, maybe we should never be very confident in our FAI-predictions, but to the extent that we are confident, we can allow that confidence to influence our other beliefs and decisions, and we should be confident in some things to some extent at least (the alternative, complete and paralyzing skepticism, is absurd) Could you explain more what you meant, or explain what you think my mistake is in the above reasoning?
Of course, Bayesians want to be Mr. degrees-of-belief Carneades, not Mr. know-nothing Berkeley. Far be it from me to suggest that we ought to stop making models. It just worried me that you were so willing to adjust your behavior based on inherently untrustworthy predictions.
An acquaintance of mine liked to claim that superhuman intelligence was one of Superman’s powers. The idea immediately struck me as contradictory: nothing Superman does will ever be indicative of superhuman intelligence as long as the scriptwriter is human. My point here is the same: Your model of an ideal FAI will fall short of accurately simulating the ideal just as much as your own mind falls short of being ideal.
This is a really good point.
It is easier to determine whether you are doing “better” than your current self than it is to determine how well you line up with a perceived ideal being. So perhaps the lesson to take away is to try to just be better rather than be perfect.
Really? That doesn’t seem obvious to me. Could you justify that claim?
An “ideal” being is many layers of “better” than you are, whereas something that is simply better is only one layer better. To get to ideal, you would have to imagine someone better, then imagine what that person would consider better, and so forth, until you hit a state where there are no further improvements to be made.
In the picture you just drew, the ideal being is derived from a series of better beings, thus it is (trivially) easier to imagine a better being than to imagine an ideal being.
I see it differently: The ideal being maximizes all good qualities, whereas imperfect beings have differing levels of the various good qualities. Thus to compare a non-ideal being to an ideal being, we only need to recognize how the ideal being does better than the non-ideal being in each good quality. But to compare two non-ideal beings, we need to evaluate trade-offs between their various attributes (unless one is strictly greater than the other)
Thinking about it more, I am not happy with either of the above models. One question that arises is: Does the same reasoning extend to other cases as well? i.e. are we better off thinking about incremental improvements than about the ideal society? Are we better off thinking about incremental improvements than about the ideal chess algorithm?
I think in some cases maybe we are, but in some cases we aren’t—ideals are useful sometimes. I’d go farther to say that some aspects of many ideals must be arrived at by iterating, but other aspects can be concluded more directly. An uninteresting conclusion, but one that supports my overall point: I wasn’t claiming that I knew everything about the ideal FAI, just that I had justified high confidence in some things.