The “mistake” Michael is talking about it the belief that utility maximization can lead to counter intuitive actions, in particular actions that humanly speaking are bound to be useless, such as accepting a Wager or a Mugging.
This is in fact not a mistake at all, but a simple fact (as Carl Shulman and Nick Tarleton suspect.) The belief that it does not is simply a result of Anthropomorphic Optimism as Eliezer describes it; i.e. “This particular optimization process, especially because it satisfies certain criteria of rationality, must come to the same conclusions I do.” Have you ever considered the possibility that your conclusions do not satisfy those criteria of rationality?
The “mistake” Michael is talking about it the belief that utility maximization can lead to counter intuitive actions, in particular actions that humanly speaking are bound to be useless, such as accepting a Wager or a Mugging.
This is in fact not a mistake at all, but a simple fact (as Carl Shulman and Nick Tarleton suspect.) The belief that it does not is simply a result of Anthropomorphic Optimism as Eliezer describes it; i.e. “This particular optimization process, especially because it satisfies certain criteria of rationality, must come to the same conclusions I do.” Have you ever considered the possibility that your conclusions do not satisfy those criteria of rationality?