For any given concept of “rational (action)” that’s not defined as “(the action) arranging for the best expected winning”, you can of course find a situation where that concept and winning are at odds. But if you define them to be the same, it’s no longer possible. At that point, you can be taxed for being a given program and not other program (of for the fact that pi is less than 10, for that matter), something you don’t control, but such criterion won’t be about rationality of your decision-making, because it doesn’t provide a suggestion one can theoretically use to improve one’s performance. For example, “be less rational” would translate as “act in a way that leads to less winning”, which is never desirable.
Note the distinction between Eliezer’s quote and AdamBell’s quote: Eliezer specifically talks about rationality of actions, a notion that’s not vulnerable to taxing your algorithm, while AdamBell talks less concretely about “choosing decision theory”, which is ambiguous between “choosing your program” (which is not possible), and choosing to self-improve (which is just a very special kind of action).
For any given concept of “rational (action)” that’s not defined as “(the action) arranging for the best expected winning”, you can of course find a situation where that concept and winning are at odds. But if you define them to be the same, it’s no longer possible. At that point, you can be taxed for being a given program and not other program (of for the fact that pi is less than 10, for that matter), something you don’t control, but such criterion won’t be about rationality of your decision-making, because it doesn’t provide a suggestion one can theoretically use to improve one’s performance. For example, “be less rational” would translate as “act in a way that leads to less winning”, which is never desirable.
Note the distinction between Eliezer’s quote and AdamBell’s quote: Eliezer specifically talks about rationality of actions, a notion that’s not vulnerable to taxing your algorithm, while AdamBell talks less concretely about “choosing decision theory”, which is ambiguous between “choosing your program” (which is not possible), and choosing to self-improve (which is just a very special kind of action).