I should be explicit that I don’t assign strong confidence to either of the following intuitions, but FWIW:
One point where intuitively this may be more likely to fail than my trick is when you try to replace yourself by something that’s expected to be better than you at maximizing its expected utility with bounded computational resources. If you want something better than yourself, it seems rather limiting to require that it won’t do anything you wouldn’t do. (That’s an argument against quining approaches being the right thing, not for my approach being that.) -- On the other hand, an intuition in the other direction would be that if your original AI has a parameter n such that for n → infinity, it converges to an expected utility maximizer, then your approach might still lead to an increasing sequence of better and better EU maximizers.
I should be explicit that I don’t assign strong confidence to either of the following intuitions, but FWIW:
One point where intuitively this may be more likely to fail than my trick is when you try to replace yourself by something that’s expected to be better than you at maximizing its expected utility with bounded computational resources. If you want something better than yourself, it seems rather limiting to require that it won’t do anything you wouldn’t do. (That’s an argument against quining approaches being the right thing, not for my approach being that.) -- On the other hand, an intuition in the other direction would be that if your original AI has a parameter n such that for n → infinity, it converges to an expected utility maximizer, then your approach might still lead to an increasing sequence of better and better EU maximizers.