>Anthropically, our existence provides evidence for them being favored.
There are some complications here. It depends a bit on how you make anthropic updates (if you do them at all). But it turns out that the version of updating that “works” with EDT basically doesn’t make the update that you’re in the majority. See my draft on decision making with anthropic updates.
>It seems plausible that evolutionary pressures select for utility functions broadly as ours
Well, at least in some ways similar as ours, right? On questions like whether rooms are better painted red or green, I assume there isn’t much reason to expect convergence. But on questions of whether happiness is better than suffering, I think one should expect evolved agents to mostly give the right answers.
>to compare such maximizations, you already need a decision theory (which tells you what “maximizing your goals” even is).
Incidentally I published a blog post about this only a few weeks ago (which will probably not contain any ideas that are new to you).
>Might there be some situation in which an agent wants to ensure all of its correlates are Good Twins
Yep! That was also my intuition behind “all meta-updates (hopes and fears) balancing out” above.
I don’t think this is possible.
If you mean it’s not possible to ensure all your correlates are Good, I don’t see how doing more research about the question doesn’t get you ever closer to that (even if you never reach the ideal limit of literally all correlates being Good).
If you mean no one would want to do that, it might seem like you’d be happy to be uncorrelated from your Evil Twins. But this might again be a naïve view that breaks upon considering meta-reasoning.
>Anthropically, our existence provides evidence for them being favored.
There are some complications here. It depends a bit on how you make anthropic updates (if you do them at all). But it turns out that the version of updating that “works” with EDT basically doesn’t make the update that you’re in the majority. See my draft on decision making with anthropic updates.
>Annex: EDT being counter-intuitive?
I mean, in regular probability calculus, this is all unproblematic, right? Because of the Tower Rule a.k.a. Law of total expectation or similarly conservation of expected evidence. There are also issues of updatelessness, though, you touch on at various places in the post. E.g., see Almond’s “lack of knowledge is [evidential] power” or scenarios like the Transparent Newcomb’s problem wherein EDT wants to prevent itself from seeing the content of the boxes.
>It seems plausible that evolutionary pressures select for utility functions broadly as ours
Well, at least in some ways similar as ours, right? On questions like whether rooms are better painted red or green, I assume there isn’t much reason to expect convergence. But on questions of whether happiness is better than suffering, I think one should expect evolved agents to mostly give the right answers.
>to compare such maximizations, you already need a decision theory (which tells you what “maximizing your goals” even is).
Incidentally I published a blog post about this only a few weeks ago (which will probably not contain any ideas that are new to you).
>Might there be some situation in which an agent wants to ensure all of its correlates are Good Twins
I don’t think this is possible.
Thank you for your comment! I hadn’t had the time to read de se choice but am looking forward to it! Thank you also for the other recommendations.
Yep! That was also my intuition behind “all meta-updates (hopes and fears) balancing out” above.
If you mean it’s not possible to ensure all your correlates are Good, I don’t see how doing more research about the question doesn’t get you ever closer to that (even if you never reach the ideal limit of literally all correlates being Good).
If you mean no one would want to do that, it might seem like you’d be happy to be uncorrelated from your Evil Twins. But this might again be a naïve view that breaks upon considering meta-reasoning.