Ok, let’s do some basic friendly AI theory: Would a friendly AI discount the welfare of “weaker” beings as you and me (compared to this hyper-agent) lexically? Could that possibly be a fAI? If not, then I think we should also rethink our moral behaviour towards weaker beings in our game here for our decisions can result in bad things for them correspondingly.
My bad about the ritual. Thanks. Out of interest about your preferences: Imagine the grandmother and the dog next to each other. A perfect scientist starts to exchange pairs of atoms (let’s assume here that both individuals contain the same amount of atoms) so that the grandmother more and more transforms into the dog (of course there will be several weird intermediary stages). For the scientist knows his experiment very well, none of the objects will die; in the end it’ll look like the two objects changed their places. At which point does the mother stop counting lexically more than the dog? Sometimes continuity arguments can be defeated by saying: “No I don’t draw an arbitrary line; I adjust gradually whereas in the beginning I care a lot about the grandmother and in the end just very little about the remaining dog.” But I think that this argument doesn’t work here for we deal with a lexical prioritization. How would you act in such a scenario?
It seems to me that we often treat EDT decisions with some sort of hindsight bias. For instance, given that we know that the action A (turning on sprinklers) doesn’t increase the probability of the outcome O (rain) it looks very foolish to do A. Likewise, a DT that suggests doing A may look foolish. But isn’t the point here that the deciding agent doesn’t know that? All he knows is, that P(E|A)>P(E) and P(O|E)>P(O). Of course A still might have no or even a negative causal effect on O, but yet we have more reason the believe otherwise. To illustrate that, consider the following scenario:
Imagine you find yourself in a white room with one red button. You have no idea why you’re there and what this is all about. During the first half hour you are undecided whether you should press the button. Finally your curiosity dominates other considerations and you press the button. Immediately you feel a blissful release of happiness hormones. If you use induction it seems plausible to infer that considering certain time intervalls (f.i. of 1 minute) P(bliss|button) > P(bliss). Now the effect has ceased and you wished to be shot up again. Is it now rational to press the button a second time? I would say yes. And I don’t think that this is controversial. And since we can repeat the pattern with further variables it should also work with the example above.
From that point of view it doesn’t seem to be foolish at all—talking about the sprinkler again—to have a non-zero credence in A (turning sprinkler on) increasing the probability of O (rain). In situations with that few knowledge and no further counter-evidence (which f.i. might suggest that A might have no or a negative influence on O) this should lead an agent to do A.
Considering the doctor, again, I think we have to stay clear about what the doctor actually knows. Let’s imagine a doctor who lost all his knowledge about medicine. Now he reads one study which shows that P(Y|A) > P(Y). It seems to me that given that piece of information (and only that!) a rational doctor shouldn’t do A. However, once he reads the next study he can figure out that C (the trait “cancer”) is confounding the previous assessment because most patients who are treated with A show C as well, whereas ~A don’t. This update (depending on the probability values respectively) will then lead to a shift favoring the action A again.
To summarize: I think many objections against EDT fail once we really clarify what the agent knows respectively. In scenarios with few knowledge EDT seems to give the right answers. Once we add further knowledge an EDT updates his beliefs and won’t turn on the sprinkler in order to increase the probability of rain. As we know it from the hindsight bias it might be difficult to really imagine what actually would be different if we didn’t know what we do know now.
Maybe that’s all streaked with flaws, so if you find some please hand me over the lottery tickets ; )