I think this scenario is ingenious. Here are a few ideas, but I’m really not sure how far one can pursue them / how ‘much work’ they can do:
(1) Perhaps the agent needs some way of ‘absolving itself of responsibility’ for the evil/arbitrary/unreasonable actions of another being. The action to be performed is the one that yields highest expected utility but only along causal pathways that don’t go through an adversary that has been labelled as ‘unreasonable’.
(Except this approach doesn’t defuse the variation that goes “You can never wipe your nose because you’ve computed that the probability of this action killing 3^^^^3 people in a parallel universe is ever so slightly greater than the probability of it saving that number of people”.)
(2) We only have a fixed amount of ‘moral concern’, apportioned somehow or other to the beings we care about. Our utility function looks like: Sum(over beings X) ConcernFor(X)*HappinessOf(X). Allocation of ‘moral concern’ is a ‘competitive’ process. The only way we can gain some concern about Y is to lose a bit of concern about some X, but if we have regular and in some sense ‘positive’ interactions with X then our concern for X will be constantly ‘replenishing itself’. When the magician appears and tells us his story, we may acquire a tiny bit of concern about him and the people he mentions, but the parts of us that care about the people we know (a) aren’t ‘told’ the magician’s story and thus (b) refuse to ‘relinquish’ very much.
The trouble with is that it sounds too reminiscent of the insanely stupid moral behaviour of human beings (where e.g. they give exactly as much money to save a hundred penguins as ten thousand.)
(3) We completely abandon the principle of using minimum description length as some kind of ‘universal prior’. (For some reason. And replace it with something else. For some reason.)
I think this scenario is ingenious. Here are a few ideas, but I’m really not sure how far one can pursue them / how ‘much work’ they can do:
(1) Perhaps the agent needs some way of ‘absolving itself of responsibility’ for the evil/arbitrary/unreasonable actions of another being. The action to be performed is the one that yields highest expected utility but only along causal pathways that don’t go through an adversary that has been labelled as ‘unreasonable’.
(Except this approach doesn’t defuse the variation that goes “You can never wipe your nose because you’ve computed that the probability of this action killing 3^^^^3 people in a parallel universe is ever so slightly greater than the probability of it saving that number of people”.)
(2) We only have a fixed amount of ‘moral concern’, apportioned somehow or other to the beings we care about. Our utility function looks like: Sum(over beings X) ConcernFor(X)*HappinessOf(X). Allocation of ‘moral concern’ is a ‘competitive’ process. The only way we can gain some concern about Y is to lose a bit of concern about some X, but if we have regular and in some sense ‘positive’ interactions with X then our concern for X will be constantly ‘replenishing itself’. When the magician appears and tells us his story, we may acquire a tiny bit of concern about him and the people he mentions, but the parts of us that care about the people we know (a) aren’t ‘told’ the magician’s story and thus (b) refuse to ‘relinquish’ very much.
The trouble with is that it sounds too reminiscent of the insanely stupid moral behaviour of human beings (where e.g. they give exactly as much money to save a hundred penguins as ten thousand.)
(3) We completely abandon the principle of using minimum description length as some kind of ‘universal prior’. (For some reason. And replace it with something else. For some reason.)