Now suppose that the said “Matrix Lord” opens the sky, splits the Red Sea, demonstrates his duplicator box on some fish and, sure, creates a humanoid Patronus. Now do I have more reason to believe that he is a Time Lord? Perhaps. Do I have reason to think that he will save 3^^^3 lives if I give him $5? I don’t see convincing reason to believe so, but I don’t see either view as problematic.
Honestly, at this point, I would strongly update in the direction that I am being deceived in some manner. Possibly I am dreaming, or drugged, of the person in front of me has some sort of perception-control device. I do not see any reason why someone who could open the sky, split the Red Sea, and so on, would need $5; and if he did, why not make it himself? Or sell the fish?
The only reasons I can imagine for a genuine Matrix Lord pulling this on me are very bad for me. Either he’s a sadist who likes people to suffer—in which case I’m doomed no matter what I do—or there’s something that he’s not telling me (perhaps doing what he says once surrenders my free will, allowing him to control me forever?), which implies that he believes that I would reject his demand if I knew the truth behind it, which strongly prompts me to reject his demand.
Or he’s insane, following no discernable rules, in which case the only thing to do is to try to evade notice (something I’ve clearly already failed at).
Either he’s a sadist who likes people to suffer—in which case I’m doomed no matter what I do—or there’s something that he’s not telling me (perhaps doing what he says once surrenders my free will, allowing him to control me forever?), which implies that he believes that I would reject his demand if I knew the truth behind it, which strongly prompts me to reject his demand.
That your universe is controlled by a sadist doesn’t suggest that every possible action you could do is equivalent. Maybe all your possible fates are miserable, but some are far more miserable than others. More importantly, a being might be sadistic in some respects/situations but not in others.
I also have to assign a very, very low prior to anyone’s being able to figure out in 5 minutes what the Matrix Lord’s exact motivations are. Your options are too simplistic even to describe minds of human-level complexity, much less ones of the complexity required to design or oversee physics-breakingly large simulations.
I think indifference to our preferences (except as incidental to some other goal, e.g., paperclipping) is more likely than either sadism or beneficence. Only very small portions of the space of values focus on human-style suffering or joy. Even in hypotheticals that seem designed to play with human moral intuitions. Eliezer’s decision theory conference explanation makes as much sense as any.
That your universe is controlled by a sadist doesn’t suggest that every possible action you could do is equivalent. Maybe all your possible fates are miserable, but some are far more miserable than others.
You are right. However, I can see no way to decide which course of action is best (or least miserable). My own decision process becomes questionable in such a situation; I can’t imagine any strategy that is convincingly better than taking random actions.
When I say “doomed no matter what I do”, I do not mean doomed with certainty. I mean that I have a high probability of doom, for any given action, and I cannot find a way to minimise that probability through my own actions.
I think indifference to our preferences (except as incidental to some other goal, e.g., paperclipping) is more likely than either sadism or beneficence.
Thinking about this, I think that you are right. I still consider sadism more likely than beneficence, but I had been setting the prior for indifference too low. This implies that the Matrix Lord has preferences, but these preferences are unknown and possibly unknowable (perhaps he wants to maximise slood).
...
This make the question of which action to best take even more difficult to answer. I do not know anything about slood; I cannot, because it only exists outside the Matrix. The only source of information from outside the Matrix is the Matrix Lord. This implies that, before reaching any decision, I should spend a long time interviewing the Matrix Lord, in an attempt to better be able to model him.
However, I can see no way to decide which course of action is best (or least miserable). My own decision process becomes questionable in such a situation; I can’t imagine any strategy that is convincingly better than taking random actions.
Well, this Matrix Lord seems very interested in decision theory and utilitarianism. Sadistic or not, I expect such a being to respond more favorably to attempts to take the dilemmas he raised seriously than to an epistemic meltdown. Taking the guy at his word and trying to reason your way through the problem is likely to give him more useful data than attempts to rebel or go crazy, and if you’re useful then it’s less likely that he’ll punish you or pull the plug on your universe’s simulation.
It seems reasonably likely that this will lead to a response of ”...alright, I’ve got the data that I wanted, no need to keep this simulation running any longer...” and then pulling the plug on my universe. While it is true that this strategy is likely to lead to a happier Matrix Lord (especially if the data that I give him coincides with the data he expects), I’m not convinced that it leads to a longer existence for my universe.
That may be true too. It depends on the priors we have for generic superhuman agents’ reasons for keeping a simulation running (e.g., having some other science experiments planned, wanting to reward you for providing data...) vs. for shutting it down (e.g., vindictiveness, energy conservation, being interested only in one data point per simulation...).
We do have some data to work with here, since we have experience with the differential effects of power, intelligence, curiosity, etc. among humans. That data is only weakly applicable to such an exotic agent, but it does play a role, so our uncertainty isn’t absolute. My main point was that unusual situations like this don’t call for complete decision-theoretic despair; we still need to make choices, and we can still do so reasonably, though our confidence that the best decision is also a winning decision is greatly diminished.
Well, if I’m going to free-form speculate about the scenario, rather than use it to explore the question it was introduced to explore, the most likely explanation that occurs to me is that the entity is doing the Matrix Lord equivalent of free-form speculating… that is, it’s wondering “what would humans do, given this choice and that information?” And, it being a Matrix Lord, its act of wondering creates a human mind (in this case, mine) and gives it that choice and information.
Which makes it likely that I haven’t actually lived through most of the life I remember, and that I won’t continue to exist much longer than this interaction, and that most of what I think is in the world around me doesn’t actually exist.
That said, I’m not sure what use free-form speculating about such bizarre and underspecified scenarios really is, though I’ll admit it’s kind of fun.
That said, I’m not sure what use free-form speculating about such bizarre and underspecified scenarios really is, though I’ll admit it’s kind of fun.
It’s kind of fun. Isn’t that reason enough?
Looking at the original question—i.e. how to handle very large utilities with very small probability—I find that I have a mental safety net there. The safety net says that the situation is a lie. It does not matter how much utility is claimed, because anyone can state any arbitrarily large number, and a number has been chosen (in this case, by the Matrix Lord) in a specific attempt to overwhelm my utility function. The small probability is chosen (a) because I would not believe a larger probability and (b) so that I have no recourse when it fails to happen.
I am reluctant to fiddle with my mental safety nets because, well, they’re safety nets—they’re there for a reason. And in this case, the reason is that such a fantastically unlikely event is unlikely enough that it’s not likely to happen ever, to anyone. Not even once in the whole history of the universe. If I (out of all the hundreds of billions of people in all of history) do ever run across such a situation, then it’s so incredibly overwhelmingly more likely that I am being deceived that I’m far more likely to gain by immediately jumping to the conclusion of ‘deceit’ than by assuming that there’s any chance of this being true.
Honestly, at this point, I would strongly update in the direction that I am being deceived in some manner. Possibly I am dreaming, or drugged, of the person in front of me has some sort of perception-control device. I do not see any reason why someone who could open the sky, split the Red Sea, and so on, would need $5; and if he did, why not make it himself? Or sell the fish?
The only reasons I can imagine for a genuine Matrix Lord pulling this on me are very bad for me. Either he’s a sadist who likes people to suffer—in which case I’m doomed no matter what I do—or there’s something that he’s not telling me (perhaps doing what he says once surrenders my free will, allowing him to control me forever?), which implies that he believes that I would reject his demand if I knew the truth behind it, which strongly prompts me to reject his demand.
Or he’s insane, following no discernable rules, in which case the only thing to do is to try to evade notice (something I’ve clearly already failed at).
That your universe is controlled by a sadist doesn’t suggest that every possible action you could do is equivalent. Maybe all your possible fates are miserable, but some are far more miserable than others. More importantly, a being might be sadistic in some respects/situations but not in others.
I also have to assign a very, very low prior to anyone’s being able to figure out in 5 minutes what the Matrix Lord’s exact motivations are. Your options are too simplistic even to describe minds of human-level complexity, much less ones of the complexity required to design or oversee physics-breakingly large simulations.
I think indifference to our preferences (except as incidental to some other goal, e.g., paperclipping) is more likely than either sadism or beneficence. Only very small portions of the space of values focus on human-style suffering or joy. Even in hypotheticals that seem designed to play with human moral intuitions. Eliezer’s decision theory conference explanation makes as much sense as any.
You are right. However, I can see no way to decide which course of action is best (or least miserable). My own decision process becomes questionable in such a situation; I can’t imagine any strategy that is convincingly better than taking random actions.
When I say “doomed no matter what I do”, I do not mean doomed with certainty. I mean that I have a high probability of doom, for any given action, and I cannot find a way to minimise that probability through my own actions.
Thinking about this, I think that you are right. I still consider sadism more likely than beneficence, but I had been setting the prior for indifference too low. This implies that the Matrix Lord has preferences, but these preferences are unknown and possibly unknowable (perhaps he wants to maximise slood).
...
This make the question of which action to best take even more difficult to answer. I do not know anything about slood; I cannot, because it only exists outside the Matrix. The only source of information from outside the Matrix is the Matrix Lord. This implies that, before reaching any decision, I should spend a long time interviewing the Matrix Lord, in an attempt to better be able to model him.
Well, this Matrix Lord seems very interested in decision theory and utilitarianism. Sadistic or not, I expect such a being to respond more favorably to attempts to take the dilemmas he raised seriously than to an epistemic meltdown. Taking the guy at his word and trying to reason your way through the problem is likely to give him more useful data than attempts to rebel or go crazy, and if you’re useful then it’s less likely that he’ll punish you or pull the plug on your universe’s simulation.
It seems reasonably likely that this will lead to a response of ”...alright, I’ve got the data that I wanted, no need to keep this simulation running any longer...” and then pulling the plug on my universe. While it is true that this strategy is likely to lead to a happier Matrix Lord (especially if the data that I give him coincides with the data he expects), I’m not convinced that it leads to a longer existence for my universe.
That may be true too. It depends on the priors we have for generic superhuman agents’ reasons for keeping a simulation running (e.g., having some other science experiments planned, wanting to reward you for providing data...) vs. for shutting it down (e.g., vindictiveness, energy conservation, being interested only in one data point per simulation...).
We do have some data to work with here, since we have experience with the differential effects of power, intelligence, curiosity, etc. among humans. That data is only weakly applicable to such an exotic agent, but it does play a role, so our uncertainty isn’t absolute. My main point was that unusual situations like this don’t call for complete decision-theoretic despair; we still need to make choices, and we can still do so reasonably, though our confidence that the best decision is also a winning decision is greatly diminished.
Well, if I’m going to free-form speculate about the scenario, rather than use it to explore the question it was introduced to explore, the most likely explanation that occurs to me is that the entity is doing the Matrix Lord equivalent of free-form speculating… that is, it’s wondering “what would humans do, given this choice and that information?” And, it being a Matrix Lord, its act of wondering creates a human mind (in this case, mine) and gives it that choice and information.
Which makes it likely that I haven’t actually lived through most of the life I remember, and that I won’t continue to exist much longer than this interaction, and that most of what I think is in the world around me doesn’t actually exist.
That said, I’m not sure what use free-form speculating about such bizarre and underspecified scenarios really is, though I’ll admit it’s kind of fun.
It’s kind of fun. Isn’t that reason enough?
Looking at the original question—i.e. how to handle very large utilities with very small probability—I find that I have a mental safety net there. The safety net says that the situation is a lie. It does not matter how much utility is claimed, because anyone can state any arbitrarily large number, and a number has been chosen (in this case, by the Matrix Lord) in a specific attempt to overwhelm my utility function. The small probability is chosen (a) because I would not believe a larger probability and (b) so that I have no recourse when it fails to happen.
I am reluctant to fiddle with my mental safety nets because, well, they’re safety nets—they’re there for a reason. And in this case, the reason is that such a fantastically unlikely event is unlikely enough that it’s not likely to happen ever, to anyone. Not even once in the whole history of the universe. If I (out of all the hundreds of billions of people in all of history) do ever run across such a situation, then it’s so incredibly overwhelmingly more likely that I am being deceived that I’m far more likely to gain by immediately jumping to the conclusion of ‘deceit’ than by assuming that there’s any chance of this being true.
(nods) Sure. My reply here applies here as well.