Omega is supposed to be always truthful, so either he rewards the sims as well, or you know something the sims don’t and hence it’s not obvious you’ll do the same as them.
Even if he’s not, after he’s given a $1m simulated reward, does he then have to keep up a simulated environment for the sim to actually spend the money?
If he can lie to sims, then you can’t know he’s not lying to you unless you know you’re not a sim. If you do, it’s not obvious you’d choose the same way as if you didn’t.
Well, as long as you believe Omega enough to think no box contains sudden death or otherwise negative utility, you’d open them to see what was inside. But yes, you might not believe Omega at all.
General question: suppose we encounter an alien. We have no idea what its motivations, values, goals, or abilities are. On the other hand, if may have observed any amount of human comm traffic from wireless EM signals since the invention of radio, and from actual spy-probes before the human invention of high tech that would detect them.
It signals us in Morse code from its remote starship, offering mutually benefitial trade.
What prior should we have about the alien’s intention? Should we use a native uniform prior that would tell us it’s as likely to mean us good as harm, and so never reply because we don’t know how it will try to influence our actions via communications? Should it tell us different agents who don’t explicitly value one another will conflict to the extent their values differ, and so since value-space is vast and a randomly selected alien is unlikely to share many values with us, we should prepare for war? Should it tell us we can make some assumptions (which?) about naturally evolved agents or their Friendly-to-themselves creations? How safe are we if we try to “just read” English text written by an unknown, possibly-superintelligence which may have observed all our broadcast traffic since the age of radio? What does our non-detection of this alien civ until they chose to initiate contact tell us? Etc.
A 50% chance of meaning us good vs harm isn’t a prior I find terribly compelling.
There’s a lot to say here, but my short answer is that this is both an incredibly dangerous and incredibly valuable situation, in which both the potential opportunity costs and the potential actual costs are literally astronomical, and in which there are very few things I can legitimately be confident of.
The best I can do in such a situation is to accept that my best guess is overwhelmingly likely to be wrong, but that it’s slightly less likely to be wrong than my second-best guess, so I should operate on the basis of my best guess despite expecting it to be wrong. Where “best guess” here is the thing I consider most likely to be true, not the thing with the highest expected value.
I should also note that my priors about aliens in general—that is, what I consider likely about a randomly selected alien intelligence—are less relevant to this scenario than what I consider likely about this particular intelligence, given that it has observed us for long enough to learn our language, revealed itself to us, communicated with us in Morse code, offered mutually beneficial trade, etc.
The most tempting belief for me is that the alien’s intentions are essentially similar to ours. I can even construct a plausible sounding argument for that as my best guess… we’re the only other species I know capable of communicating the desire for mutually beneficial trade in an artificial signalling system, so our behavior constitutes strong evidence for their behavior. OTOH, it’s pretty clear to me that the reason I’m tempted to believe that is because I can do something with that belief; it gives me a lot of traction for thinking about what to do next. (In a nutshell, I would conclude from that assumption that it means to exploit us for its long-term benefit, and whether that’s good or bad for us depends entirely on what our most valuable-to-it resources are and how it can most easily obtain them and whether we benefit from that process.) Since that has almost nothing to do with the likelihood of it being true, I should distrust my desire to believe that.
Ultimately, I think what I do is reply that I value mutually beneficial trade with them, but that I don’t actually trust them and must therefore treat them as a potential threat until I have gathered more information about them, while at the same time refraining from doing anything that would significantly reduce our chances of engaging in mutually beneficial trade in the future, and what do they think about all that?
He can certainly give them counterfactual ‘realities’. It would seem that he should be assumed to at least provide counterfactual realities wherein information provided by the simulation’s representation of Omega indicates that he is perfectly trustworthy.
Even if he’s not, after he’s given a $1m simulated reward, does he then have to keep up a simulated environment for the sim to actually spend the money?
No. But if for whatever reason the simulated environment persists it should be one that is consistent with Omega keeping his word. Or, if part of the specification of the problem or the declarations made by Omega directly pertain to claims about what He will do regarding simulation then he will implement that policy.
Omega is supposed to be always truthful, so either he rewards the sims as well, or you know something the sims don’t and hence it’s not obvious you’ll do the same as them.
I thought Omega was allowed to lie to sims.
Even if he’s not, after he’s given a $1m simulated reward, does he then have to keep up a simulated environment for the sim to actually spend the money?
If he can lie to sims, then you can’t know he’s not lying to you unless you know you’re not a sim. If you do, it’s not obvious you’d choose the same way as if you didn’t.
For instance, if you think Omega is lying and completely ignore everything he says, you obviously two-box.
Why not zero-box in this case? I mean, what reason would I have to expect any money at all?
Well, as long as you believe Omega enough to think no box contains sudden death or otherwise negative utility, you’d open them to see what was inside. But yes, you might not believe Omega at all.
General question: suppose we encounter an alien. We have no idea what its motivations, values, goals, or abilities are. On the other hand, if may have observed any amount of human comm traffic from wireless EM signals since the invention of radio, and from actual spy-probes before the human invention of high tech that would detect them.
It signals us in Morse code from its remote starship, offering mutually benefitial trade.
What prior should we have about the alien’s intention? Should we use a native uniform prior that would tell us it’s as likely to mean us good as harm, and so never reply because we don’t know how it will try to influence our actions via communications? Should it tell us different agents who don’t explicitly value one another will conflict to the extent their values differ, and so since value-space is vast and a randomly selected alien is unlikely to share many values with us, we should prepare for war? Should it tell us we can make some assumptions (which?) about naturally evolved agents or their Friendly-to-themselves creations? How safe are we if we try to “just read” English text written by an unknown, possibly-superintelligence which may have observed all our broadcast traffic since the age of radio? What does our non-detection of this alien civ until they chose to initiate contact tell us? Etc.
A 50% chance of meaning us good vs harm isn’t a prior I find terribly compelling.
There’s a lot to say here, but my short answer is that this is both an incredibly dangerous and incredibly valuable situation, in which both the potential opportunity costs and the potential actual costs are literally astronomical, and in which there are very few things I can legitimately be confident of.
The best I can do in such a situation is to accept that my best guess is overwhelmingly likely to be wrong, but that it’s slightly less likely to be wrong than my second-best guess, so I should operate on the basis of my best guess despite expecting it to be wrong. Where “best guess” here is the thing I consider most likely to be true, not the thing with the highest expected value.
I should also note that my priors about aliens in general—that is, what I consider likely about a randomly selected alien intelligence—are less relevant to this scenario than what I consider likely about this particular intelligence, given that it has observed us for long enough to learn our language, revealed itself to us, communicated with us in Morse code, offered mutually beneficial trade, etc.
The most tempting belief for me is that the alien’s intentions are essentially similar to ours. I can even construct a plausible sounding argument for that as my best guess… we’re the only other species I know capable of communicating the desire for mutually beneficial trade in an artificial signalling system, so our behavior constitutes strong evidence for their behavior. OTOH, it’s pretty clear to me that the reason I’m tempted to believe that is because I can do something with that belief; it gives me a lot of traction for thinking about what to do next. (In a nutshell, I would conclude from that assumption that it means to exploit us for its long-term benefit, and whether that’s good or bad for us depends entirely on what our most valuable-to-it resources are and how it can most easily obtain them and whether we benefit from that process.) Since that has almost nothing to do with the likelihood of it being true, I should distrust my desire to believe that.
Ultimately, I think what I do is reply that I value mutually beneficial trade with them, but that I don’t actually trust them and must therefore treat them as a potential threat until I have gathered more information about them, while at the same time refraining from doing anything that would significantly reduce our chances of engaging in mutually beneficial trade in the future, and what do they think about all that?
He can certainly give them counterfactual ‘realities’. It would seem that he should be assumed to at least provide counterfactual realities wherein information provided by the simulation’s representation of Omega indicates that he is perfectly trustworthy.
No. But if for whatever reason the simulated environment persists it should be one that is consistent with Omega keeping his word. Or, if part of the specification of the problem or the declarations made by Omega directly pertain to claims about what He will do regarding simulation then he will implement that policy.