Going back to the envelopes example, a nosy neighbor hypothesis would be “the left envelope contains $100, even in the world where the right envelope contains $100”. Or if we have an AI that’s unsure whether it values paperclips or staples, a nosy neighbor hypothesis would be “I value paperclips, even in the world where I value staples”. I’m not sure how that makes sense. Can you give some scenario where a nosy neighbor hypothesis makes sense?
I think so, yes, but I want to note that my position is consistent with nosy-neighbor hypotheses not making sense. A big part of my point is that there’s a lot of nonsense in a broad prior. I think it’s hard to rule out the nonsense without learning. If someone thought nosy neighbors always ‘make sense’, it could be an argument against my whole position. (Because that person might be just fine with UDT, thinking that my nosy-neighbor ‘problems’ are just counterfactual muggings.)
Here’s an argument that nosy neighbors can make sense.
For values, as I mentioned, a nosy-neighbors hypothesis is a value system which cares about what happens in many different universes, not just the ‘actual’ universe. For example, a utility function which assigns some value to statements of mathematics.
For probability, a nosy-neighbor is like the Lizard World hypothesis mentioned in the post: it’s a world where what happens there depends a lot on what happens in other worlds.
I think what you wrote about staples vs paperclips nosy-neighbors is basically right, but maybe if we rephrase it it can ‘make more sense’?: “I (actual me) value paperclips being produced in the counterfactual(-from-my-perspective) world where I (counterfactual me) don’t value paperclips.”
Anyway, whether or not it makes intuitive sense, it’s mathematically fine. The idea is that a world will contain facts that are a good lens into alternative worlds (such as facts of Peano Arithmetic), which utility hypotheses / probabilistic hypotheses can care about. So although a hypothesis is only mathematically defined as a function of worlds where it holds, it “sneakily” depends on stuff that goes on in other worlds as well.
Going back to the envelopes example, a nosy neighbor hypothesis would be “the left envelope contains $100, even in the world where the right envelope contains $100”. Or if we have an AI that’s unsure whether it values paperclips or staples, a nosy neighbor hypothesis would be “I value paperclips, even in the world where I value staples”. I’m not sure how that makes sense. Can you give some scenario where a nosy neighbor hypothesis makes sense?
I think so, yes, but I want to note that my position is consistent with nosy-neighbor hypotheses not making sense. A big part of my point is that there’s a lot of nonsense in a broad prior. I think it’s hard to rule out the nonsense without learning. If someone thought nosy neighbors always ‘make sense’, it could be an argument against my whole position. (Because that person might be just fine with UDT, thinking that my nosy-neighbor ‘problems’ are just counterfactual muggings.)
Here’s an argument that nosy neighbors can make sense.
For values, as I mentioned, a nosy-neighbors hypothesis is a value system which cares about what happens in many different universes, not just the ‘actual’ universe. For example, a utility function which assigns some value to statements of mathematics.
For probability, a nosy-neighbor is like the Lizard World hypothesis mentioned in the post: it’s a world where what happens there depends a lot on what happens in other worlds.
I think what you wrote about staples vs paperclips nosy-neighbors is basically right, but maybe if we rephrase it it can ‘make more sense’?: “I (actual me) value paperclips being produced in the counterfactual(-from-my-perspective) world where I (counterfactual me) don’t value paperclips.”
Anyway, whether or not it makes intuitive sense, it’s mathematically fine. The idea is that a world will contain facts that are a good lens into alternative worlds (such as facts of Peano Arithmetic), which utility hypotheses / probabilistic hypotheses can care about. So although a hypothesis is only mathematically defined as a function of worlds where it holds, it “sneakily” depends on stuff that goes on in other worlds as well.