Suppose that I had a plan that will wipe out humanity with probability 0.999999999, but will lead to a positive Singularity with probability 0.000000001 minus epsilon. I understand why that would be a terrible plan
Hm. Okay. Maybe you could explain it to me then. Why is it a terrible plan, even though all the future people who count will think it was great? None of these futures actually exist to choose between, you’re just planning based on a model of the world. Why not just plan so that the people who matter in your model will have a good time? “When I am, Death is Not, and when Death is, I am Not,” so why should we weight it negatively?
(I’d actually be interested in how you’d answer this).
My favorite answer: It’s just like how you shouldn’t do meth. “Meth” here refers to any drug that rewires your utility function so that you feel really great, right until you drop dead, though you don’t care about that part as much when you’re on meth. If you just want to make your future self happy by their own standards, you should do meth—it feels great once you do it!
Similarly, in the case of either quantum or classical russian roulette, you may be rich if you win, but “zooming in” on only the person who wins is a type of trying to make this model person happy by their own standards. This is a generally flawed decision procedure. You should be able to choose based on your own standards, not the standards of the people who live / are on meth.
Hm. Okay. Maybe you could explain it to me then. Why is it a terrible plan, even though all the future people who count will think it was great? None of these futures actually exist to choose between, you’re just planning based on a model of the world. Why not just plan so that the people who matter in your model will have a good time? “When I am, Death is Not, and when Death is, I am Not,” so why should we weight it negatively?
(I’d actually be interested in how you’d answer this).
My favorite answer: It’s just like how you shouldn’t do meth. “Meth” here refers to any drug that rewires your utility function so that you feel really great, right until you drop dead, though you don’t care about that part as much when you’re on meth. If you just want to make your future self happy by their own standards, you should do meth—it feels great once you do it!
Similarly, in the case of either quantum or classical russian roulette, you may be rich if you win, but “zooming in” on only the person who wins is a type of trying to make this model person happy by their own standards. This is a generally flawed decision procedure. You should be able to choose based on your own standards, not the standards of the people who live / are on meth.