I argue that right now, sarting from the present state, the true quantum probability of achieving the Glorious Future is way higher than 2^-75, or if not, then we should probably work on something other than AI safety. Me and Ryan argue for this in the last few comments. It’s not a terribly important point, you can just say the true quantum probability is 1 in a billion, when it’s still worth it for you to work on the problem, but it becomes rough to trade for keeping humanity physically alive that can cause one year of delay to the AI.
But I would like you to acknowledge that “vastly below 2^-75 true quantum probability, as starting from now” is probably mistaken, or explain why our logic is wrong about how this implies you should work on malaria.
Starting from now? I agree that that’s true in some worlds that I consider plausible, at least, and I agree that worlds whose survival-probabilities are sensitive to my choices are the ones that render my choices meaningful (regardless of how determinisic they are).
Conditional on Earth being utterly doomed, are we (today) fewer than 75 qbitflips from being in a good state? I’m not sure, it probably varies across the doomed worlds where I have decent amounts of subjective probability. It depends how much time we have on the clock, depends where the points of no-return are. I haven’t thought about this a ton. My best guess is it would take more than 75 qbitflips to save us now, but maybe I’m not thinking creatively enough about how to spend them, and I haven’t thought about it in detail and expect I’d be sensitive to argument about it /shrug.
(If you start from 50 years ago? Very likely! 75 bits is a lot of population rerolls. If you start after people hear the thunder of the self-replicating factories barrelling towards them, and wait until the very last moments that they would consider becoming a distinct person who is about to die from AI, and who wishes to draw upon your reassurance that they will be saved? Very likely not! Those people look very, very dead.)
One possible point of miscommunication is that when I said something like “obviously it’s worse than 2^-75 at the extreme where it’s actually them who is supposed to survive” was intended to apply to the sort of person who has seen the skies darken and has heard the thunder, rather than the version of them that exists here in 2024. This was not intended to be some bold or suprising claim. It was an attempt to establish an obvious basepoint at one very extreme end of a spectrum, that we could start interpolating from (asking questions like “how far back from there are the points of no return?” and “how much more entropy would they have than god, if people from that branchpoint spent stars trying to figure out what happened after those points?”).
(The 2^-75 was not intended to be even an esitmate of how dead the people on the one end of the extreme are. It is the “can you buy a star” threshold. I was trying to say something like “the individuals who actually die obviously can’t buy themselves a star just because they inhabit Tegmark III, now let’s drag the cursor backwards and talk about whether, at any point, we cross the a-star-for-everyone threshold”.)
If that doesn’t clear things up and you really want to argue that, conditional on Earth being as doomed as it superficially looks to me, most of those worlds are obviously <100 quantum bitflips from victory today, I’m willing to field those arguments; maybe you see some clever use of qbitflips I don’t and that would be kinda cool. But I caveat that this doesn’t seem like a crux to me and that I acknowledge that the other worlds (where Earth merely looks unsavlageable) are the ones motivating action.
I have not followed this thread in all of its detail, but it sounds like it might be getting caught up on the difference between the underlying ratio of different quantum worlds (which can be expressed as a probability over one’s future) and one’s probabilistic uncertainty over the underlying ratio of different quantum worlds (which can also be expressed as a probability over the future but does not seem to me to have the same implications for behavior).
Insofar as it seems to readers like a bad idea to optimize for different outcomes in a deterministic universe, I recommend reading the Free Will (Solution) sequence by Eliezer Yudkowsky, which I found fairly convincing on the matter of why it’s still right to optimize in a fully deterministic universe, as well as in a universe running on quantum mechanics (interpreted to have many worlds).
I argue that right now, sarting from the present state, the true quantum probability of achieving the Glorious Future is way higher than 2^-75, or if not, then we should probably work on something other than AI safety. Me and Ryan argue for this in the last few comments. It’s not a terribly important point, you can just say the true quantum probability is 1 in a billion, when it’s still worth it for you to work on the problem, but it becomes rough to trade for keeping humanity physically alive that can cause one year of delay to the AI.
But I would like you to acknowledge that “vastly below 2^-75 true quantum probability, as starting from now” is probably mistaken, or explain why our logic is wrong about how this implies you should work on malaria.
Starting from now? I agree that that’s true in some worlds that I consider plausible, at least, and I agree that worlds whose survival-probabilities are sensitive to my choices are the ones that render my choices meaningful (regardless of how determinisic they are).
Conditional on Earth being utterly doomed, are we (today) fewer than 75 qbitflips from being in a good state? I’m not sure, it probably varies across the doomed worlds where I have decent amounts of subjective probability. It depends how much time we have on the clock, depends where the points of no-return are. I haven’t thought about this a ton. My best guess is it would take more than 75 qbitflips to save us now, but maybe I’m not thinking creatively enough about how to spend them, and I haven’t thought about it in detail and expect I’d be sensitive to argument about it /shrug.
(If you start from 50 years ago? Very likely! 75 bits is a lot of population rerolls. If you start after people hear the thunder of the self-replicating factories barrelling towards them, and wait until the very last moments that they would consider becoming a distinct person who is about to die from AI, and who wishes to draw upon your reassurance that they will be saved? Very likely not! Those people look very, very dead.)
One possible point of miscommunication is that when I said something like “obviously it’s worse than 2^-75 at the extreme where it’s actually them who is supposed to survive” was intended to apply to the sort of person who has seen the skies darken and has heard the thunder, rather than the version of them that exists here in 2024. This was not intended to be some bold or suprising claim. It was an attempt to establish an obvious basepoint at one very extreme end of a spectrum, that we could start interpolating from (asking questions like “how far back from there are the points of no return?” and “how much more entropy would they have than god, if people from that branchpoint spent stars trying to figure out what happened after those points?”).
(The 2^-75 was not intended to be even an esitmate of how dead the people on the one end of the extreme are. It is the “can you buy a star” threshold. I was trying to say something like “the individuals who actually die obviously can’t buy themselves a star just because they inhabit Tegmark III, now let’s drag the cursor backwards and talk about whether, at any point, we cross the a-star-for-everyone threshold”.)
If that doesn’t clear things up and you really want to argue that, conditional on Earth being as doomed as it superficially looks to me, most of those worlds are obviously <100 quantum bitflips from victory today, I’m willing to field those arguments; maybe you see some clever use of qbitflips I don’t and that would be kinda cool. But I caveat that this doesn’t seem like a crux to me and that I acknowledge that the other worlds (where Earth merely looks unsavlageable) are the ones motivating action.
I have not followed this thread in all of its detail, but it sounds like it might be getting caught up on the difference between the underlying ratio of different quantum worlds (which can be expressed as a probability over one’s future) and one’s probabilistic uncertainty over the underlying ratio of different quantum worlds (which can also be expressed as a probability over the future but does not seem to me to have the same implications for behavior).
Insofar as it seems to readers like a bad idea to optimize for different outcomes in a deterministic universe, I recommend reading the Free Will (Solution) sequence by Eliezer Yudkowsky, which I found fairly convincing on the matter of why it’s still right to optimize in a fully deterministic universe, as well as in a universe running on quantum mechanics (interpreted to have many worlds).