Say you’re playing Russian Roulette with a 6-round revolver which either has 1 or 0 live rounds in it. Pull the trigger 4 times—every time you end up still alive. According to what you have said, your probability estimates for either
there being a single round in the revolver or
the revolver being unloaded
should be the same as before you had played any rounds at all.
If before every time I pull the trigger, I spin the revolver in such a way that it comes to a stop in a position that is completely uncorrelated with its pre-spin position, then yes, IMO the probability is the same as before I had played any rounds at all (namely .5).
If an evil demon were to adjust the revolver after I spin it and before I pull the trigger, that is a selection effect. If the demon’s adjustments are skillful enough and made for the purpose of deceiving me, my trigger pulls are no longer a random sample from the space of possible outcomes.
Probability is not a property of reality but rather a property of an observer. If a particular observer is not robust enough to survive a particular experiment, the observer will not be able to learn from the experiment the same way a more robust observer can. As I play Russian roulette, the P(gun has bullet) assigned by someone watching me at a safe distance can change, but my P(gun has bullet) cannot change because of the law of conservation of expected evidence.
In particular, a trigger pull that does not result in a bang does not decrease my probability that the gun contains a bullet because a trigger pull that results in a bang does not increase it (because I do not survive a trigger pull that results in a bang).
In particular, a trigger pull that does not result in a bang does not decrease my probability that the gun contains a bullet because a trigger pull that results in a bang does not increase it (because I do not survive a trigger pull that results in a bang).
I’m not sure this would work in practice. Let’s say you’re betting on this particular game, with the winnings/losings being useful in some way even if you don’t survive the game. Then, after spinning and pulling the trigger a million times, would you still bet as though the odds were 1:1? I’m pretty sure that’s not a winning strategy, when viewed from the outside (therefore, still not winning when viewed from the inside).
You have persuaded me that my analysis in grandparent of the Russian-roulette scenario is probably incorrect.
The scenario of the black box that responds with either “heads” or “tails” is different because in the Russian-roulette scenario, we have a partial causal model of the “bang”/”no bang” event. (In particular, we know that the revolver contains either one bullet or zero bullets.) Apparently, causal knowledge can interact with knowledge of past behavior to produce knowledge of future behavior even if the knowledge of past behavior is subject to the strongest kind of observational selection efffects.
Your last point was persuasive… though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing.
Would it be within the first AGI’s capabilities to immediately effect my destruction before I am able to update on its existence—provided that (a) it is developed by the private sector and not e.g. some special access DoD program, and (b) ETAs up to “sometime this century” are accurate? I think not, though I admit to being fairly uncertain.
I acknowledge that this line of reasoning presented in my original comment was not of high caliber—though I still dispute Tiiba’s claim regarding an AI advanced enough to scrape by in conversation with a 5 year old, as well as that distributive capabilities are the greatest power at play here.
Would it be within the first AGI’s capabilities to immediately effect my destruction before I am able to update on its existence . . .?
I humbly suggest that the answer to your question would not shed any particular light on what we have been talking about because even if we would certainly have noticed the birth of the AGI, there’s a selection effect if it would have killed us before we got around to having this conversation (i.e. if it would have killed us by now).
The AGI’s causing our deaths is not the only thing that would cause a selection effect: the AGI’s deleting our memories of the existence of the AGI would also do it. But the AGI’s causing our deaths is the mostly likely selection-effecting mechanism.
A nice summary of my position is that when we try to estimate the safety of AGI research done in the past, the fact that P(we would have noticed our doom by now|the research killed us or will kill us) is high does not support the safety of the research as much as one might naively think. For us to use that fact the way we use most facts, not only must we notice our doom, but also we must survive long enough to have this conversation.
Actually, we can generalize that last sentence: for a group of people correctly to use the outcome of past AGI research to help assess the safety of AGI, awareness of both possible outcomes (the good outcome and the bad outcome) of the past research must be able to reach the group and in particular must be able to reach the assessment process. More precisely, if there is a mechanism that is more likely to prevent awareness of one outcome from reaching the assessment process than the other outcome, the process has to adjust for that, and if the very existence of the assessment process completely depends on one outcome, the adjustment completely wipes out the “evidentiary value” of awareness of the outcome. The likelihood ratio gets adjusted to 1. The posterior probability (i.e., the probability after updating on the outcome of the research) that AGI is safe is the same as the prior probability.
Your last point was persuasive… though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing.
Like I said yesterday I retract my position on the Russian roulette. (Selection effects operate, I still believe, but not to the extent of making past behavior completely useless for predicting future behavior.)
If before every time I pull the trigger, I spin the revolver in such a way that it comes to a stop in a position that is completely uncorrelated with its pre-spin position, then yes, IMO the probability is the same as before I had played any rounds at all (namely .5).
If an evil demon were to adjust the revolver after I spin it and before I pull the trigger, that is a selection effect. If the demon’s adjustments are skillful enough and made for the purpose of deceiving me, my trigger pulls are no longer a random sample from the space of possible outcomes.
Probability is not a property of reality but rather a property of an observer. If a particular observer is not robust enough to survive a particular experiment, the observer will not be able to learn from the experiment the same way a more robust observer can. As I play Russian roulette, the P(gun has bullet) assigned by someone watching me at a safe distance can change, but my P(gun has bullet) cannot change because of the law of conservation of expected evidence.
In particular, a trigger pull that does not result in a bang does not decrease my probability that the gun contains a bullet because a trigger pull that results in a bang does not increase it (because I do not survive a trigger pull that results in a bang).
I’m not sure this would work in practice. Let’s say you’re betting on this particular game, with the winnings/losings being useful in some way even if you don’t survive the game. Then, after spinning and pulling the trigger a million times, would you still bet as though the odds were 1:1? I’m pretty sure that’s not a winning strategy, when viewed from the outside (therefore, still not winning when viewed from the inside).
You have persuaded me that my analysis in grandparent of the Russian-roulette scenario is probably incorrect.
The scenario of the black box that responds with either “heads” or “tails” is different because in the Russian-roulette scenario, we have a partial causal model of the “bang”/”no bang” event. (In particular, we know that the revolver contains either one bullet or zero bullets.) Apparently, causal knowledge can interact with knowledge of past behavior to produce knowledge of future behavior even if the knowledge of past behavior is subject to the strongest kind of observational selection efffects.
Your last point was persuasive… though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing.
Would it be within the first AGI’s capabilities to immediately effect my destruction before I am able to update on its existence—provided that (a) it is developed by the private sector and not e.g. some special access DoD program, and (b) ETAs up to “sometime this century” are accurate? I think not, though I admit to being fairly uncertain.
I acknowledge that this line of reasoning presented in my original comment was not of high caliber—though I still dispute Tiiba’s claim regarding an AI advanced enough to scrape by in conversation with a 5 year old, as well as that distributive capabilities are the greatest power at play here.
I humbly suggest that the answer to your question would not shed any particular light on what we have been talking about because even if we would certainly have noticed the birth of the AGI, there’s a selection effect if it would have killed us before we got around to having this conversation (i.e. if it would have killed us by now).
The AGI’s causing our deaths is not the only thing that would cause a selection effect: the AGI’s deleting our memories of the existence of the AGI would also do it. But the AGI’s causing our deaths is the mostly likely selection-effecting mechanism.
A nice summary of my position is that when we try to estimate the safety of AGI research done in the past, the fact that P(we would have noticed our doom by now|the research killed us or will kill us) is high does not support the safety of the research as much as one might naively think. For us to use that fact the way we use most facts, not only must we notice our doom, but also we must survive long enough to have this conversation.
Actually, we can generalize that last sentence: for a group of people correctly to use the outcome of past AGI research to help assess the safety of AGI, awareness of both possible outcomes (the good outcome and the bad outcome) of the past research must be able to reach the group and in particular must be able to reach the assessment process. More precisely, if there is a mechanism that is more likely to prevent awareness of one outcome from reaching the assessment process than the other outcome, the process has to adjust for that, and if the very existence of the assessment process completely depends on one outcome, the adjustment completely wipes out the “evidentiary value” of awareness of the outcome. The likelihood ratio gets adjusted to 1. The posterior probability (i.e., the probability after updating on the outcome of the research) that AGI is safe is the same as the prior probability.
Like I said yesterday I retract my position on the Russian roulette. (Selection effects operate, I still believe, but not to the extent of making past behavior completely useless for predicting future behavior.)