Your last point was persuasive… though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing.
Would it be within the first AGI’s capabilities to immediately effect my destruction before I am able to update on its existence—provided that (a) it is developed by the private sector and not e.g. some special access DoD program, and (b) ETAs up to “sometime this century” are accurate? I think not, though I admit to being fairly uncertain.
I acknowledge that this line of reasoning presented in my original comment was not of high caliber—though I still dispute Tiiba’s claim regarding an AI advanced enough to scrape by in conversation with a 5 year old, as well as that distributive capabilities are the greatest power at play here.
Would it be within the first AGI’s capabilities to immediately effect my destruction before I am able to update on its existence . . .?
I humbly suggest that the answer to your question would not shed any particular light on what we have been talking about because even if we would certainly have noticed the birth of the AGI, there’s a selection effect if it would have killed us before we got around to having this conversation (i.e. if it would have killed us by now).
The AGI’s causing our deaths is not the only thing that would cause a selection effect: the AGI’s deleting our memories of the existence of the AGI would also do it. But the AGI’s causing our deaths is the mostly likely selection-effecting mechanism.
A nice summary of my position is that when we try to estimate the safety of AGI research done in the past, the fact that P(we would have noticed our doom by now|the research killed us or will kill us) is high does not support the safety of the research as much as one might naively think. For us to use that fact the way we use most facts, not only must we notice our doom, but also we must survive long enough to have this conversation.
Actually, we can generalize that last sentence: for a group of people correctly to use the outcome of past AGI research to help assess the safety of AGI, awareness of both possible outcomes (the good outcome and the bad outcome) of the past research must be able to reach the group and in particular must be able to reach the assessment process. More precisely, if there is a mechanism that is more likely to prevent awareness of one outcome from reaching the assessment process than the other outcome, the process has to adjust for that, and if the very existence of the assessment process completely depends on one outcome, the adjustment completely wipes out the “evidentiary value” of awareness of the outcome. The likelihood ratio gets adjusted to 1. The posterior probability (i.e., the probability after updating on the outcome of the research) that AGI is safe is the same as the prior probability.
Your last point was persuasive… though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing.
Like I said yesterday I retract my position on the Russian roulette. (Selection effects operate, I still believe, but not to the extent of making past behavior completely useless for predicting future behavior.)
Your last point was persuasive… though I still have some uneasiness about accepting that k pulls of the trigger, for arbitrary k, still gives the player nothing.
Would it be within the first AGI’s capabilities to immediately effect my destruction before I am able to update on its existence—provided that (a) it is developed by the private sector and not e.g. some special access DoD program, and (b) ETAs up to “sometime this century” are accurate? I think not, though I admit to being fairly uncertain.
I acknowledge that this line of reasoning presented in my original comment was not of high caliber—though I still dispute Tiiba’s claim regarding an AI advanced enough to scrape by in conversation with a 5 year old, as well as that distributive capabilities are the greatest power at play here.
I humbly suggest that the answer to your question would not shed any particular light on what we have been talking about because even if we would certainly have noticed the birth of the AGI, there’s a selection effect if it would have killed us before we got around to having this conversation (i.e. if it would have killed us by now).
The AGI’s causing our deaths is not the only thing that would cause a selection effect: the AGI’s deleting our memories of the existence of the AGI would also do it. But the AGI’s causing our deaths is the mostly likely selection-effecting mechanism.
A nice summary of my position is that when we try to estimate the safety of AGI research done in the past, the fact that P(we would have noticed our doom by now|the research killed us or will kill us) is high does not support the safety of the research as much as one might naively think. For us to use that fact the way we use most facts, not only must we notice our doom, but also we must survive long enough to have this conversation.
Actually, we can generalize that last sentence: for a group of people correctly to use the outcome of past AGI research to help assess the safety of AGI, awareness of both possible outcomes (the good outcome and the bad outcome) of the past research must be able to reach the group and in particular must be able to reach the assessment process. More precisely, if there is a mechanism that is more likely to prevent awareness of one outcome from reaching the assessment process than the other outcome, the process has to adjust for that, and if the very existence of the assessment process completely depends on one outcome, the adjustment completely wipes out the “evidentiary value” of awareness of the outcome. The likelihood ratio gets adjusted to 1. The posterior probability (i.e., the probability after updating on the outcome of the research) that AGI is safe is the same as the prior probability.
Like I said yesterday I retract my position on the Russian roulette. (Selection effects operate, I still believe, but not to the extent of making past behavior completely useless for predicting future behavior.)