I’d be interested in reading the literature you mention that suggests positive outcomes are more likely than negative outcomes, conditioned on AGI being developed. My sense is that if AGI is developed and the transition goes badly for humans, but an individual still lives for a long time, then it’s quite likely that the individual has a bad life since if you select uniformly from environments that keep humans alive but are otherwise unoptimized for wellbeing, I’d expect most to be quite unhappy.
It also seems like you place around 66% probability (2.5 : 1.3) on our chances of successfully navigating the intelligence explosion. This seems quite high and may be worth pulling out into a separate variable just to make it more explicit.
I don’t remember many specific instances of literature suggesting this, but have definitely seen it here and there, and have discussed it with other people of the safety community who seem to strongly share the assumption.
I’d agree that there would be almost all unhappiness if taken from the distribution of randomly chosen situations that could keep humans alive, but this doesn’t seem like the right distribution to me. I think there’s a much likelier chance that an AI would keep us alive if it were the sake of making us happy, than for any other purpose. I honestly don’t know what humans are good for in the long run other thank being kept alive for enjoyment / for near-AGI negotiation deals.
I think an analogy is humans and animals. In the short run we may be making things worse for them, but in the long run I’d bet we won’t really care about them (we’d have other ways of getting nutrition) and would either kill them all or keep some in zoos/museums/protected lands. Their future basically depends on what uses they could satisfice, and in the long run it’s hard to satisfice anything better than fancy tech.
On point two:
The vast majority of the probability is in us failing the intelligence explosion and all dying. In this model that equates to 0 Expected Value. (It states that it doesn’t take into account opportunity cost). So more specifically, this model claims around a ~99.7% chance that a given person will die, ~0.25% chance they will live on indefinitely in a good way, and a ~0.16% chance they will live indefinitely in a bad way.
I’d be interested in reading the literature you mention that suggests positive outcomes are more likely than negative outcomes, conditioned on AGI being developed. My sense is that if AGI is developed and the transition goes badly for humans, but an individual still lives for a long time, then it’s quite likely that the individual has a bad life since if you select uniformly from environments that keep humans alive but are otherwise unoptimized for wellbeing, I’d expect most to be quite unhappy.
It also seems like you place around 66% probability (2.5 : 1.3) on our chances of successfully navigating the intelligence explosion. This seems quite high and may be worth pulling out into a separate variable just to make it more explicit.
I don’t remember many specific instances of literature suggesting this, but have definitely seen it here and there, and have discussed it with other people of the safety community who seem to strongly share the assumption.
I’d agree that there would be almost all unhappiness if taken from the distribution of randomly chosen situations that could keep humans alive, but this doesn’t seem like the right distribution to me. I think there’s a much likelier chance that an AI would keep us alive if it were the sake of making us happy, than for any other purpose. I honestly don’t know what humans are good for in the long run other thank being kept alive for enjoyment / for near-AGI negotiation deals.
I think an analogy is humans and animals. In the short run we may be making things worse for them, but in the long run I’d bet we won’t really care about them (we’d have other ways of getting nutrition) and would either kill them all or keep some in zoos/museums/protected lands. Their future basically depends on what uses they could satisfice, and in the long run it’s hard to satisfice anything better than fancy tech.
On point two:
The vast majority of the probability is in us failing the intelligence explosion and all dying. In this model that equates to 0 Expected Value. (It states that it doesn’t take into account opportunity cost). So more specifically, this model claims around a ~99.7% chance that a given person will die, ~0.25% chance they will live on indefinitely in a good way, and a ~0.16% chance they will live indefinitely in a bad way.