I’ll note that I don’t claim much justification for my views, I’m mostly just stating them to promote some thought:
The E(V) definition is definitionally true, but needs translation to gain any practical meaning, whereas Bostrom’s definition of loss of potential is theoretically reasonable kosher, works across a lot of value systems, and has the additional virtue of being more practically meaningful. Extinction misses a bunch of bad fail-scenarios, but is at least concrete. So it’s better to use the latter two by default, and it’s rare that we’d need to use the E(V) definition.
The main other type of definition for existential risk, which Daniel Dewey had suggested to me is existential risk as the reduction of the number of options available to you. This definition seems kind of doomed to me, because I have a feeling it collapses into some kind of physical considerations. e.g. how many particles you have / how many conformations they can take / how much energy or entropy they have—something like that. I think you either need to anchor your definition on extinction, or expected value, or something in-between like “potential”, but if you try to define ‘options’ objectively, I think you end up with something we don’t care about anymore.
Existential eucatastrophes are interesting. Perhaps making a simulation would count. However, they might not exist if we believe the technological maturity conjecture. The notions of expanding cubically / astronomical waste feel kinda relevant here—if failing some extinction event, everything’s going to end up expanding cubically anyway, then you have to decide how you can counterfactually stuff extra value in there. It still seems like the main point of leverage over the future would be when an AI is created. And in that context, I think we want people to cooperate and satisfice (reduce existential risk) rather than quibble and all try to create personal eucatastrophes. So I’m not sure the terminology helps.
And in that context, I think we want people to cooperate and satisfice (reduce existential risk) rather than quibble and all try to create personal eucatastrophes. So I’m not sure the terminology helps.
I tend to agree that the main point of leverage over the future will be whether there is a long future. However I think focusing on the risks may focus attention too much on strategies which address the risks directly, where we would be better aiming at a positive intermediate state.
I’ll note that I don’t claim much justification for my views, I’m mostly just stating them to promote some thought:
The E(V) definition is definitionally true, but needs translation to gain any practical meaning, whereas Bostrom’s definition of loss of potential is theoretically reasonable kosher, works across a lot of value systems, and has the additional virtue of being more practically meaningful. Extinction misses a bunch of bad fail-scenarios, but is at least concrete. So it’s better to use the latter two by default, and it’s rare that we’d need to use the E(V) definition.
The main other type of definition for existential risk, which Daniel Dewey had suggested to me is existential risk as the reduction of the number of options available to you. This definition seems kind of doomed to me, because I have a feeling it collapses into some kind of physical considerations. e.g. how many particles you have / how many conformations they can take / how much energy or entropy they have—something like that. I think you either need to anchor your definition on extinction, or expected value, or something in-between like “potential”, but if you try to define ‘options’ objectively, I think you end up with something we don’t care about anymore.
Existential eucatastrophes are interesting. Perhaps making a simulation would count. However, they might not exist if we believe the technological maturity conjecture. The notions of expanding cubically / astronomical waste feel kinda relevant here—if failing some extinction event, everything’s going to end up expanding cubically anyway, then you have to decide how you can counterfactually stuff extra value in there. It still seems like the main point of leverage over the future would be when an AI is created. And in that context, I think we want people to cooperate and satisfice (reduce existential risk) rather than quibble and all try to create personal eucatastrophes. So I’m not sure the terminology helps.
Anyhow, it’s all interesting to think about.
I tend to agree that the main point of leverage over the future will be whether there is a long future. However I think focusing on the risks may focus attention too much on strategies which address the risks directly, where we would be better aiming at a positive intermediate state.