I think that we should expect it to be extremely unlikely for an agent to have the required power and willingness to kill 3^^^^3 people. The shortest explanation for why we should believe this is that any agent that gathers “power” will increase his ability to influence the state of the universe. As his “power” grows, the number of possible states that the agent can successfully attain scales like the factorial with respect to power. If we denote power with x, then the number of possible states that the agent can attain is proportional to x!. Now, the expected utility of the threat (the utility of the threat becoming true times the probability of the threat being done) becomes proportional to -x * (1 / x!). From this, we can see that actually the greater the threat, the smaller its absolute magnitude. From this, we can conclude that the expected utility of the threat is 0 for a sufficiently large x.
As for why the number of attainable states is factorial? Well, I think that it is self-explanatory that as an agent’s power over space, time, matter, and energy grows, it can arrange them in many different arrangements, and there is roughly x! of those arrangements, where x is the number of parameters it is working with. This is not mathematically rigorous as it is a blog post and I am not a mathematician but I think this should be logical to anyone thinking about this problem.
I think that we should expect it to be extremely unlikely for an agent to have the required power and willingness to kill 3^^^^3 people. The shortest explanation for why we should believe this is that any agent that gathers “power” will increase his ability to influence the state of the universe. As his “power” grows, the number of possible states that the agent can successfully attain scales like the factorial with respect to power. If we denote power with x, then the number of possible states that the agent can attain is proportional to x!. Now, the expected utility of the threat (the utility of the threat becoming true times the probability of the threat being done) becomes proportional to -x * (1 / x!). From this, we can see that actually the greater the threat, the smaller its absolute magnitude. From this, we can conclude that the expected utility of the threat is 0 for a sufficiently large x.
As for why the number of attainable states is factorial? Well, I think that it is self-explanatory that as an agent’s power over space, time, matter, and energy grows, it can arrange them in many different arrangements, and there is roughly x! of those arrangements, where x is the number of parameters it is working with. This is not mathematically rigorous as it is a blog post and I am not a mathematician but I think this should be logical to anyone thinking about this problem.