After reading through all of the comments, I think I may have failed to address your central point here.
Your central point seems to be “a rational agent should take a risk that might result in universal destruction in exchange for increased utility”.
The problem here is I’m not sure that this is even a meaningful argument to begin with. Obviously universal destruction is extremely bad, but the problem is that utility probably includes all life NOT being extinguished. Or, in other words, this isn’t necessarily a meaningful calculation if we assume that the alternative makes it more likely that universal annihilation will occur.
Say the Nazis gain an excessive amount of power. What happens then? Well, there’s the risk that they make some sort of plague to cleanse humanity, screw it up, and wipe everyone out. That scenario seems MORE likely in a Nazi-run world than one which isn’t. And—let’s face it—chances are the Nazis will try and develop nuclear weapons, too, so at best you only bought a few years. And if the wrong people develop them first, you’re in a lot of trouble. So the fact of the matter is that the risk is going to be taken regardless, which further diminishes the loss of utility you could expect from universal annihilation—sooner or later, someone is going to do it, and if it isn’t you, then it will be someone else who gains whatever benefits there are from it.
The higher utility situation likely decreases the future odds of universal annihilation, meaning that, in other words, it is entirely rational to take that risk simply because the odds of destroying the world NOW are less than the odds of the world being destroyed further on down the line by someone else if you don’t make this decision, especially if you can be reasonably certain someone else is going to try it out anyway. And given the odds are incredibly low, it is a lot less meaningful of a choice to begin with.
After reading through all of the comments, I think I may have failed to address your central point here.
Your central point seems to be “a rational agent should take a risk that might result in universal destruction in exchange for increased utility”.
The problem here is I’m not sure that this is even a meaningful argument to begin with. Obviously universal destruction is extremely bad, but the problem is that utility probably includes all life NOT being extinguished. Or, in other words, this isn’t necessarily a meaningful calculation if we assume that the alternative makes it more likely that universal annihilation will occur.
Say the Nazis gain an excessive amount of power. What happens then? Well, there’s the risk that they make some sort of plague to cleanse humanity, screw it up, and wipe everyone out. That scenario seems MORE likely in a Nazi-run world than one which isn’t. And—let’s face it—chances are the Nazis will try and develop nuclear weapons, too, so at best you only bought a few years. And if the wrong people develop them first, you’re in a lot of trouble. So the fact of the matter is that the risk is going to be taken regardless, which further diminishes the loss of utility you could expect from universal annihilation—sooner or later, someone is going to do it, and if it isn’t you, then it will be someone else who gains whatever benefits there are from it.
The higher utility situation likely decreases the future odds of universal annihilation, meaning that, in other words, it is entirely rational to take that risk simply because the odds of destroying the world NOW are less than the odds of the world being destroyed further on down the line by someone else if you don’t make this decision, especially if you can be reasonably certain someone else is going to try it out anyway. And given the odds are incredibly low, it is a lot less meaningful of a choice to begin with.