I think the lesson is that if you decide to launch the nukes it’s better to claim incompetence rather than malice because then opinion of you among the survivors won’t suffer as much.
I think we learned that when you tell people to not destroy the world they try to not destroy the world. How is [press this button and the world ends → don’t press button] different from [press this button or else the world ends → press button]?
The asymmetry is the button itself. If I understand correctly, Chris got this message on a separate channel, and the button still looked the same; it still said “enter launch codes to destroy LessWrong”. It was still clearly meant to represent the launch of nukes.
Stretching just a bit, I think you might be able to draw an analogy here, where real people who might actually launch nuclear weapons (or have done so in other branches of the multiverse) have thought they had reasons important enough to justify doing it. But in fact, the rule is not “don’t launch nukes unless there seems to be sufficient reason for it”, but rather “don’t launch nukes”.
Good point. I didn’t see the button setup before it went down, and I was thinking the OP did not receive the main email and just got the “special instructions” they posted. This does make it more analogous to a “false alarm” situation.
I think this is the wrong lesson. If the Enemy knows you have precommitted to never press the button, then they are not deterred from striking first. MAD is game theory. In order to not blow up the world, you have to be willing to blow up the world. It’s a Newcomblike problem: It feels like there are two decisions to be made, but there is only one, in advance.
But we are not in a game theory situation. We are in an imperfect world with imperfect information. There are malfunctioning warning systems and liars. And we are humans and not programs that get to read each others source code. There are no perfect commitments and if there where, there would be no way of verifying them.
So I think that the lesson is, that what ever your public stance, and whether or not you think that there are counterfactual situation where you should nuke. In practice, you should not nuke.
Game theory was pioneered by Schelling with the central and most important application being handling nuclear armed conflicts. To say that game theory doesn’t apply to nuclear conflict because we live in an imperfect world is just not accurate. Game theory doesn’t require a perfect world nor does it require that actors know each other’s source code. It is designed to guide decisions made in the real world.
I know that it is designed to guide decisions made in the real world. This does not force me to agree with the conclusions in all circumstances. Lots of models are not up to the task they are designed to deal with.
But I should have said “not in that game theory situation”, becasue there is probably a way to construct some game theory game that applies here. That was my bad.
However, I stand by the claim that the full information game is too far from reality to be a good guide in this case. With stakes this high even small uncertainty becomes important.
Game theory is very much applicable to the real world. Imperfect information is just a different game. You are correct that assuming perfect information is a simplification. But assuming imperfect information, what does that change?
You want to lie to the Enemy, convince them that you will always push the button if they cross the line, then never actually do it, and the Enemy knows this!
Sometimes all available options are risky. Betting your life on a coin flip is not generally a good idea, but if the only alternative is a lottery ticket, the coin flip looks pretty good. If the Enemy knows there’s a significant chance that you won’t press the button, in a sufficiently desperate situation, the Enemy might bet on that and strike first. But if the Enemy knows self-destruction is assured, then striking first looks like a bad option.
What possible reason could Petrov or those in similar situations have had for not pushing the button? Maybe he believed that the US would retaliate and kill his family at home, and that deterred him. In other words, he believed his enemy would push the button.
Applied to the real world, game theory is not just about how to play the games. It’s also about the effects of changing the rules.
What possible reason could Petrov or those in similar situations have had for not pushing the button? Maybe he believed that the US would retaliate and kill his family at home, and that deterred him. In other words, he believed his enemy would push the button.
Or maybe he just did not want to kill millions of people?
In Petrov’s case in particular, the new satellite-based early warning system was unproven so he didn’t completely trust it, and he didn’t believe a US first strike would use only one missile, or later, only four more, instead of hundreds. Furthermore, ground radar didn’t confirm. And, of course, attacking on a false alarm would be suicidal because he believed the Enemy would push the button, so striking first “just in case”, failed his cost-benefit analysis.
I should probably have said “we are not in that game theory situation”. (Though I do think that the real world is more complex that current game theory can handle. E.g. I don’t think current game theory can fully handle unknown-unknown, but I could be wrong on this point)
The game of mutually assured destruction is very different even when just including known unknown.
In other words, if your defense is “just following orders”, you’re in the wrong. Petrov, too, was strongly influenced to launch the nukes, and still refused… Like that Soviet submarine commander who, during the Cuban Missile Crisis, thought he was engaged with live depth charges by the US Navy.
To not have buttons that can be pressed to destroy the world, since the possibility existing for many agents is the real issue, because circumstances could deliver compelling reasons to press it and the more buttons exist the more likely is it that it will happen .
Reality won’t deliver the same circumstances twice. If the petrovdayadmin wanted to go for symmetry the message would have said: someone of the 270 pressed the red button, if you want to keep the home page online 5 people have to press it within etc, etc...
From this we learn that you should not launch nukes, even if someone tells you to do it.
I think the lesson is that if you decide to launch the nukes it’s better to claim incompetence rather than malice because then opinion of you among the survivors won’t suffer as much.
I think we learned that when you tell people to not destroy the world they try to not destroy the world. How is [press this button and the world ends → don’t press button] different from [press this button or else the world ends → press button]?
I think we learned that trolls will destroy the world.
The asymmetry is the button itself. If I understand correctly, Chris got this message on a separate channel, and the button still looked the same; it still said “enter launch codes to destroy LessWrong”. It was still clearly meant to represent the launch of nukes.
Stretching just a bit, I think you might be able to draw an analogy here, where real people who might actually launch nuclear weapons (or have done so in other branches of the multiverse) have thought they had reasons important enough to justify doing it. But in fact, the rule is not “don’t launch nukes unless there seems to be sufficient reason for it”, but rather “don’t launch nukes”.
Good point. I didn’t see the button setup before it went down, and I was thinking the OP did not receive the main email and just got the “special instructions” they posted. This does make it more analogous to a “false alarm” situation.
I recieved both messages
I think this is the wrong lesson. If the Enemy knows you have precommitted to never press the button, then they are not deterred from striking first. MAD is game theory. In order to not blow up the world, you have to be willing to blow up the world. It’s a Newcomblike problem: It feels like there are two decisions to be made, but there is only one, in advance.
But we are not in a game theory situation. We are in an imperfect world with imperfect information. There are malfunctioning warning systems and liars. And we are humans and not programs that get to read each others source code. There are no perfect commitments and if there where, there would be no way of verifying them.
So I think that the lesson is, that what ever your public stance, and whether or not you think that there are counterfactual situation where you should nuke. In practice, you should not nuke.
Do you see what I’m getting at?
Game theory was pioneered by Schelling with the central and most important application being handling nuclear armed conflicts. To say that game theory doesn’t apply to nuclear conflict because we live in an imperfect world is just not accurate. Game theory doesn’t require a perfect world nor does it require that actors know each other’s source code. It is designed to guide decisions made in the real world.
I know that it is designed to guide decisions made in the real world. This does not force me to agree with the conclusions in all circumstances. Lots of models are not up to the task they are designed to deal with.
But I should have said “not in that game theory situation”, becasue there is probably a way to construct some game theory game that applies here. That was my bad.
However, I stand by the claim that the full information game is too far from reality to be a good guide in this case. With stakes this high even small uncertainty becomes important.
Game theory is very much applicable to the real world. Imperfect information is just a different game. You are correct that assuming perfect information is a simplification. But assuming imperfect information, what does that change?
You want to lie to the Enemy, convince them that you will always push the button if they cross the line, then never actually do it, and the Enemy knows this!
Sometimes all available options are risky. Betting your life on a coin flip is not generally a good idea, but if the only alternative is a lottery ticket, the coin flip looks pretty good. If the Enemy knows there’s a significant chance that you won’t press the button, in a sufficiently desperate situation, the Enemy might bet on that and strike first. But if the Enemy knows self-destruction is assured, then striking first looks like a bad option.
What possible reason could Petrov or those in similar situations have had for not pushing the button? Maybe he believed that the US would retaliate and kill his family at home, and that deterred him. In other words, he believed his enemy would push the button.
Applied to the real world, game theory is not just about how to play the games. It’s also about the effects of changing the rules.
Or maybe he just did not want to kill millions of people?
In Petrov’s case in particular, the new satellite-based early warning system was unproven so he didn’t completely trust it, and he didn’t believe a US first strike would use only one missile, or later, only four more, instead of hundreds. Furthermore, ground radar didn’t confirm. And, of course, attacking on a false alarm would be suicidal because he believed the Enemy would push the button, so striking first “just in case”, failed his cost-benefit analysis.
It was not “just” a commitment to pacifism.
I should probably have said “we are not in that game theory situation”.
(Though I do think that the real world is more complex that current game theory can handle. E.g. I don’t think current game theory can fully handle unknown-unknown, but I could be wrong on this point)
The game of mutually assured destruction is very different even when just including known unknown.
In other words, if your defense is “just following orders”, you’re in the wrong. Petrov, too, was strongly influenced to launch the nukes, and still refused… Like that Soviet submarine commander who, during the Cuban Missile Crisis, thought he was engaged with live depth charges by the US Navy.
I think the lessons are:
To not have buttons that can be pressed to destroy the world, since the possibility existing for many agents is the real issue, because circumstances could deliver compelling reasons to press it and the more buttons exist the more likely is it that it will happen .
Reality won’t deliver the same circumstances twice. If the petrovdayadmin wanted to go for symmetry the message would have said: someone of the 270 pressed the red button, if you want to keep the home page online 5 people have to press it within etc, etc...