I was rather intemperate, and on a different day maybe I would have been less so; or maybe I wouldn’t. I am sorry that I offended Wei Dai.
But then, Wei Dai’s posting was intemperate, as is your comment. I mention this not to excuse mine, just to point out how easily this happens. This may be partly the dynamics of the online medium, but in the present case I think it is also because we are dealing in fantasy here, and fantasy always has to be more extreme than reality, to make up for its own unreality.
You compare the problem to Eliezer’s one of TORTURE vs SPECKS, but there is an important difference between them. TORTURE vs SPECKS is fiction, while Wei Dai spoke of an actual juncture in history in living memory, and actions that actually could have been taken.
What is the TORTURE vs SPECKS problem? The formulation of the problem is at that link, but what sort of thing is this problem? Given the followup posting the very next day, it seems likely to me that the intention was to manifest people’s reactions to the problem. Perhaps it is also a touchstone, to see who has and who has not learned the material on which it stands. What it is not is a genuine problem which anyone needs to solve as anything but a thought experiment. TORTURE vs SPECKS is not going to happen. Other tradeoffs between great evil to one and small evils to many do happen; this one never will. While 50 years of torture is, regrettably, conceivably possible here and now in the real world, and may be happening to someone, somewhere, right now, there is no possibility of 3^^^3 specks. Why 3^^^3? Because that is intended to be a number large enough to produce the desired conclusion. Anyone whose objection is that it isn’t a big enough number, besides manifesting a poor grasp of its magnitude, can simply add another uparrow. The problem is a fictional one, and as such exhibits the reverse meta-causality characteristic of fiction: 3^^^3 is in the problem because the point of the problem is for the solution to be TORTURE; that TORTURE is the solution is not caused by an actual possibility of 3^^^3 specks.
In another posting a year later, Eliezer speaks of ethical rules of the sort that you just don’t break, as safety rails on a cliff he didn’t see. This does not sit well with the TORTURE vs SPECKS material, but it doesn’t have to: TORTURE vs SPECKS is fiction and the later posting is about real (though unspecified) actions.
So, the Cold War. Wei Dai would have the US after WWII threatening to nuke any country attempting to develop or test nuclear weapons. To the scenario of later discovering that (for example) the UK has a well-developed covert nuclear program, he responds:
I’d give the following announcement: “People of the UK, please vote your government out of office and shut down your nuclear program. If you fail to do so, we will start nuking the following sites in sequence, one per day, starting [some date].” Well, I’d go through some secret diplomacy first, but that would be my endgame if all else failed. Some backward induction should convince the UK government not to start the nuclear program in the first place.
It should, should it? And that, in Wei’s mind, is adequate justification for pressing the button to kill millions of people for not doing what he told them to do. Is this rationality, or the politics of two-year-olds with nukes?
I seem to be getting intemperate again.
It’s a poor sort of rationality that only works against people rational enough to lose. Or perhaps they can be superrational and precommit to developing their programme regardless of what threats you make? Then rationally, you must see that it would therefore be futile to make such threats. And so on. How’s TDT/UDT with self-modifying agents modelling themselves and each other coming along?
This is fantasy masquerading as rationality. I stand by this that I said back then:
[I]t’s easy to win these games in your imagination. You just have to think, I will do this, and then my opponent must rationally do that. You have a completely watertight argument. Then your opponent goes and does something else. It does not matter that you followed the rules of the logical system if the system itself is inconsistent.
To make these threats, you must be willing to actually do what you have said you will do if your enemy does not surrender. The moment you think “but rationally he has to surrender so I won’t have to do this”, you are making an excuse for yourself to not carry it out. Whatever belief you can muster that you would will evaporate like dew in the desert when the time comes.
But then, Wei Dai’s posting was intemperate, as is your comment. I mention this not to excuse mine, just to point out how easily this happens.
Using the word “intemperate” in this way is a remarkable dodge. Wei Dai’s comment was entirely within the scope of the (admittedly extreme) hypothetical under discussion. Your comment contained a paragraph composed solely of vile personal insult and slanted misrepresentation of Wei Dai’s statements. The tone of my response was deliberate and quite restrained relative to how I felt.
This may be partly the dynamics of the online medium, but in the present case I think it is also because we are dealing in fantasy here, and fantasy always has to be more extreme than reality, to make up for its own unreality.
Huh? You’re “not excusing” the extremity of your interpersonal behavior on the grounds that the topic was fictional, and fiction is more extreme than reality? And then go on to explain that you don’t behave similarly toward Eliezer with respect to his position on TORTURE vs SPECKS because that topic is even more fictional?
Is this rationality, or the politics of two-year-olds with nukes?
Is this a constructive point, or just more gesturing?
As for the rest of your comment: Thank you! This is the discussion I wanted to be reading all along. Aside from a general feeling that you’re still not really trying to be fair, my remaining points are mercifully non-meta. To dampen political distractions, I’ll refer to the nuke-holding country as H, and a nuke-developing country as D.
You’re very focused on Wei Dai’s statement about backward induction, but I think you’re missing a key point: His strategy does not depend on D reasoning the way he expects them to, it’s just heavily optimized for this outcome. I believe he’s right to say that backward induction should convince D to comply, in the sense that it is in their own best interest to do so.
Or perhaps they can be superrational and precommit to developing their programme regardless of what threats you make? Then rationally, you must see that it would therefore be futile to make such threats.
Don’t see how this follows. If both countries precommit, D gets bombed until it halts or otherwise cannot continue development. While this is not H’s preferred outcome, H’s entire strategy is predicated on weighing irreversible nuclear proliferation and its consequences more heavily than the millions of lives lost in the event of a suicidal failure to comply. In other words, D doesn’t wield sufficient power in this scenario to affect H’s decision, while H holds sufficient power to skew local incentives toward mutually beneficial outcomes.
Speaking of nuclear proliferation and its consequences, you’ve been pretty silent on this topic considering that preventing proliferation is the entire motivation for Wei Dai’s strategy. Talking about “murdering millions” without at least framing it alongside the horror of proliferation is not productive.
How are you going to launch those nukes, anyway?
Practical considerations like this strike me as by far the best arguments against extreme, theory-heavy strategies. Messy real-world noise can easily make a high-stakes gambit more trouble than it’s worth.
Is this rationality, or the politics of two-year-olds with nukes?
Is this a constructive point, or just more gesturing?
It is a gesture concluding a constructive point.
You’re very focused on Wei Dai’s statement about backward induction, but I think you’re missing a key point: His strategy does not depend on D reasoning the way he expects them to, it’s just heavily optimized for this outcome. I believe he’s right to say that backward induction should convince D to comply, in the sense that it is in their own best interest to do so.
This is a distinction without a difference. If H bombs D, H has lost (and D has lost more).
If both countries precommit, D gets bombed until it halts or otherwise cannot continue development.
That depends on who precommits “first”. That’s a problematic concept for rational actors who have plenty of time to model each others’ possible strategies in advance of taking action. If H, without even being informed of it by D, considers this possible precommitment strategy of D, is it still rational for H to persist and threaten D anyway? Or perhaps H can precommit to ignoring such a precommitment by D? Or should D already have anticipated H’s original threat and backed down in advance of the threat ever having been made? I am reminded of the Forbidden Topic. Counterfactual blackmail isn’t just for superintelligences. As I asked before, does the decision theory exist yet to handle self-modifying agents modelling themselves and others, demonstrating how real actions can arise from this seething mass of virtual possibilities?
Then also, in what you dismiss as “messy real-world noise”, there may be a lot of other things D might do, such as fomenting insurrection in H, or sharing their research with every other country besides H (and blaming foreign spies), or assassinating H’s leader, or doing any and all of these while overtly appearing to back down.
The moment H makes that threat, the whole world is H’s enemy. H has declared a war that it hopes to win by the mere possession of overwhelming force.
Speaking of nuclear proliferation and its consequences, you’ve been pretty silent on this topic considering that preventing proliferation is the entire motivation for Wei Dai’s strategy. Talking about “murdering millions” without at least framing it alongside the horror of proliferation is not productive.
I look around at the world since WWII and fail to see this horror. I look at Wei Dai’s strategy and see the horror. loqi remarked about Everett branches, but imagining the measure of the wave function where the Cold War ended with nuclear conflagration fails to convince me of anything.
This is a distinction without a difference. If H bombs D, H has lost
This assumption determines (or at least greatly alters) the debate, and you need to make a better case for it. If H really “loses” by bombing D (meaning H considers this outcome less preferable than proliferation), then H’s threat is not credible, and the strategy breaks down, no exotic decision theory necessary. Looks like a crucial difference to me.
That depends on who precommits “first”. [...]
This entire paragraph depends on the above assumption. If I grant you that assumption and (artificially) hold constant H’s intent to precommit, then we’ve entered the realm of bluffing, and yes, the game tree gets pathological.
loqi remarked about Everett branches, but imagining the measure of the wave function where the Cold War ended with nuclear conflagration fails to convince me of anything.
My mention of Everett branches was an indirect (and counter-productive) way of accusing you of hindsight bias.
Your talk of “convincing you” is distractingly binary. Do you admit that the severity and number of close calls in the Cold War is relevant to this discussion, and that these are positively correlated with the underlying justification for Wei Dai’s strategy? (Not necessarily its feasibility!)
I look around at the world since WWII and fail to see this horror. I look at Wei Dai’s strategy and see the horror.
Let’s set aside scale and comparisons for a moment, because your position looks suspiciously one-sided. You fail to see the horror of nuclear proliferation? If I may ask, what is your estimate for the probability that a nuclear weapon will be deployed in the next 100 years? Did you even ask yourself this question, or are you just selectively attending to the low-probability horrors of Wei Dai’s strategy?
Then also, in what you dismiss as “messy real-world noise”
Emphasis mine. You are compromised. Please take a deep breath (really!) and re-read my comment. I was not dismissing your point in the slightest, I was in fact stating my belief that it exemplified a class of particularly effective counter-arguments in this context.
I was rather intemperate, and on a different day maybe I would have been less so; or maybe I wouldn’t. I am sorry that I offended Wei Dai.
But then, Wei Dai’s posting was intemperate, as is your comment. I mention this not to excuse mine, just to point out how easily this happens. This may be partly the dynamics of the online medium, but in the present case I think it is also because we are dealing in fantasy here, and fantasy always has to be more extreme than reality, to make up for its own unreality.
You compare the problem to Eliezer’s one of TORTURE vs SPECKS, but there is an important difference between them. TORTURE vs SPECKS is fiction, while Wei Dai spoke of an actual juncture in history in living memory, and actions that actually could have been taken.
What is the TORTURE vs SPECKS problem? The formulation of the problem is at that link, but what sort of thing is this problem? Given the followup posting the very next day, it seems likely to me that the intention was to manifest people’s reactions to the problem. Perhaps it is also a touchstone, to see who has and who has not learned the material on which it stands. What it is not is a genuine problem which anyone needs to solve as anything but a thought experiment. TORTURE vs SPECKS is not going to happen. Other tradeoffs between great evil to one and small evils to many do happen; this one never will. While 50 years of torture is, regrettably, conceivably possible here and now in the real world, and may be happening to someone, somewhere, right now, there is no possibility of 3^^^3 specks. Why 3^^^3? Because that is intended to be a number large enough to produce the desired conclusion. Anyone whose objection is that it isn’t a big enough number, besides manifesting a poor grasp of its magnitude, can simply add another uparrow. The problem is a fictional one, and as such exhibits the reverse meta-causality characteristic of fiction: 3^^^3 is in the problem because the point of the problem is for the solution to be TORTURE; that TORTURE is the solution is not caused by an actual possibility of 3^^^3 specks.
In another posting a year later, Eliezer speaks of ethical rules of the sort that you just don’t break, as safety rails on a cliff he didn’t see. This does not sit well with the TORTURE vs SPECKS material, but it doesn’t have to: TORTURE vs SPECKS is fiction and the later posting is about real (though unspecified) actions.
So, the Cold War. Wei Dai would have the US after WWII threatening to nuke any country attempting to develop or test nuclear weapons. To the scenario of later discovering that (for example) the UK has a well-developed covert nuclear program, he responds:
It should, should it? And that, in Wei’s mind, is adequate justification for pressing the button to kill millions of people for not doing what he told them to do. Is this rationality, or the politics of two-year-olds with nukes?
I seem to be getting intemperate again.
It’s a poor sort of rationality that only works against people rational enough to lose. Or perhaps they can be superrational and precommit to developing their programme regardless of what threats you make? Then rationally, you must see that it would therefore be futile to make such threats. And so on. How’s TDT/UDT with self-modifying agents modelling themselves and each other coming along?
This is fantasy masquerading as rationality. I stand by this that I said back then:
To make these threats, you must be willing to actually do what you have said you will do if your enemy does not surrender. The moment you think “but rationally he has to surrender so I won’t have to do this”, you are making an excuse for yourself to not carry it out. Whatever belief you can muster that you would will evaporate like dew in the desert when the time comes.
How are you going to launch those nukes, anyway?
Using the word “intemperate” in this way is a remarkable dodge. Wei Dai’s comment was entirely within the scope of the (admittedly extreme) hypothetical under discussion. Your comment contained a paragraph composed solely of vile personal insult and slanted misrepresentation of Wei Dai’s statements. The tone of my response was deliberate and quite restrained relative to how I felt.
Huh? You’re “not excusing” the extremity of your interpersonal behavior on the grounds that the topic was fictional, and fiction is more extreme than reality? And then go on to explain that you don’t behave similarly toward Eliezer with respect to his position on TORTURE vs SPECKS because that topic is even more fictional?
Is this a constructive point, or just more gesturing?
As for the rest of your comment: Thank you! This is the discussion I wanted to be reading all along. Aside from a general feeling that you’re still not really trying to be fair, my remaining points are mercifully non-meta. To dampen political distractions, I’ll refer to the nuke-holding country as H, and a nuke-developing country as D.
You’re very focused on Wei Dai’s statement about backward induction, but I think you’re missing a key point: His strategy does not depend on D reasoning the way he expects them to, it’s just heavily optimized for this outcome. I believe he’s right to say that backward induction should convince D to comply, in the sense that it is in their own best interest to do so.
Don’t see how this follows. If both countries precommit, D gets bombed until it halts or otherwise cannot continue development. While this is not H’s preferred outcome, H’s entire strategy is predicated on weighing irreversible nuclear proliferation and its consequences more heavily than the millions of lives lost in the event of a suicidal failure to comply. In other words, D doesn’t wield sufficient power in this scenario to affect H’s decision, while H holds sufficient power to skew local incentives toward mutually beneficial outcomes.
Speaking of nuclear proliferation and its consequences, you’ve been pretty silent on this topic considering that preventing proliferation is the entire motivation for Wei Dai’s strategy. Talking about “murdering millions” without at least framing it alongside the horror of proliferation is not productive.
Practical considerations like this strike me as by far the best arguments against extreme, theory-heavy strategies. Messy real-world noise can easily make a high-stakes gambit more trouble than it’s worth.
It is a gesture concluding a constructive point.
This is a distinction without a difference. If H bombs D, H has lost (and D has lost more).
That depends on who precommits “first”. That’s a problematic concept for rational actors who have plenty of time to model each others’ possible strategies in advance of taking action. If H, without even being informed of it by D, considers this possible precommitment strategy of D, is it still rational for H to persist and threaten D anyway? Or perhaps H can precommit to ignoring such a precommitment by D? Or should D already have anticipated H’s original threat and backed down in advance of the threat ever having been made? I am reminded of the Forbidden Topic. Counterfactual blackmail isn’t just for superintelligences. As I asked before, does the decision theory exist yet to handle self-modifying agents modelling themselves and others, demonstrating how real actions can arise from this seething mass of virtual possibilities?
Then also, in what you dismiss as “messy real-world noise”, there may be a lot of other things D might do, such as fomenting insurrection in H, or sharing their research with every other country besides H (and blaming foreign spies), or assassinating H’s leader, or doing any and all of these while overtly appearing to back down.
The moment H makes that threat, the whole world is H’s enemy. H has declared a war that it hopes to win by the mere possession of overwhelming force.
I look around at the world since WWII and fail to see this horror. I look at Wei Dai’s strategy and see the horror. loqi remarked about Everett branches, but imagining the measure of the wave function where the Cold War ended with nuclear conflagration fails to convince me of anything.
This assumption determines (or at least greatly alters) the debate, and you need to make a better case for it. If H really “loses” by bombing D (meaning H considers this outcome less preferable than proliferation), then H’s threat is not credible, and the strategy breaks down, no exotic decision theory necessary. Looks like a crucial difference to me.
This entire paragraph depends on the above assumption. If I grant you that assumption and (artificially) hold constant H’s intent to precommit, then we’ve entered the realm of bluffing, and yes, the game tree gets pathological.
My mention of Everett branches was an indirect (and counter-productive) way of accusing you of hindsight bias.
Your talk of “convincing you” is distractingly binary. Do you admit that the severity and number of close calls in the Cold War is relevant to this discussion, and that these are positively correlated with the underlying justification for Wei Dai’s strategy? (Not necessarily its feasibility!)
Let’s set aside scale and comparisons for a moment, because your position looks suspiciously one-sided. You fail to see the horror of nuclear proliferation? If I may ask, what is your estimate for the probability that a nuclear weapon will be deployed in the next 100 years? Did you even ask yourself this question, or are you just selectively attending to the low-probability horrors of Wei Dai’s strategy?
Emphasis mine. You are compromised. Please take a deep breath (really!) and re-read my comment. I was not dismissing your point in the slightest, I was in fact stating my belief that it exemplified a class of particularly effective counter-arguments in this context.