Sorry, I know I said I’d stop, and I will stop after this, but that 3E22 number is just too interesting to leave alone.
The last time humanity was almost destroyed was about 80,000 years ago, when a volcanic eruption reduced the human population below 1,000. So say events that can destroy humanity happen on average every hundred thousand years (conservative assumption, right?). That means the chance of a humanity-destroying event per year is 1⁄100,000. Say 90% of all humanity destroying events can be predicted with at least one day’s notice by eg asteroid monitoring. This leaves hard-to-detect asteroids, sudden volcanic eruptions, weird things like sudden methane release from the ocean, et cetera. So 1⁄1 million years we get an unexpected humanity destroying event. That means the “background rate” of humanity destroying events is 1⁄300 million days.
Suppose Omega told you, the day before the LHC was switched on, that tomorrow humankind would be destroyed. If 1/3E22 were your true probability, you would say “there’s still vastly less than one in a billion chance the apocalypse has anything to do with the LHC, it must just be a coincidence.” Even if you were the LHC project coordinator, you might not even bother to tell them not to switch on the project, because it wasn’t worth the effort it would take to go to the telephone.
Let’s look at it a different way. Suppose a scientist has a one in a thousand chance of having a psychotic break. Now suppose the world’s top physicist, so brilliant as to be literally infallible as long as he is sane, comes up with new calculations that say the LHC will destroy the world. Suppose you ask the world’s best psychiatrist, who also is literally never wrong, whether the physicist is insane, and she says no. If your probability is truly 1/3E22, then it is more likely that both the physicist and the psychiatrist have simultaneously gone insane than that the physicist is correct; what is more, even if you have no other evidence bearing on the sanity of either, your probability should still be less than one in a trillion that the LHC will destroy the Earth.
There was some discussion in LW a while back on how it might be a prediction of anthropic theory that if the LHC destroys the world, improbable occurences will prevent the LHC from working. Suppose the LHC is set up so well that the only thing that could stop it from running is a direct asteroid hit to Geneva, such that if turning the LHC on would destroy the world, we would observe an asteroid strike to Geneva with probability 1. Let’s say a biggish asteroid hits the Earth about once every thousand years (last one was Tunguska), and that each one affects one one-hundredth of the Earth’s surface (Tunguska was much less, but others could be bigger). That means there’s a 1⁄30 million chance of a big asteroid strike to Geneva each day. If your true probability is 1/3E22, you could try to turn the LHC on, have an asteroid strike Geneva the day before, and still have less than a one in a billion chance that the asteroid was anything other than a random asteroid.
In fact, all of these combined do not equal 3E22, so if the world’s top infallible physicist agreed the LHC would destroy the world and was certified sane by an infallible psychiatrist, AND an asteroid struck Geneva the last time you planned to turn the LHC on, AND you know the world will end the day the LHC is activated, then if your real probability was 3E22 you should now (by my calculations) think that, on balance, there’s about a one in three chance the LHC would destroy the world.
This is why I don’t like using numbers like 3E22 as probabilities.
The last time humanity was almost destroyed was about 80,000 years ago, when a volcanic eruption reduced the human population below 1,000. So say events that can destroy humanity happen on average every hundred thousand years (conservative assumption, right?).
The estimates there range from 2,000 to 20,000 individuals.
The population may not have been significantly bigger before the eruption:
Scientists from the University of Utah in Salt Lake City in the U.S. have calculated that 1.2 million years ago, at a time when our ancestors were spreading through Africa, Europe and Asia, there were probably only around 18,500 individuals capable of breeding (and no more than 26,000).
In fact, all of these combined do not equal 3E22, so if the world’s top infallible physicist agreed the LHC would destroy the world and was certified sane by an infallible psychiatrist, AND an asteroid struck Geneva the last time you planned to turn the LHC on, AND you know the world will end the day the LHC is activated, then if your real probability was 3E22 you should now (by my calculations) think that, on balance, there’s about a one in three chance the LHC would destroy the world.
This is why I don’t like using numbers like 3E22 as probabilities.
Brilliant. Is there any chance I could persuade you to present this as a top level post on the front page? This is a comment I expect to reference when related subjects come up in the future.
I haven’t yet entered this particular discussion, but it is of interest to me, so I hope you won’t mind persisting a bit longer, with a different interlocutor.
This is why I don’t like using numbers like 3E22 as probabilities.
May I ask just what your lower bound is on probability estimates?
I can’t, really, because it’s context dependent. If the question was “What is the probability that a program which selects one atom at random from all those in the universe (and is guaranteed by Omega genuinely random) picks this particular phosphorous atom on here the tip of my finger”, then my probability would be much less than 3E22.
Likewise, “destroy the Earth” is a relatively simple occurrence—it just needs a big enough burst of energy or mass or something. If it’s “What is the probability that the LHC will create a hamster in a tutu on top of Big Ben on noon at Christmas Day singing ‘Greensleeves’ while fighting a lightsaber duel with the ghost of Alexander the Great”, then my probability would again be less than 3E22 (at least before I formed this thought—I don’t know if having said it aloud makes the probability that malevolent aliens will enact it go above 1/3E22 or not).
Thanks for the clarification; that’s quite reasonable.
I’ll note, however, that your own arguments (the world’s greatest physicist certified sane by the world’s greatest psychiatrist...) still apply!
The point being that our “counterintuitiveness detector” shouldn’t get to automatically override calculated probabilities, especially in situations that intuition wasn’t designed to handle.
As for the LHC, it’s worth pointing out that potential benefits also have to be factored into the expected utility calculation, a fact which I don’t think I’ve seen mentioned in the current discussion.
Yvain: [...] “What is the probability that the LHC will create a hamster in a tutu on top of Big Ben on noon at Christmas Day singing ‘Greensleeves’ while fighting a lightsaber duel with the ghost of Alexander the Great”, then my probability would again be less than 3E22 (at least before I formed this thought—I don’t know if having said it aloud makes the probability that malevolent aliens will enact it go above 1/3E22 or not).
komponisto: Thanks for the clarification; that’s quite reasonable.
Or in a slightly different variant of the experiment, if your real probability is 1/3E22, if someone reliably told you that in a year from now you’d assign a probability of 1/3E12, you’d have to conclude it was probably because your rationality was going to break down (assuming the probability of such breakdowns isn’t too extremely low).
Starting this discussion, I gave a probability of one in a million. After reading up on the subject further, I found a physicist who said one in fifty million, and am willing to bow to his superior expertise.
Was there only a one in fifty chance my probability would change this much? This doesn’t seem right, because I knew going in that I knew very little about the subject and if you’d asked me whether I expected my probability to change by a factor of at least fifty, I would have said yes (though of course I couldn’t have predicted in which direction).
It seems to me it would be fine for David to believe with high probability that he would get new evidence that would change his probability to 3E12, as long as he believed it equally possible he’d get new evidence that would change it to 3E32
A 1/3e22 probability means you believe there’s a 1/3e22 chance of the event happening.
If you have, for example a 1/1e9 chance of finding evidence that increases that to 1/3e12, then you have a 1/1e9*1/3e12 chance of the event happening.
Which is 1/3e21.
So, in order to be consistent, you must believe that there is, at most, a 1/1e10 chance of you finding evidence that increases the probability to 1/3e12.
At which point, the probability of losing your rationality is obviously higher.
Yes. Yvain’s 1 in 50 million example, on the other hand, is fine, because the probability went down. In a more extreme example, it could have had a 50-50 chance of going down to 0 (dropping by a factor infinity) as long as there had been a 50-50 chance of it doubling. Conservation of expected evidence.
In several places I’d say you tilt your probability estimates in the most favorable direction to your argument. For example, you underestimate how much evidence the meteorite would give − 1/100th of the earth’s surface destroyed every 1000 years is far too much. There have been 0 humanity-wiping-out events so far, over 1 million-ish years, this does not work out to P=10^-5. In estimating based off of expert opinion you load the intuitive die with “the calculations say” rather than “the physicist says”; calculations are either right or wrong.
I agree that the estimate of 10^-22 is likely too low, but I have a negative reaction to how you’re arguing it.
The post you’re responding to didn’t use 3E22 as a probability. It gave 3E22 as a number of previous experiments.
Now, as the link you cited in this response shows, they’re not necessarily quite identical experiments (although some might result in identical experiments).
you should now (by my calculations) think that, on balance, there’s about a one in three chance the LHC would destroy the world.
So what was your answer to the original question of if the LHC should be switched on? Your citation to Lifeboat is saying they really think it shouldn’t. I presumed you had posted this because you agreed.
That is: you have numbers yourself now. Are those numbers strong enough for you to seriously advocate the LHC should be switched off and kept off?
If “yes”, what are you doing about it? If “no”, then I don’t understand the point of all the above.
I was kinda hoping you wouldn’t ask that. This whole thing came up because I said it was “reasonable” to worry about the LHC, and I stick to that. But the whole thing seems like a Pascal’s Mugging to me, and I don’t have a perfect answer to that class of problem.
I don’t think it should be switched off now, because its failure to destroy the world so far is even better evidence than the cosmic ray argument that it won’t destroy the world the next time it’s used. But if you’d asked before it was turned on? I guess I would agree with Aleksei Riikonen’s point in one of the other LW threads that this is really the sort of thing that could be done just as well after the Singularity.
But I also agree with Eliezer (I could have avoided this entire discussion if I’d just been able to find that post the first time I looked for it when you asked for a citation!) that in reality I wouldn’t lose sleep over it. Basically, I notice I am confused, and my only objection to you was the suggestion that reasonable people couldn’t worry about it, not that I have any great idea how to address the issue myself.
You mean, asking the whole actual real-life question at hand: whether the LHC is too risky to run.
“Is it reasonable to think X?” is only a useful question to consider in relation to X as part of the actual discussion of X. It’s not a useful sort of question in itself until it’s applied to something. Without considering the X itself, it’s a question about philosophy, not about the X. If you’re going to claim something about the LHC, I expect you to be saying something useful about the LHC itself.
Given you appear to regard application as a question you’d rather not have asked, what expected usefulness should I now assign to going through your comments on the subject in close detail, trying to understand them?
(I really am going WHAT? WHAT THE HELL WAS THE ACTUAL POINT OF ALL THAT, THAT WAS WORTH BOTHERING WITH? If you’re going to claim something about the LHC, I expect you to be saying something useful about the LHC itself.)
Yvain was, I suspect, trying to illustrate failures in your thought, rather than in your conclusion.
If you see someone arguing that dogs are mammals because they have tongues, you may choose to correct them, despite agreeing with their conclusion. Especially if you’re on a board related to rationality.
You don’t think an argument that something which you thought was certain is actually confusing is valuable? If an agnostic convinced a fundamentalist that God’s existence was less cut-and-dried obvious than the fundamentalist had always thought, but admitted ey wasn’t really sure about the God question emself, wouldn’t that still be a useful service?
This reads to me as an admission that you were not, nor were you intending to, at any point say anything useful or interesting about the LHC. This suggests that if you want people not to feel like you’re wasting their time and leading them on a merry dance rather than talking about the apparent topic of discussion (which is how I feel now—well and properly trolled. Well done.) then you may want to pick examples where you don’t have to hope no-one ever asks “so what is the point of all this bloviating?”
You asked for a citation for my mention that worrying about the LHC was “reasonable”. I interpreted “reasonable” to mean “there are good arguments for not turning it on”. I am not sure whether I fully believe those arguments and I am confused about how to deal with them, but I do believe there are good arguments and I presented them to you because you asked for them. I didn’t enjoy spending a few hours defending an assertion I made that was tangential to my main point either.
Aside from the whole “ability to think critically about probabilities of existential risk will probably determine the fate of humankind and all other sapient species” thing, no, it doesn’t have any practical implications. But this is a thread about philosophy on a philosophy site, and you asked a philosophical question to a former philosophy student, so I don’t think it’s fair to expect me to anticipate that you wanted to avoid discussions that were purely philosophical.
Seriously, and minus the snark, it’s possible I don’t understand your objections. I promise I was not trying to troll you and I’m sorry if you feel like this has wasted your time.
Sorry, I know I said I’d stop, and I will stop after this, but that 3E22 number is just too interesting to leave alone.
The last time humanity was almost destroyed was about 80,000 years ago, when a volcanic eruption reduced the human population below 1,000. So say events that can destroy humanity happen on average every hundred thousand years (conservative assumption, right?). That means the chance of a humanity-destroying event per year is 1⁄100,000. Say 90% of all humanity destroying events can be predicted with at least one day’s notice by eg asteroid monitoring. This leaves hard-to-detect asteroids, sudden volcanic eruptions, weird things like sudden methane release from the ocean, et cetera. So 1⁄1 million years we get an unexpected humanity destroying event. That means the “background rate” of humanity destroying events is 1⁄300 million days.
Suppose Omega told you, the day before the LHC was switched on, that tomorrow humankind would be destroyed. If 1/3E22 were your true probability, you would say “there’s still vastly less than one in a billion chance the apocalypse has anything to do with the LHC, it must just be a coincidence.” Even if you were the LHC project coordinator, you might not even bother to tell them not to switch on the project, because it wasn’t worth the effort it would take to go to the telephone.
Let’s look at it a different way. Suppose a scientist has a one in a thousand chance of having a psychotic break. Now suppose the world’s top physicist, so brilliant as to be literally infallible as long as he is sane, comes up with new calculations that say the LHC will destroy the world. Suppose you ask the world’s best psychiatrist, who also is literally never wrong, whether the physicist is insane, and she says no. If your probability is truly 1/3E22, then it is more likely that both the physicist and the psychiatrist have simultaneously gone insane than that the physicist is correct; what is more, even if you have no other evidence bearing on the sanity of either, your probability should still be less than one in a trillion that the LHC will destroy the Earth.
There was some discussion in LW a while back on how it might be a prediction of anthropic theory that if the LHC destroys the world, improbable occurences will prevent the LHC from working. Suppose the LHC is set up so well that the only thing that could stop it from running is a direct asteroid hit to Geneva, such that if turning the LHC on would destroy the world, we would observe an asteroid strike to Geneva with probability 1. Let’s say a biggish asteroid hits the Earth about once every thousand years (last one was Tunguska), and that each one affects one one-hundredth of the Earth’s surface (Tunguska was much less, but others could be bigger). That means there’s a 1⁄30 million chance of a big asteroid strike to Geneva each day. If your true probability is 1/3E22, you could try to turn the LHC on, have an asteroid strike Geneva the day before, and still have less than a one in a billion chance that the asteroid was anything other than a random asteroid.
In fact, all of these combined do not equal 3E22, so if the world’s top infallible physicist agreed the LHC would destroy the world and was certified sane by an infallible psychiatrist, AND an asteroid struck Geneva the last time you planned to turn the LHC on, AND you know the world will end the day the LHC is activated, then if your real probability was 3E22 you should now (by my calculations) think that, on balance, there’s about a one in three chance the LHC would destroy the world.
This is why I don’t like using numbers like 3E22 as probabilities.
This seems in conflict with http://en.wikipedia.org/wiki/Toba_catastrophe_theory
The estimates there range from 2,000 to 20,000 individuals.
The population may not have been significantly bigger before the eruption:
http://www.physorg.com/news183278038.html
A volcanic eruption is obviously much less likely to threaten humanity’s existence today than when there were only a handful of us in the first place.
Brilliant. Is there any chance I could persuade you to present this as a top level post on the front page? This is a comment I expect to reference when related subjects come up in the future.
I haven’t yet entered this particular discussion, but it is of interest to me, so I hope you won’t mind persisting a bit longer, with a different interlocutor.
May I ask just what your lower bound is on probability estimates?
I can’t, really, because it’s context dependent. If the question was “What is the probability that a program which selects one atom at random from all those in the universe (and is guaranteed by Omega genuinely random) picks this particular phosphorous atom on here the tip of my finger”, then my probability would be much less than 3E22.
Likewise, “destroy the Earth” is a relatively simple occurrence—it just needs a big enough burst of energy or mass or something. If it’s “What is the probability that the LHC will create a hamster in a tutu on top of Big Ben on noon at Christmas Day singing ‘Greensleeves’ while fighting a lightsaber duel with the ghost of Alexander the Great”, then my probability would again be less than 3E22 (at least before I formed this thought—I don’t know if having said it aloud makes the probability that malevolent aliens will enact it go above 1/3E22 or not).
Thanks for the clarification; that’s quite reasonable.
I’ll note, however, that your own arguments (the world’s greatest physicist certified sane by the world’s greatest psychiatrist...) still apply!
The point being that our “counterintuitiveness detector” shouldn’t get to automatically override calculated probabilities, especially in situations that intuition wasn’t designed to handle.
As for the LHC, it’s worth pointing out that potential benefits also have to be factored into the expected utility calculation, a fact which I don’t think I’ve seen mentioned in the current discussion.
Yvain: [...] “What is the probability that the LHC will create a hamster in a tutu on top of Big Ben on noon at Christmas Day singing ‘Greensleeves’ while fighting a lightsaber duel with the ghost of Alexander the Great”, then my probability would again be less than 3E22 (at least before I formed this thought—I don’t know if having said it aloud makes the probability that malevolent aliens will enact it go above 1/3E22 or not).
komponisto: Thanks for the clarification; that’s quite reasonable.
^Awesome :-)
Or in a slightly different variant of the experiment, if your real probability is 1/3E22, if someone reliably told you that in a year from now you’d assign a probability of 1/3E12, you’d have to conclude it was probably because your rationality was going to break down (assuming the probability of such breakdowns isn’t too extremely low).
Okay, now I’m confused, or misunderstanding you.
Starting this discussion, I gave a probability of one in a million. After reading up on the subject further, I found a physicist who said one in fifty million, and am willing to bow to his superior expertise.
Was there only a one in fifty chance my probability would change this much? This doesn’t seem right, because I knew going in that I knew very little about the subject and if you’d asked me whether I expected my probability to change by a factor of at least fifty, I would have said yes (though of course I couldn’t have predicted in which direction).
It seems to me it would be fine for David to believe with high probability that he would get new evidence that would change his probability to 3E12, as long as he believed it equally possible he’d get new evidence that would change it to 3E32
A 1/3e22 probability means you believe there’s a 1/3e22 chance of the event happening.
If you have, for example a 1/1e9 chance of finding evidence that increases that to 1/3e12, then you have a 1/1e9*1/3e12 chance of the event happening.
Which is 1/3e21.
So, in order to be consistent, you must believe that there is, at most, a 1/1e10 chance of you finding evidence that increases the probability to 1/3e12.
At which point, the probability of losing your rationality is obviously higher.
Yes. Yvain’s 1 in 50 million example, on the other hand, is fine, because the probability went down. In a more extreme example, it could have had a 50-50 chance of going down to 0 (dropping by a factor infinity) as long as there had been a 50-50 chance of it doubling. Conservation of expected evidence.
On the one hand, everything you say would be true, if we were assigning consistent probabilities.
On the other hand, I’ve never been able to assign consistent probabilities over the LHC and knowing this hasn’t helped me either.
In several places I’d say you tilt your probability estimates in the most favorable direction to your argument. For example, you underestimate how much evidence the meteorite would give − 1/100th of the earth’s surface destroyed every 1000 years is far too much. There have been 0 humanity-wiping-out events so far, over 1 million-ish years, this does not work out to P=10^-5. In estimating based off of expert opinion you load the intuitive die with “the calculations say” rather than “the physicist says”; calculations are either right or wrong.
I agree that the estimate of 10^-22 is likely too low, but I have a negative reaction to how you’re arguing it.
The post you’re responding to didn’t use 3E22 as a probability. It gave 3E22 as a number of previous experiments.
Now, as the link you cited in this response shows, they’re not necessarily quite identical experiments (although some might result in identical experiments).
But you’re attacking an error which was not made.
“1 in 3E22” was surely a probability. Yvain made a typo at the end of his comment.
Ah, I was mistaken. For some reason I didn’t notice the link.
Are there any alternative colour-schemes for this site? Links seem to show up poorly on blue-backgrounded posts.
I would like you to consider turning your comment into a top-level post. Thanks.
So what was your answer to the original question of if the LHC should be switched on? Your citation to Lifeboat is saying they really think it shouldn’t. I presumed you had posted this because you agreed.
That is: you have numbers yourself now. Are those numbers strong enough for you to seriously advocate the LHC should be switched off and kept off?
If “yes”, what are you doing about it? If “no”, then I don’t understand the point of all the above.
I was kinda hoping you wouldn’t ask that. This whole thing came up because I said it was “reasonable” to worry about the LHC, and I stick to that. But the whole thing seems like a Pascal’s Mugging to me, and I don’t have a perfect answer to that class of problem.
I don’t think it should be switched off now, because its failure to destroy the world so far is even better evidence than the cosmic ray argument that it won’t destroy the world the next time it’s used. But if you’d asked before it was turned on? I guess I would agree with Aleksei Riikonen’s point in one of the other LW threads that this is really the sort of thing that could be done just as well after the Singularity.
But I also agree with Eliezer (I could have avoided this entire discussion if I’d just been able to find that post the first time I looked for it when you asked for a citation!) that in reality I wouldn’t lose sleep over it. Basically, I notice I am confused, and my only objection to you was the suggestion that reasonable people couldn’t worry about it, not that I have any great idea how to address the issue myself.
You mean, asking the whole actual real-life question at hand: whether the LHC is too risky to run.
“Is it reasonable to think X?” is only a useful question to consider in relation to X as part of the actual discussion of X. It’s not a useful sort of question in itself until it’s applied to something. Without considering the X itself, it’s a question about philosophy, not about the X. If you’re going to claim something about the LHC, I expect you to be saying something useful about the LHC itself.
Given you appear to regard application as a question you’d rather not have asked, what expected usefulness should I now assign to going through your comments on the subject in close detail, trying to understand them?
(I really am going WHAT? WHAT THE HELL WAS THE ACTUAL POINT OF ALL THAT, THAT WAS WORTH BOTHERING WITH? If you’re going to claim something about the LHC, I expect you to be saying something useful about the LHC itself.)
Yvain was, I suspect, trying to illustrate failures in your thought, rather than in your conclusion.
If you see someone arguing that dogs are mammals because they have tongues, you may choose to correct them, despite agreeing with their conclusion. Especially if you’re on a board related to rationality.
You don’t think an argument that something which you thought was certain is actually confusing is valuable? If an agnostic convinced a fundamentalist that God’s existence was less cut-and-dried obvious than the fundamentalist had always thought, but admitted ey wasn’t really sure about the God question emself, wouldn’t that still be a useful service?
This reads to me as an admission that you were not, nor were you intending to, at any point say anything useful or interesting about the LHC. This suggests that if you want people not to feel like you’re wasting their time and leading them on a merry dance rather than talking about the apparent topic of discussion (which is how I feel now—well and properly trolled. Well done.) then you may want to pick examples where you don’t have to hope no-one ever asks “so what is the point of all this bloviating?”
You asked for a citation for my mention that worrying about the LHC was “reasonable”. I interpreted “reasonable” to mean “there are good arguments for not turning it on”. I am not sure whether I fully believe those arguments and I am confused about how to deal with them, but I do believe there are good arguments and I presented them to you because you asked for them. I didn’t enjoy spending a few hours defending an assertion I made that was tangential to my main point either.
Aside from the whole “ability to think critically about probabilities of existential risk will probably determine the fate of humankind and all other sapient species” thing, no, it doesn’t have any practical implications. But this is a thread about philosophy on a philosophy site, and you asked a philosophical question to a former philosophy student, so I don’t think it’s fair to expect me to anticipate that you wanted to avoid discussions that were purely philosophical.
Seriously, and minus the snark, it’s possible I don’t understand your objections. I promise I was not trying to troll you and I’m sorry if you feel like this has wasted your time.