I think few here would give an estimate higher than 95% for the probability that humanity will survive the next 100 years; plenty might put a figure less than 50% on it.
For the record I would put it at levels overwhelmingly higher than 95%. More like 99.999%.
You can’t get away with having such extreme probabilities when a bunch of smart and rational people disagree. There are reasons why the whole Aumann agreement thing doesn’t work perfectly in real life, but this is an extreme failure.
If a bunch of people on LW think it’s only 50% likely and you think theres only a 0.1% chance that they’re right and you’re wrong (which is already ridiculously low) it still brings your probability estimate down to around 99.95. This is a 50 fold increase in the probability that the world is going to end over what you stated. Either you have some magic information that you haven’t shared, or you’re hugely overconfident.
You cannot selectively apply Aumann agreement. If you want to count tiny bunch of people who believe in AI foom, you must also take into account 7 billion people, many of them really smart, who definitely don’t.
I don’t have this problem, as I don’t really believe that using Aumann agreement is useful with real humans.
Or you could count my awareness of insider overconfidence as magic information:
Large groups of smart people are frequently wrong about the future, and overwhelmingly so about the non-immediate future. 0.1% may be low but it’s not ridiculously so.
(Also “they’re right and you’re wrong” is redundant. This has nothing to do with any set of scenario probabilities being “right”. And any debate of “p=.9” “no, p=.1″ is essentially silly because it misunderstands both the meaning of probability as a function of knowledge and our ability to create models which give meaningfully-accurate probabilities.)
And any debate of “p=.9” “no, p=.1″ is essentially silly because it misunderstands both the meaning of probability as a function of knowledge and our ability to create models which give meaningfully-accurate probabilities.
Subjective probability is (in particular) a tool for elicitation of model parameters from expert human gut-feelings, which you can then use to find further probabilities and align them with other gut-feelings and decisions, gaining precision from redundancy and removing inconsistencies. The subjective probabilities don’t promise to immediately align with physical frequencies, even where the notion makes sense.
It is a well-studied and useful process, you’d need a substantially more constructive reference than “it’s silly” (or you could just seek a reasonable interpretation).
Do you have a link for a top-level post that puts this kind of caveat on probability assignments? Personally, I think that if most people here understood it that way, they’d use more qualified language when talking about subjective probability. I also think that developing and standardizing such qualified language would be a useful project.
It is the sense in which the term “probability” is generally understood on OB/LW, with varying levels of comprehension by specific individuals. There are many posts on probability, both as an imprecise tool and an ideal (but subjective) construction. They should probably be organized in the Bayesian probability article on the wiki. In the meantime, you are welcome to look for references in the Overcoming Bias archives.
I myself would be disappointed if over half of LW put the probability of a single biological human (not an upload, not a reconstruction—an actual descendent with the appropriate number of ancestors alive today) alive in 100 years under 95%. I would consider that to be a gross instance of all kinds of biases. I’m not going to argue about scenarios, here, just point out that there any scenario which tends inevitably to wipe out humanity within one lifetime is totally unimaginable. That doesn’t mean implausible, but it does mean improbable.
Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events (“something that’s never happened before which causes X” where X is a defined consequence such as humanity’s extinction) to within even an order of magnitude for p/(1-p). I doubt anybody can get even to within two orders of magnitude consistently. (I also doubt that this hypothesis of mine will be clearly decidable within the next 20 years, so I’m not particularly inclined to listen to philosophical arguments from people who’d like to discard it.)
What I’m saying is, we should stop trying to put numbers on this without big error bars. And I’ve yet to see anybody propose an intelligent way to deal with probabilities like 10^(-6 +/- 4); just meta-averaging it over the distribution of possible probabilities, to come up with something like 10^-3 seems to be discarding data and to lead to problems. However, that’s the kind of probability I’d put on this lemma. (“Earth made uninhabitable by normal cosmic event and rescue plans fail” would probably put a floor somewhere above 10^-22 per year.)
“The chance we’re all wrong about something totally unprecedented has got to be less than 99.9%” is total hubris. Yes, totally unprecedented things happen every day. But telling yourselves stories about AGI and foom does not make these stories likely.
This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences. But if you’re all getting that carried away, then Less Wrong should just be renamed More Wrong.
Oh, also, I’d accept that the risk of humanity being seriously hosed within 100 years, or extinct within 1000 years, is significant—say, 10^(-3 +/- 4) which meta-averages to something like 15%.
(“Seriously hosed” means gigadeath events, total enslavement, or the like. Note that we’re already moderately hosed and always have been, but that seriously hosed is still distinguishable.)
I myself would be disappointed if over half of LW put the probability of a single biological human alive in 100 years under 95%.
This is an assertion of your confidence in extinction risk being below 5%.
Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events [...] This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences.
Not understanding a phenomenon, being unable to estimate its probability, doesn’t give you an ability to place its probability below a strict bound. Your assertion of confidence contradicts your assertion of confusion.
I have confidence that nobody here has secret information that makes human extinction much more likely—because almost no information which currently exists could have more than a marginal bearing on a result which, if likely, is a result of human (that is, intelligent) interaction. Therefore I have confidence that the difference in estimates is largely not due to information, but to models. I have confidence that inductive models—say, “how often does a random species survive any hundred year period, correcting for initial population” give answers over 95% which should be considered the default. Therefore, I have confidence that a community of people who generally give lower estimates is subject to some biases (such as narrative bias).
Doesn’t mean LW’s wrong and I’m right. But to believe that human extinction within a century is likely clearly puts LW in the minority of humanity in your beliefs—even in the minority of rational atheists. And the fact that there is substantial agreement within the LW community on this, when uncertainty is clearly so high that orders of magnitude of disagreement are possible, makes me suspect bias.
Also, I find it funny that people will argue passionately over estimates that differ in log(p/q) from −1 to +1 (~10% to ~90%), but couldn’t care less over the difference from say −9 to −7 (.0001% vs .000001%) or 7 to 9. This is in one sense the right attitude for people who think they can do something about it, but it ends up biasing numbers towards log(p/q)=0 [ie 50%], since you are more likely to get argument from somebody who has an estimate on the other side of 50% as yours is.
The fact that we believe something unusual is only weak evidence for the validity of that unusual belief, you are right on that. And given the hypothesis that we are wrong, which is dominant while all you have is the observation that we believe something unusual, you can draw a conclusion that we are wrong because of some systematic error of judgment that makes most here to claim the unusual belief.
To move past this point, you have to consider the specific arguments, and decide for yourself whether to accept them.
Most of the beliefs people can hold intuitively are about 50% in certainty. The beliefs far away from this point aren’t useful as primitive concepts, classifying the possible events on one side or the other, as most everything is only on one side, and human mind can’t keep track of their levels of certainty. New concepts get constructed, that are more native to human mind and express the high-certainty concepts in question only in combinations, or that are supported by non-intuitive procedures for processing levels of certainty. But if the argument is dependent on use of intuition, you aren’t always capable of moving towards certainty, so you remain in doubt. This is the case for unknown unknowns, in particular.
You clipped out “to within an order of magnitude”. I stated that my best-guess probability for human extinction within a century was 10^(-6 +/- 4). This is a huge confusion − 9 orders of magnitude on the probability—yet still means that I have over 80% confidence that the probability is under 10^-2. There is no contradiction here.
(It also means that, despite believing that extinction is probably one-in-a-million, I should treat it as more like one-in-a-thousand, because averaging over the meta-probability distribution naturally weights the high end. It would be a pity if this effect, of uncertainty inflating small probabilities, resulted in social feedback. When you hear me say “we should treat it as a .1% risk”, I am implicitly stating that all models I can credit give a significantly lower risk. If your best model’s risk-estimate is .01%, I am actually telling you that I think your model overestimates the risk.)
So, where did you get those numbers from? 10^-6? 10^-2? Why not, say, 1-10^-6 instead? Gut feeling again, and that’s inevitable. You either name a number, or make decisions without the help of even this feeble model, choosing directly. From what people on this site know, they believe differently from you.
I have one of the lowest estimates, 30% for not killing off 90% of the population by 2100. Most of it comes from Unfriendly AI, with estimate of 50% of AGI foom by 2070, or 70% by 2100 (expectation of relatively low-hanging fruit, it levels off as time goes on) if nothing goes wrong with the world, 3⁄4 of that to Unfriendly AI, given my understanding of how hard it is to find the right answer from many efficient world-eating possibilities, and human irrationality, making it likely that the person to invent the first mind won’t think about the consequences. That’s already 55% total extinction risk, add some more for biological (at least, human-inhabiting) weapons, such as an engineered pandemic (not total extinction, but easily 90%), and new possible goodies the future has to offer. It’ll only get worse until it gets better. On second thought, I should lower my confidence from these explicit models, they seem too much like planning. Make that 50%.
When you speak of “the probability”, what information do you mean that to take into account and what information do you mean that not to take into account? What things does a rational agent need to know for the agent’s subjective probability to become equal to the probability? (Not a rhetorical question.)
“the probability” means something like the following: take a random selection of universe-histories starting with a state consistent with my/your observable past and proceeding 100 years forward, with no uncaused discontinuities in the laws of physics, to a compact portion of a wave function (that is “one quantum universe”, modulo quantum computers which are turned on). What portion of those universes satisfy the given end state?
Yes, I’m doing what I can to duck the measure problem of universes, sorry. And of course this is underdefined and unobservable. Yet it contains the basic elements: both knowledge and uncertainty about the current state of the universe, and definite laws of physics, assumed to independently exist, which strongly constrain the possible outcomes from a given initial state.
On a more practical level, it seems to be the case that, given enough information and study of a class of situations, post-hoc polynomial-computable models which use non-determinism to model the effects of details which have been abstracted out, can provide predictions about some salient aspects of that situation under certain constraints. For instance, the statement “42% of technological societies of intelligent biological agents with access to fissile materiels destroy themselves in a nuclear holocaust” could, subject to the definitions of terms that would be necessary to build a useful model, be a true or false statement.
Note that this allows for three completely different kinds of uncertainty: uncertainty about the appropriate model(s), uncertainty about the correct parameters for those model(s), and uncertainty inherent within a given model. In almost all questions involving predicting nonlinear interactions of intelligent agents, the first kind of uncertainty currently dominates. That is the kind of uncertainty I’m trying (and of course failing) to capture with the error bar in the exponent. Still, I think my failure, which at least acknowledges the overwhelming probability that I’m wrong (albeit in a limited sense) is better than a form of estimation that presents an estimate garnered from a clearly limited set of models as a final one.
In other words: I’m probably wrong. You’re probably wrong too. Since giving an estimate under 95% requires certain specific extrapolations, while almost any induction points to estimates over 95%, I would expect most rational people to arrive at an estimate over 95%, and would suspect any community with the reverse situation to be subject to biases (of which selection bias is the most innocuous). This suspicion would not apply when dealing with individuals.
To get the right answer, you need to make a honest effort to construct a model which is an unbiased composite of evidence-based models. Metaphorical reasoning is permitted as weak evidence, but cannot be the only sort of evidence.
And you also need to be lucky. I mean, unless you have the resources to fully simulate universes, you can never know that you have the right answer. But the process above, iterated, will tend to improve your answer.
Without even going into different specific risks, you should beware the conjunction fallacy (or, more accurately, its flip side) when assigning such a high probability. A lack of details tends to depress estimates of an event that could occur as a result of many different causes, since if you aren’t visualizing a full scenario it’s tempting to say there’s no way for it to occur.
You’re effectively asserting that not only are all of the proposed risks to humanity’s survival this minuscule in aggregate, but that you’re also better than 99.9% confident that there won’t be invented or discovered anything else that presents a plausible existential threat. How do you arrive at such confidence of that?
Then, as a necessary condition (leaving other risks from the discussion for the moment), you either don’t believe in the feasibility of AGI, or you believe in the objective morality, which any AGI will “discover”. Which one is that?
I don’t believe in feasibility of any scenario like AGI foom.
First, I fail to see how anybody taking an outside view on AI research—which is a clear instance of class of sciences with extraordinary claims and very long history of failure to deliver in spite of unusually adequate funding—can think otherwise—to me it all seems like extreme case of insider bias to assign non-negligible probabilities to scenarios like that. Virtually none sciences with this characteristics delivered what they promised (even if they delivered something useful and vaguely related).
Even if AGI happens, it is extraordinarily unlikely it will be any kind of foom, again based on outside view argument that virtually none of disruptive technologies were ever foom-like.
Both extraordinarily unlikely events would have to occur before we would be exposed to risk of AGI-caused destruction of humanity, which even in this case is far from certain.
It’s not reverse stupidity—it’s “reference class forecasting”, which is a more specific instance of our generic “outside view” concept. I gather data about AI research as an instance, look at other cases with similar characteristics (hyped overpromised and underdelivered over a very long time span) and estimate based on that. It is proven to work better than inside view of estimating based on details of a particular case.
I agree that reference class forecasting is reasonable here. I disagree that you can get anything like the 99.999% probability you claim from applying reference class forecasting to AI projects. Since rare events happen, well, rarely, it would take an exceedingly large data-set before an “outside view” or frequency-based analysis would imply that our actual expected rate should be placed as low as your stated 0.001%. (If I flip a coin with unknown weighting 20 times, and get no heads, I should conclude that heads are probably rare, but my notion of “rare” here should be on the order of 1 in 20, not of 1 in 100,000.)
With more precision: let’s say that there’s a “true probability”, p, that any given project’s “AI will be created by us” claim is correct. And let’s model p as being identical for all projects and times. Then, if we assume a uniform prior over p, and if n AI projects that have been tried to date have failed to deliver, we should assign a probability of ((1+n)/n+2) to the chance that the next project from which AI is forecast will also fail to deliver. (You can work this out by an integral, or just plug into Laplace’s rule of succession).
If people have been forecasting AI since about 1950, and if the rate of forecasts or AI projects per decade has been more or less unchanged, the above reference class forecasting model leaves us with something like a 1/[number of decades since 1950 + 2] = 1⁄8 probability of some “our project will make AI” forecast being correct in the next decade.
That said, I still take issue with reference class forecasting as support for this statement:
I don’t believe in feasibility of any scenario like AGI foom.
Considering that the general question “is the foom scenario feasible?” doesn’t have any concrete timelines attached to it, the speed and direction of AI research don’t bear too heavily on it. All you can say about it based on reference class forecasting is that it’s a long way away if it’s both possible and requires much AI research progress.
Even if AGI happens, it is extraordinarily unlikely it will be any kind of foom, again based on outside view argument that virtually none of disruptive technologies were ever foom-like.
I’m not sure “disruptive technology” is the obvious category for AGI. The term basically dereferences to “engineered human-level intelligence”, easily suggesting comparisons to various humans, hominids, primates, etc.
I don’t know if inside view forecasting can ever be more reliable than outside view forecasting. It seems that insiders as a general and very robust rule tend to be strongly overconfident, and see all kinds of reason why their particular instance is different and will have better outcome than the reference class.
I don’t know if inside view forecasting can ever be more reliable than outside view forecasting. It seems that insiders as a general and very robust rule tend to be strongly overconfident, and see all kinds of reason why their particular instance is different and will have better outcome than the reference class.
Try applying that to physics, engineering, biology, or any other technical field. In many cases, the outside view doesn’t stand a chance.
For the record I would put it at levels overwhelmingly higher than 95%. More like 99.999%.
You can’t get away with having such extreme probabilities when a bunch of smart and rational people disagree. There are reasons why the whole Aumann agreement thing doesn’t work perfectly in real life, but this is an extreme failure.
If a bunch of people on LW think it’s only 50% likely and you think theres only a 0.1% chance that they’re right and you’re wrong (which is already ridiculously low) it still brings your probability estimate down to around 99.95. This is a 50 fold increase in the probability that the world is going to end over what you stated. Either you have some magic information that you haven’t shared, or you’re hugely overconfident.
http://lesswrong.com/lw/9x/metauncertainty/ http://lesswrong.com/lw/3j/rationality_cryonics_and_pascals_wager/69t#comments
You cannot selectively apply Aumann agreement. If you want to count tiny bunch of people who believe in AI foom, you must also take into account 7 billion people, many of them really smart, who definitely don’t.
I don’t have this problem, as I don’t really believe that using Aumann agreement is useful with real humans.
Or you could count my awareness of insider overconfidence as magic information:
http://www.overcomingbias.com/2007/07/beware-the-insi.html
This is Less Wrong we’re talking about. Insider overconfidence isn’t “magic information”.
See my top level post for a full response.
Large groups of smart people are frequently wrong about the future, and overwhelmingly so about the non-immediate future. 0.1% may be low but it’s not ridiculously so.
(Also “they’re right and you’re wrong” is redundant. This has nothing to do with any set of scenario probabilities being “right”. And any debate of “p=.9” “no, p=.1″ is essentially silly because it misunderstands both the meaning of probability as a function of knowledge and our ability to create models which give meaningfully-accurate probabilities.)
Subjective probability is (in particular) a tool for elicitation of model parameters from expert human gut-feelings, which you can then use to find further probabilities and align them with other gut-feelings and decisions, gaining precision from redundancy and removing inconsistencies. The subjective probabilities don’t promise to immediately align with physical frequencies, even where the notion makes sense.
It is a well-studied and useful process, you’d need a substantially more constructive reference than “it’s silly” (or you could just seek a reasonable interpretation).
As you explain it, it’s not silly.
Do you have a link for a top-level post that puts this kind of caveat on probability assignments? Personally, I think that if most people here understood it that way, they’d use more qualified language when talking about subjective probability. I also think that developing and standardizing such qualified language would be a useful project.
It is the sense in which the term “probability” is generally understood on OB/LW, with varying levels of comprehension by specific individuals. There are many posts on probability, both as an imprecise tool and an ideal (but subjective) construction. They should probably be organized in the Bayesian probability article on the wiki. In the meantime, you are welcome to look for references in the Overcoming Bias archives.
You may be interested in the following two posts, related to this discussion:
Probability is in the Mind
When (Not) To Use Probabilities
I myself would be disappointed if over half of LW put the probability of a single biological human (not an upload, not a reconstruction—an actual descendent with the appropriate number of ancestors alive today) alive in 100 years under 95%. I would consider that to be a gross instance of all kinds of biases. I’m not going to argue about scenarios, here, just point out that there any scenario which tends inevitably to wipe out humanity within one lifetime is totally unimaginable. That doesn’t mean implausible, but it does mean improbable.
Personally, I do not believe that any person, group of people, or human-built model to date can consistently predict the probability of defined classes of black-swan events (“something that’s never happened before which causes X” where X is a defined consequence such as humanity’s extinction) to within even an order of magnitude for p/(1-p). I doubt anybody can get even to within two orders of magnitude consistently. (I also doubt that this hypothesis of mine will be clearly decidable within the next 20 years, so I’m not particularly inclined to listen to philosophical arguments from people who’d like to discard it.)
What I’m saying is, we should stop trying to put numbers on this without big error bars. And I’ve yet to see anybody propose an intelligent way to deal with probabilities like 10^(-6 +/- 4); just meta-averaging it over the distribution of possible probabilities, to come up with something like 10^-3 seems to be discarding data and to lead to problems. However, that’s the kind of probability I’d put on this lemma. (“Earth made uninhabitable by normal cosmic event and rescue plans fail” would probably put a floor somewhere above 10^-22 per year.)
“The chance we’re all wrong about something totally unprecedented has got to be less than 99.9%” is total hubris. Yes, totally unprecedented things happen every day. But telling yourselves stories about AGI and foom does not make these stories likely.
This is not, by the way, an argument to ignore existential risk. Even at the 10^-6 (or, averaged over meta-probabilities, 10^-3) level which I estimated, it is clearly worth thinking about, given the consequences. But if you’re all getting that carried away, then Less Wrong should just be renamed More Wrong.
Oh, also, I’d accept that the risk of humanity being seriously hosed within 100 years, or extinct within 1000 years, is significant—say, 10^(-3 +/- 4) which meta-averages to something like 15%.
(“Seriously hosed” means gigadeath events, total enslavement, or the like. Note that we’re already moderately hosed and always have been, but that seriously hosed is still distinguishable.)
This is an assertion of your confidence in extinction risk being below 5%.
Not understanding a phenomenon, being unable to estimate its probability, doesn’t give you an ability to place its probability below a strict bound. Your assertion of confidence contradicts your assertion of confusion.
I have confidence that nobody here has secret information that makes human extinction much more likely—because almost no information which currently exists could have more than a marginal bearing on a result which, if likely, is a result of human (that is, intelligent) interaction. Therefore I have confidence that the difference in estimates is largely not due to information, but to models. I have confidence that inductive models—say, “how often does a random species survive any hundred year period, correcting for initial population” give answers over 95% which should be considered the default. Therefore, I have confidence that a community of people who generally give lower estimates is subject to some biases (such as narrative bias).
Doesn’t mean LW’s wrong and I’m right. But to believe that human extinction within a century is likely clearly puts LW in the minority of humanity in your beliefs—even in the minority of rational atheists. And the fact that there is substantial agreement within the LW community on this, when uncertainty is clearly so high that orders of magnitude of disagreement are possible, makes me suspect bias.
Also, I find it funny that people will argue passionately over estimates that differ in log(p/q) from −1 to +1 (~10% to ~90%), but couldn’t care less over the difference from say −9 to −7 (.0001% vs .000001%) or 7 to 9. This is in one sense the right attitude for people who think they can do something about it, but it ends up biasing numbers towards log(p/q)=0 [ie 50%], since you are more likely to get argument from somebody who has an estimate on the other side of 50% as yours is.
The fact that we believe something unusual is only weak evidence for the validity of that unusual belief, you are right on that. And given the hypothesis that we are wrong, which is dominant while all you have is the observation that we believe something unusual, you can draw a conclusion that we are wrong because of some systematic error of judgment that makes most here to claim the unusual belief.
To move past this point, you have to consider the specific arguments, and decide for yourself whether to accept them.
Most of the beliefs people can hold intuitively are about 50% in certainty. The beliefs far away from this point aren’t useful as primitive concepts, classifying the possible events on one side or the other, as most everything is only on one side, and human mind can’t keep track of their levels of certainty. New concepts get constructed, that are more native to human mind and express the high-certainty concepts in question only in combinations, or that are supported by non-intuitive procedures for processing levels of certainty. But if the argument is dependent on use of intuition, you aren’t always capable of moving towards certainty, so you remain in doubt. This is the case for unknown unknowns, in particular.
You clipped out “to within an order of magnitude”. I stated that my best-guess probability for human extinction within a century was 10^(-6 +/- 4). This is a huge confusion − 9 orders of magnitude on the probability—yet still means that I have over 80% confidence that the probability is under 10^-2. There is no contradiction here.
(It also means that, despite believing that extinction is probably one-in-a-million, I should treat it as more like one-in-a-thousand, because averaging over the meta-probability distribution naturally weights the high end. It would be a pity if this effect, of uncertainty inflating small probabilities, resulted in social feedback. When you hear me say “we should treat it as a .1% risk”, I am implicitly stating that all models I can credit give a significantly lower risk. If your best model’s risk-estimate is .01%, I am actually telling you that I think your model overestimates the risk.)
So, where did you get those numbers from? 10^-6? 10^-2? Why not, say, 1-10^-6 instead? Gut feeling again, and that’s inevitable. You either name a number, or make decisions without the help of even this feeble model, choosing directly. From what people on this site know, they believe differently from you.
I have one of the lowest estimates, 30% for not killing off 90% of the population by 2100. Most of it comes from Unfriendly AI, with estimate of 50% of AGI foom by 2070, or 70% by 2100 (expectation of relatively low-hanging fruit, it levels off as time goes on) if nothing goes wrong with the world, 3⁄4 of that to Unfriendly AI, given my understanding of how hard it is to find the right answer from many efficient world-eating possibilities, and human irrationality, making it likely that the person to invent the first mind won’t think about the consequences. That’s already 55% total extinction risk, add some more for biological (at least, human-inhabiting) weapons, such as an engineered pandemic (not total extinction, but easily 90%), and new possible goodies the future has to offer. It’ll only get worse until it gets better. On second thought, I should lower my confidence from these explicit models, they seem too much like planning. Make that 50%.
When you speak of “the probability”, what information do you mean that to take into account and what information do you mean that not to take into account? What things does a rational agent need to know for the agent’s subjective probability to become equal to the probability? (Not a rhetorical question.)
“the probability” means something like the following: take a random selection of universe-histories starting with a state consistent with my/your observable past and proceeding 100 years forward, with no uncaused discontinuities in the laws of physics, to a compact portion of a wave function (that is “one quantum universe”, modulo quantum computers which are turned on). What portion of those universes satisfy the given end state?
Yes, I’m doing what I can to duck the measure problem of universes, sorry. And of course this is underdefined and unobservable. Yet it contains the basic elements: both knowledge and uncertainty about the current state of the universe, and definite laws of physics, assumed to independently exist, which strongly constrain the possible outcomes from a given initial state.
On a more practical level, it seems to be the case that, given enough information and study of a class of situations, post-hoc polynomial-computable models which use non-determinism to model the effects of details which have been abstracted out, can provide predictions about some salient aspects of that situation under certain constraints. For instance, the statement “42% of technological societies of intelligent biological agents with access to fissile materiels destroy themselves in a nuclear holocaust” could, subject to the definitions of terms that would be necessary to build a useful model, be a true or false statement.
Note that this allows for three completely different kinds of uncertainty: uncertainty about the appropriate model(s), uncertainty about the correct parameters for those model(s), and uncertainty inherent within a given model. In almost all questions involving predicting nonlinear interactions of intelligent agents, the first kind of uncertainty currently dominates. That is the kind of uncertainty I’m trying (and of course failing) to capture with the error bar in the exponent. Still, I think my failure, which at least acknowledges the overwhelming probability that I’m wrong (albeit in a limited sense) is better than a form of estimation that presents an estimate garnered from a clearly limited set of models as a final one.
In other words: I’m probably wrong. You’re probably wrong too. Since giving an estimate under 95% requires certain specific extrapolations, while almost any induction points to estimates over 95%, I would expect most rational people to arrive at an estimate over 95%, and would suspect any community with the reverse situation to be subject to biases (of which selection bias is the most innocuous). This suspicion would not apply when dealing with individuals.
See the posts “Priors as Mathematical Objects”, “Probability is Subjectively Objective” linked from the Priors wiki article.
To get the right answer, you need to make a honest effort to construct a model which is an unbiased composite of evidence-based models. Metaphorical reasoning is permitted as weak evidence, but cannot be the only sort of evidence.
And you also need to be lucky. I mean, unless you have the resources to fully simulate universes, you can never know that you have the right answer. But the process above, iterated, will tend to improve your answer.
Without even going into different specific risks, you should beware the conjunction fallacy (or, more accurately, its flip side) when assigning such a high probability. A lack of details tends to depress estimates of an event that could occur as a result of many different causes, since if you aren’t visualizing a full scenario it’s tempting to say there’s no way for it to occur.
You’re effectively asserting that not only are all of the proposed risks to humanity’s survival this minuscule in aggregate, but that you’re also better than 99.9% confident that there won’t be invented or discovered anything else that presents a plausible existential threat. How do you arrive at such confidence of that?
Then, as a necessary condition (leaving other risks from the discussion for the moment), you either don’t believe in the feasibility of AGI, or you believe in the objective morality, which any AGI will “discover”. Which one is that?
I don’t believe in feasibility of any scenario like AGI foom.
First, I fail to see how anybody taking an outside view on AI research—which is a clear instance of class of sciences with extraordinary claims and very long history of failure to deliver in spite of unusually adequate funding—can think otherwise—to me it all seems like extreme case of insider bias to assign non-negligible probabilities to scenarios like that. Virtually none sciences with this characteristics delivered what they promised (even if they delivered something useful and vaguely related).
Even if AGI happens, it is extraordinarily unlikely it will be any kind of foom, again based on outside view argument that virtually none of disruptive technologies were ever foom-like.
Both extraordinarily unlikely events would have to occur before we would be exposed to risk of AGI-caused destruction of humanity, which even in this case is far from certain.
It seems like you’re reversing stupidity here. What correlation does a failed prediction have with the future?
It’s not reverse stupidity—it’s “reference class forecasting”, which is a more specific instance of our generic “outside view” concept. I gather data about AI research as an instance, look at other cases with similar characteristics (hyped overpromised and underdelivered over a very long time span) and estimate based on that. It is proven to work better than inside view of estimating based on details of a particular case.
http://en.wikipedia.org/wiki/Reference_class_forecasting
I agree that reference class forecasting is reasonable here. I disagree that you can get anything like the 99.999% probability you claim from applying reference class forecasting to AI projects. Since rare events happen, well, rarely, it would take an exceedingly large data-set before an “outside view” or frequency-based analysis would imply that our actual expected rate should be placed as low as your stated 0.001%. (If I flip a coin with unknown weighting 20 times, and get no heads, I should conclude that heads are probably rare, but my notion of “rare” here should be on the order of 1 in 20, not of 1 in 100,000.)
With more precision: let’s say that there’s a “true probability”, p, that any given project’s “AI will be created by us” claim is correct. And let’s model p as being identical for all projects and times. Then, if we assume a uniform prior over p, and if n AI projects that have been tried to date have failed to deliver, we should assign a probability of ((1+n)/n+2) to the chance that the next project from which AI is forecast will also fail to deliver. (You can work this out by an integral, or just plug into Laplace’s rule of succession).
If people have been forecasting AI since about 1950, and if the rate of forecasts or AI projects per decade has been more or less unchanged, the above reference class forecasting model leaves us with something like a 1/[number of decades since 1950 + 2] = 1⁄8 probability of some “our project will make AI” forecast being correct in the next decade.
Oops. You’re totally right.
That said, I still take issue with reference class forecasting as support for this statement:
Considering that the general question “is the foom scenario feasible?” doesn’t have any concrete timelines attached to it, the speed and direction of AI research don’t bear too heavily on it. All you can say about it based on reference class forecasting is that it’s a long way away if it’s both possible and requires much AI research progress.
I’m not sure “disruptive technology” is the obvious category for AGI. The term basically dereferences to “engineered human-level intelligence”, easily suggesting comparisons to various humans, hominids, primates, etc.
A reasonable position, so long as you remain truly ignorant of what AI is specifically about.
I don’t know if inside view forecasting can ever be more reliable than outside view forecasting. It seems that insiders as a general and very robust rule tend to be strongly overconfident, and see all kinds of reason why their particular instance is different and will have better outcome than the reference class.
http://www.overcomingbias.com/2007/07/beware-the-insi.html
http://en.wikipedia.org/wiki/Reference_class_forecasting
Try applying that to physics, engineering, biology, or any other technical field. In many cases, the outside view doesn’t stand a chance.