It was more than one person. Anyway, I haven’t read all of the comments yet so I might have missed some specific links. If you are talking about links to articles written by EY himself where he argues about AI going FOOM, I commented on one of them.
Here is an example of the kind of transparency in the form of strict calculations, references and evidence I expect.
As I said, I’m not sure what other links you are talking about. But if you mean the kind of LW posts dealing with antipredictions, I’m not impressed. Predicting superhuman AI to be a possible outcome of AI research is not sufficient. Where is the difference between claiming the LHC will go FOOM? I’m sure someone like EY would be able to write a thousand posts around such a scenario telling me that the high risk associated with the LHC going FOOM does outweigh its low probability. There might be sound arguments to support this conclusion. But it is a conclusion and a framework of arguments based on a assumption that is itself of unknown credibility. So is it too much to ask for some transparet evidence to fortify this basic premise? Evidence that is not somewhere to be found within hundreds of posts not directly concerned with the evidence in question but rather arguing based on the very assumption it is trying to justify?
Asteroids really are an easier problem: celestial mechanics in vacuum are pretty stable, we have the Moon providing a record of past cratering to calibrate on, etc. There’s still uncertainty about the technology of asteroid deflection (e.g. its potential for military use, or to incite conflict), but overall it’s perhaps the most tractable risk for analysis since the asteroids themselves don’t depend on recent events (save for some smallish anthropic shadow effects).
An analysis for engineered pathogens, where we have a lot of uncertainty about the difficulty of engineering various of diseases for maximum damage, and how the technology for detection, treatment and prevention will keep pace. We can make generalizations based on existing diseases and their evolutionary dynamics (selection for lower virulence over time with person-to-person transmission, etc), current public health measures, etc, the rarity of the relevant motivations, etc, but you’re still left with many more places where you can’t just plug in well-established numbers and crank forward.
You can still give probability estimates, and plug in well-understood past data where you can, but you can’t get asteroid-level exactitude.
The difference is that we understand both asteroids and particle physics far better than we do intelligence, and there is precedence for both asteroid impacts and high energy particle collisions (natural ones at far higher energy than in the LHC) while there is none for an engineered human level intelligence with access to its own source code.
So calculations of the kind you seem to be asking for just aren’t possible at this point (and calculations with exactly that level of evidence won’t be possible right up until it’s too late), while refutations of the kind LHC panic gets aren’t possible either. You should also note that Eliezer takes LHC panic more serious than most non-innumerate people.
But if you want some calculation anyway: Let’s assume there is a 1% chance of extinction by uFAI within the next 100 years. Let’s also assume that spending $10 million per year (in 2010 dollars, adjusting for inflation) allows us to reduce that risk by 10%, just by the dangers of uFAI being in the public eye and people being somewhat more cautious, and taking the right sort of caution instead of worrying about Skynet or homicidal robots. So $1 billion saves about an expected 1 million lives, a cost of $ 1000 per life, which is about the level of the most efficient conventional charities. And that’s with Robins low-balling estimate (which was for a more specific case, not uFAI extinction in general, so even Robin would likely estimate a higher chance in the case considered) and assuming that FAI research won’t succeed.
So calculations of the kind you seem to be asking for just aren’t possible at this point …
I’m asking for whatever calculations should lead people to donate most of their money to the SIAI or get nightmares from stories of distant FAI’s. Surely there must be something to outweigh the lack of evidence, or on what basis has anyone decided to take things serious?
I really don’t want to anger you but the “let’s assume X” attitude is what I have my problems with here. A 1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
I wonder why it took so long until you came along posting this comment.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
Have you yet bothered to read e.g. this synopsis of SIAI’s position:
“Many AIs will converge toward being optimizing systems, in the sense that, after self-modification, they will act to maximize some goal. For instance, AIs developed under evolutionary pressures would be selected for values that maximized reproductive fitness, and would prefer to allocate resources to reproduction rather than supporting humans. Such unsafe AIs might actively mimic safe benevolence until they became powerful, since being destroyed would prevent them from working toward their goals. Thus, a broad range of AI designs may initially appear safe, but if developed to the point of a Singularity could cause human extinction in the course of optimizing the Earth for their goals.”
Personally, I think that presents a very weak case for there being risk. It argues that there could be risk if we built these machines wrong, and the bad machines became powerful somehow. That is true—but the reader is inclined to respond “so what”. A dam can be dangerous if you build it wrong too. Such observations don’t say very much about the actual risk.
I am very sceptical about that being true for those alive now:
We have been looking for things that might hit us for a long while now—and we can see much more clearly what the chances are for that period than by looking at the historical record. Also, that is apparently assuming no mitigation attempts—which also seems totally unrealistic.
...gives 700 deaths/year for aircraft—and 1,400 deaths/year for 2km impacts—based on assumption that one quarter of the human population would perish in such an impact.
It was more than one person. Anyway, I haven’t read all of the comments yet so I might have missed some specific links. If you are talking about links to articles written by EY himself where he argues about AI going FOOM, I commented on one of them.
Here is an example of the kind of transparency in the form of strict calculations, references and evidence I expect.
As I said, I’m not sure what other links you are talking about. But if you mean the kind of LW posts dealing with antipredictions, I’m not impressed. Predicting superhuman AI to be a possible outcome of AI research is not sufficient. Where is the difference between claiming the LHC will go FOOM? I’m sure someone like EY would be able to write a thousand posts around such a scenario telling me that the high risk associated with the LHC going FOOM does outweigh its low probability. There might be sound arguments to support this conclusion. But it is a conclusion and a framework of arguments based on a assumption that is itself of unknown credibility. So is it too much to ask for some transparet evidence to fortify this basic premise? Evidence that is not somewhere to be found within hundreds of posts not directly concerned with the evidence in question but rather arguing based on the very assumption it is trying to justify?
Asteroids really are an easier problem: celestial mechanics in vacuum are pretty stable, we have the Moon providing a record of past cratering to calibrate on, etc. There’s still uncertainty about the technology of asteroid deflection (e.g. its potential for military use, or to incite conflict), but overall it’s perhaps the most tractable risk for analysis since the asteroids themselves don’t depend on recent events (save for some smallish anthropic shadow effects).
An analysis for engineered pathogens, where we have a lot of uncertainty about the difficulty of engineering various of diseases for maximum damage, and how the technology for detection, treatment and prevention will keep pace. We can make generalizations based on existing diseases and their evolutionary dynamics (selection for lower virulence over time with person-to-person transmission, etc), current public health measures, etc, the rarity of the relevant motivations, etc, but you’re still left with many more places where you can’t just plug in well-established numbers and crank forward.
You can still give probability estimates, and plug in well-understood past data where you can, but you can’t get asteroid-level exactitude.
The difference is that we understand both asteroids and particle physics far better than we do intelligence, and there is precedence for both asteroid impacts and high energy particle collisions (natural ones at far higher energy than in the LHC) while there is none for an engineered human level intelligence with access to its own source code.
So calculations of the kind you seem to be asking for just aren’t possible at this point (and calculations with exactly that level of evidence won’t be possible right up until it’s too late), while refutations of the kind LHC panic gets aren’t possible either. You should also note that Eliezer takes LHC panic more serious than most non-innumerate people.
But if you want some calculation anyway: Let’s assume there is a 1% chance of extinction by uFAI within the next 100 years. Let’s also assume that spending $10 million per year (in 2010 dollars, adjusting for inflation) allows us to reduce that risk by 10%, just by the dangers of uFAI being in the public eye and people being somewhat more cautious, and taking the right sort of caution instead of worrying about Skynet or homicidal robots. So $1 billion saves about an expected 1 million lives, a cost of $ 1000 per life, which is about the level of the most efficient conventional charities. And that’s with Robins low-balling estimate (which was for a more specific case, not uFAI extinction in general, so even Robin would likely estimate a higher chance in the case considered) and assuming that FAI research won’t succeed.
I’m asking for whatever calculations should lead people to donate most of their money to the SIAI or get nightmares from stories of distant FAI’s. Surely there must be something to outweigh the lack of evidence, or on what basis has anyone decided to take things serious?
I really don’t want to anger you but the “let’s assume X” attitude is what I have my problems with here. A 1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
So what’s the alternative?
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
This isn’t quite what you’re talking about, but a relatively accessible intro doc:
http://singinst.org/riskintro/index.html
This seems like a summary of the idea of there being significant risk:
Anna Salamon at Singularity Summit 2009 - “Shaping the Intelligence Explosion”
http://www.vimeo.com/7318055
Good comment.
However,
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
Have you yet bothered to read e.g. this synopsis of SIAI’s position:
http://singinst.org/riskintro/index.html
I’d also strongly recommend this from Bostrom:
http://www.nickbostrom.com/fut/evolution.html
(Then of course there are longer and more comprehensive texts, which I won’t recommend because you would just continue to ignore them.)
The core of:
http://singinst.org/riskintro/
...that talks about risk appears to be:
“Many AIs will converge toward being optimizing systems, in the sense that, after self-modification, they will act to maximize some goal. For instance, AIs developed under evolutionary pressures would be selected for values that maximized reproductive fitness, and would prefer to allocate resources to reproduction rather than supporting humans. Such unsafe AIs might actively mimic safe benevolence until they became powerful, since being destroyed would prevent them from working toward their goals. Thus, a broad range of AI designs may initially appear safe, but if developed to the point of a Singularity could cause human extinction in the course of optimizing the Earth for their goals.”
Personally, I think that presents a very weak case for there being risk. It argues that there could be risk if we built these machines wrong, and the bad machines became powerful somehow. That is true—but the reader is inclined to respond “so what”. A dam can be dangerous if you build it wrong too. Such observations don’t say very much about the actual risk.
This calculation places no value on the future generations whose birth depends on averting existential risk. That’s not how I see things.
That claims that “that the lifetime risk of dying from an asteroid strike is about the same as the risk of dying in a commercial airplane crash”.
It cites:
Impacts on the Earth by asteroids and comets: assessing the hazard:
http://www.nature.com/nature/journal/v367/n6458/abs/367033a0.html
I am very sceptical about that being true for those alive now:
We have been looking for things that might hit us for a long while now—and we can see much more clearly what the chances are for that period than by looking at the historical record. Also, that is apparently assuming no mitigation attempts—which also seems totally unrealistic.
Looking further:
http://users.tpg.com.au/users/tps-seti/spacegd7.html
...gives 700 deaths/year for aircraft—and 1,400 deaths/year for 2km impacts—based on assumption that one quarter of the human population would perish in such an impact.
Yet, does the SIAI provide evidence on par with the paper I linked to?
What—about the chances of superintelligence causing THE END OF THE WORLD?!?
Of course not! How could they be expected to do that?