Please inform me if anyone knows of a better charity.
As long as you presume that the SIAI saves a potential galactic civilization from extinction (i.e. from being created), and assign a high enough probably to that outcome, nobody is going to be able to inform you of a charity with an higher payoff. At least as long as no other organization is going to make similar claims (implicitly or explicitly).
The risk of human extinction by AI (irrespective of countermeasures).
Probability of the SIAI succeeding to implement an AI (see 3.) taking care of any risks thereafter.
Estimated trustworthiness of the SIAI (signaling common good (friendly AI/CEV) while following selfish objectives (unfriendly AI)).
I’d also like you to tackle some problems I see regarding the SIAI in its current form:
Transparency
How do you know that they are trying to deliver what they are selling? If you believe the premise of AI going FOOM and that the SIAI is trying to implement a binding policy based on which the first AGI is going to FOOM, then you believe that the SIAI is an organisation involved in shaping the future of the universe. If the stakes are this high there does exist a lot of incentive for deception. Can you conclude that because someone writes a lot of ethical correct articles and papers that that output is reflective of their true goals?
Agenda and Progress
The current agenda seems to be very broad and vague. Can the SIAI make effective progress given such an agenda compared to specialized charities and workshops focusing on more narrow sub-goals?
How do you estimate their progress?
What are they working on right now?
Are there other organisations working on some of the sub-goals that make better progress?
As multifoliaterose implied here, at the moment the task to recognize humans as distinguished beings already seems to be too broad a problem to tackle directly. Might it be more effective, at this point, to concentrate on supporting other causes leading towards the general goal of AI associated existential risk mitigation?
Third Party Review
Without being an expert and without any peer review, how sure can you be about the given premises (AI going FOOM etc.) and the effectiveness of their current agenda?
Also what conclusion should one draw from the fact that at least 2 people who have been working for the SIAI, or have been in close contact with it, do disagree with some of the stronger claims. Robin Hanson seems not to be convinced that donating to the SIAI is an effective way to mitigate risks from AI? Ben Goertzel does not believe into the scary idea. And Katja Grace thinks AI is no big threat.
I consider the above form of futurism to be the “narrow view”. It considers too few possibilities over too short a timespan.
AI is not the only extinction risk we face.
AI is useful for a LOT more than just preventing extinction.
FOOM isn’t necessary for AI to cause extinction.
AI seems inevitable, assuming humans survive other risks.
Human extinction by AI doesn’t require the AI to swallow its light cone (Katja).
My interpretation of Ben’s article is that he’s saying SIAI is correct in everything except the probability that they can change the outcome.
You didn’t mention third parties who support SIAI, like Nick Bostrom, who I consider to be the preeminent analyst on these topics.
I’m not academic enough to provide the defense you’re looking for. Instead, I’ll do what I did at the end of the above linked thread, and say you should read more source material. And no, I don’t know what the best material is. And yes, this is SIAI’s problem. They really do suck at marketing. I think it’d be pretty funny if they failed because they didn’t have a catchy slogan...
I will give one probability estimate, since I already linked to it: SIAI fails in their mission AND all homo sapiens are extinct by the year 2100: 90 percent. I’m donating in the hopes of reducing that estimate as much as possible.
I’ll do what I did at the end of the above linked thread, and say you should read more source material.
One of my main problems regarding risks from AI is that I do not see anything right now that would hint at the possibility of FOOM. I am aware that you can extrapolate from the chimpanzee-human bridge. But does the possibility of superchimpanzee-intelligence really imply superhuman-intelligence? Even if that was the case, which I consider sparse evidence to neglect other risks for, I do not see that it implies FOOM (e.g. vast amounts of recursive self-improvement). You might further argue that even human-level intelligence (EMS or AI) might pose a significant risk when speed-up or by means of brute-force. In any case, I do believe that the associated problems to create any such intelligence are vastly greater than the problem to limit an intelligence, its scope of action. I believe that it is reasonable to assume that there will be a gradual development with many small-impact mistakes that will lead to a thoroughly comprehension of intelligence and its risks before any superhuman-intelligence could pose an existential risk.
One of my main problems regarding risks from AI is that I do not see anything right now that would hint at the possibility of FOOM.
I see foom as a completely separate argument from FAI or AGI or extinction risks. Certainly it would make things more chaotic and difficult to handle, increasing risk and uncertainty, but it’s completely unnecessary for chaos, risk, and destruction to occur—humans are quite capable of that on their own.
Once an AGI is “out there” and starts getting copied (assuming no foom), I want to make sure they’re all pointed in the right direction, regardless of capabilities, just as I want that for nuclear and other weapons. I think there’s a possibility we’ll be arguing over the politics of enemy states getting an AGI. That doesn’t seem to be a promising future. FAI is arms control, and a whole lot more.
Once an AGI is “out there” and starts getting copied...
I do not see that. The first AGI will likely be orders of magnitude slower (not less intelligent) than a standard human and be running on some specialized computational substrate (supercomputer). If you remove FOOM from the equation then I see many other existential risks being as dangerous as AI associated risks.
Again, a point-in-time view. Maybe you’re just not playing it out in your head like I am? Because when you say, “the first AGI will likely be orders of magnitude slower”, I think to myself, uh, who cares? What about the one built three years later that’s 3x faster and runs on a microcomputer? Does the first one being slow somehow make that other one less dangerous? Or that no one else will build one? Or that AGI theory will stagnate after the first artificial mind goes online? (?!?!)
Why does it have to happen ‘in one day’ for it to be dangerous? It could take a hundred years, and still be orders of magnitude more dangerous than any other known existential risk.
Does the first one being slow somehow make that other one less dangerous?
Yes, because I believe that the development will be gradually enough to tackle any risks on the way to a superhuman AGI, if superhuman capability is possible at all. There are certain limitations. Shortly after the invention of rocket science people landed on the moon. But the development eventually halted or slowed down. We haven’t reached other star systems yet. By that metaphor I want highlight that I am not aware of good arguments or other kinds of evidence indicating that an AGI would likely result in a run-away risk at any point of its development. It is possible but I am not sure that because of its low-probability we can reasonable neglect other existential risks. I believe that once we know how to create artificial intelligence capable of learning on a human level our comprehension of its associated risks and ability to limit its scope will have increased dramatically as well.
You’re using a different definition of AI than me. I’m thinking of ‘a mind running on a computer’ and you’re apparently thinking of ‘a human-like mind running on a computer’, where ‘human-like’ includes a lot of baggage about ‘what it means to be a mind’ or ‘what it takes to have a mind’.
I think any AI built from scratch will be a complete alien, and we won’t know just how alien until it starts doing stuff for reasons we’re incapable of understanding. And history has proven that the more sophisticated and complex the program, the more bugs, and the more it goes wrong in weird, subtle ways. Most such programs don’t have will, intent, or the ability to converse with you, making them substantially less likely to run away.
And again, you’re positing that people will understand, accept, and put limits in place, where there’s substantial incentives to let it run as free and as fast as possible.
Progress is accelerating in a sense—due to synergy between developments making progress easier.
...when progress is at its fastest, things might well get pretty interesting. Much in the way of “existential” risk seems pretty unlikely to me—but with speculative far-future events, it is hard to be certain.
What does look as though it will go up against the wall are unmodified human beings—and other multicellular DNA-based lifeforms. There is no way these can compete against engineering and intelligent design—and they look like an unsuitable foundation for building directly on top of.
Some will paint that as an apocalypse—though to me it looks like a sensible, obvious and practically inevitable move. The most reasonable hope for the continued existence of biological humans is in future equivalents of museums and historical simulations—IMO.
To restate my original question, is there anyone out there doing better than your estimated 0.0000003%? Even though the number is small, it could still be the highest.
To restate my original question, is there anyone out there doing better than your estimated 0.0000003%?
None whose goal is to save humanity from an existential risk. Although asteroid surveillance might come close, I’m not sure. It is not my intention to claim that donating to the SIAI is worthless, I believe that the world does indeed need an organisation that does tackle the big picture. In other words, I am not saying that you shouldn’t be donating to the SIAI, I am happy someone does (if only because of LW). But the fervor in this thread seemed to me completely unjustified. One should seriously consider if there are other groups worthy of promotion or if there should be other groups doing the same as the SIAI or being dealing with one of its sub-goals.
My main problem is how far I should go to neglect other problems in favor of some high-impact low-probability event. If your number of possible beings of human descent is high enough, and you assign each being enough utility, you can outweigh any low probability. You could probably calculate not to help someone who is drowning because 1.) you’d risk your own life and all the money you could make to donate to the SIAI 2.) in that time you could tell 5 people about existential risks from AI. I am exaggerating to highlight my problem. I’m just not educated enough yet, I have to learn more math, especially probability. Right now I feel that it is unreasonable to donate my whole money (or a lot) to the SIAI.
It really saddens me to see how often LW perceives any critique of the SIAI as ill-intentioned. As if people want to destroy the world. There are some morons out there, but most really would like to save the world if possible. They just don’t see that the SIAI is a reasonable choice to do so.
the fervor in this thread seemed to me completely unjustified. [...] My main problem is how far I should go to neglect other problems in favor of some high-impact low-probability event.
I agree with SIAI’s goals. I don’t see it as “fervor”. I see it as: I can do something to make this world a better place (according to my own understanding, in a better way than any other possible), therefore I will do so.
I compartmentalize. Humans are self-contradictory in many ways. I can send my entire bank account to some charity in the hopes of increasing the odds of friendly AI, and I can buy a hundred dollar bottle of bourbon for my own personal enjoyment. Sometimes on the same day. I’m not ultra-rational or pure utilitarian. I’m a regular person with various drives and desires. I save frogs from my stairwell rather than driving straight to work and earning more money. I do what I can.
One should seriously consider if there are other groups worthy of promotion or if there should be other groups doing the same as the SIAI or being dealing with one of its sub-goals.
I have seriously considered it. I have looked for such groups, here and elsewhere, and no one has ever presented a contender. That’s why I made my question as simple and straightforward as possible: name something more important. No one’s named anything so far, and I still read for many hours each week on this and other such topics, so hopefully if one arises, I’ll know and be able to evaluate it.
I donate based on relative merit. As I said at the end of my original supporting post: so far, no one else seems to come close to SIAI. I’m comfortable with giving away a large portion of my income because I don’t have much use for it myself. I post it here because it encourages others to give of themselves. I think it’s the right thing to do.
I know it’s hard to see why. I wish they had better marketing materials. I was really hoping the last challenge, with projects like a landing page, a FAQ, etc., would make a difference. So far, I don’t see much in the way of results, which is upsetting.
I still think it’s the right place to put my money.
If you’re going to do this sort of explicit decomposition at all, it’s probably also worth thinking explicitly about the expected value of a donation. That is: how much does your .0001 estimate of SIAI’s chance of preventing a humanity-destroying AI go up or down based on an N$ change in its annual revenue?
Thanks, you are right. I’d actually do a lot more but I feel I am not yet ready to tackle this topic mathematically. I only started getting into math in 2009. I asked several times for an analysis with input variables I could use to come up with my own estimations of the expected value of a donation to the SIAI. I asked people who are convinced of the SIAI to provide a decision procedure on how they were convinced. I asked them to lay it open to public inspection so people could reassess the procedure and calculations to compute their own conclusion. In response they asked me to do so myself. I do not take it amiss, they do not have to convince me. I am not able to do so yet. But while learning math I try to encourage other people to think about it.
That is: how much does your .0001 estimate of SIAI’s chance of preventing a humanity-destroying AI go up or down based on an N$ change in its annual revenue?
I feel that this deserves a direct answer. I think it is not just about money. The question would be, what would they do with it, would they actually hire experts? I will assume the best-case scenario here.
If the SIAI would be able to obtain a billion dollars I’d estimate the chance of the SIAI to prevent a FOOMing uFAI 10%.
This part is the one that seems the most different from my own probabilities:
AI going FOOM being an x-risk: 5%
So, do you think the default case is a friendly AI? Or at least innocuous AI? Or that friendly AI is easy enough so that whoever first makes a fooming AI will get the friendliness part right with no influence from the SIAI?
No, I do not believe that the default case is friendly AI. But I believe that AI going FOOM is, if possible at all, very hard to accomplish. Surely everyone agrees here. But at the moment I do not share the opinion that friendliness, that is to implement scope boundaries, is a very likely failure mode. I see it this way, if one can figure out how to create an AGI that FOOM’s (no I do not think AGI implies FOOM) then you have a thorough comprehension of intelligence and its associated risks. I just don’t see that a group of researchers (I don’t believe a mere group is enough anyway) will be smart enough to create an AGI that does FOOM but somehow fail to limit its scope. Please consider reading this comment where I cover this topic in more detail. That is why I believe that only 5% of all AI’s going FOOM will be an existential risk to all of humanity. That is my current estimation, I’ll of course update on new evidence (e.g. arguments).
You seem to be assigning probabilities to this—as though it is a well defined idea—but what is it supposed to mean?
I know (I don’t), but since I asked Rain to assign probabilities to it I felt that I had to state my own as well. I asked him to do so because I read that some people are arguing in favor of making probability estimates, to say a number. But since I haven’t come across much analysis that actually does state numbers I thought I’d ask a donor who contributed the current balance of his bank account.
Well, bypassing the issue of FOOMingness, I am pretty sure that machine intelligence represents an upcoming issue that could go better or worse than average—and which humanity should try and steer in a positive direction—x-risk or no.
My concerns about the SIAI are mostly about their competence. It seems rather easy for me to imagine another organisation in the SIAI’s niche doing a much better job. Are 63 chapters of a Harry Potter fanfic really helping, for instance?
Also, if they think using fear of THE END OF THE WORLD is a good way to stimulate donations, I would be very interested to see information about the effect on society of such marketing. Will it produce a culture of fear? What about The risks of caution?
My general impression is that spreading the DOOM virus around is rarely very constructive. It may well be actively harmful. In financial markets, prophesying market crashes may actually help make them happen, since the whole system works like a big rumour mill—and if a crash is coming, it makes sense to cash in and buy gold—and, if everyone does that, then the crash happens. A case of the self-fulfilling prophesy. The prophet may look smug—but if only they had kept their mouth shut!
Have the DOOM merchants looked into this kind of thing? Where are their reassurances that prophesying DOOM—and separating passing punters from their cash in the process—is a harmless pass-time, with no side effects?
My concerns about the SIAI are mostly about their competence. It seems rather easy for me to imagine another organisation in the SIAI’s niche doing a much better job. Are 63 chapters of a Harry Potter fanfic really helping, for instance?
That isn’t an SIAI thing, that’s Eliezer’s thing. But if you really want to know, it seems from anecdotal evidence that HPMR is helping raise the general sanity waterline. Not only has it made more people be interested in LW in general, I can personally attest to it helping modify irrational beliefs that friends have had.
(Also, Tim, I know you are very fond of capitalizing “DOOM” and certain other phrases, but the rest of us find it distracting and disruptive. Could you please consider not doing it here?)
Also, if they think using fear of THE END OF THE WORLD is a good way to stimulate donations, I would be very interested to see information about the effect on society of such marketing. Will it produce a culture of fear? What about The risks of caution?
I’m not sure why you think they think that doomsday predictions are a good way to stimulate donations. They are simply being honest in their goals. Empirically, existential risk is not a great motivator for getting money. Look for example at how much trouble people concerned with asteroid impacts have getting money (although now that the WISE survey is complete we’re in much better shape in understanding and handling that risk.)
My general impression is that spreading the DOOM virus around is rarely very constructive. It may well be actively harmful.
So should people not say what they are honestly thinking?
In financial markets, prophesying market crashes may actually help make them happen, since the whole system works like a big rumour mill—and if a crash is coming, it makes sense to cash in and buy gold—and, if everyone does that, then the crash happens. A case of the self-fulfilling prophesy. The prophet may look smug—but if only they had kept their mouth shut!
Yes, that can happen in markets. What is the analogy here? Is there a situation where simply talking about the risk of unFriendly AI will somehow make unFriendly AI more likely? (And note, improbable decision-theory basilisks don’t count.)
Have the DOOM merchants looked into this kind of thing? Where are their reassurances that prophesying DOOM—and separating passing punters from their cash in the process—is a harmless pass-time, with no side effects?
If your standard is that they have to be clear there are no side effects, that’s a pretty high standard. How certain do they need to be? To return to the asteroid example, thanks to the WISE mission we now are tracking about 95% of all asteroids that could pose an extinction threat if they impacted, and are tracking a much higher percentage of those that live in severely threatening orbits. But, whenever we spend any money it means we might be missing that small percentage. We’ll feel really stupid if our donations to any cause turn out not to matter because we missed another one. If a big asteroid hits the Earth tomorrow we’ll feel really dumb. By the same token, we’ll feel really stupid if tomorrow someone makes an approximation of AIXI devoted to playing WoW that goes foom. The fact that we have the asteroids charted won’t make any difference. No matter how good an estimate we do, there’s a chance we’ll be wrong. And no matter what happens there are side effects, simply due at minimum to the fact that we have a finite set of resources. And the more we talk about any issue the less we are focusing on others. And yes, obviously if fooming turns out to not be an issue, there will be negative side effects. So where is the line?
if you really want to know, it seems from anecdotal evidence that HPMR is helping raise the general sanity waterline
I haven’t looked—but it seems to be pretty amazing behaviour to me.
I’m not sure why you think they think that doomsday predictions are a good way to stimulate donations.
Using threats of the apocalypse is an ancient method, used by religions and cults for centuries.
Look for example at how much trouble people concerned with asteroid impacts have getting money
Their smallish p(DOOM) values probably don’t help too much.
My general impression is that spreading the DOOM virus around is rarely very constructive. It may well be actively harmful.
So should people not say what they are honestly thinking?
It is up to the people involved if they want to dabble in harmful self-fulfilling prophesies. Maybe society should reward them less and ignore them more, though. I figure, if we study the DOOM merchants more scientifically, we will have a better understanding of the risks and problems they cause—and what we should do about them.
Most people already have a pretty high barrier against END OF THE WORLD schemes. It is such an obvious and well-worn routine. However, it appears that not everyone has been immunised.
What is the analogy here. Is there a situation where simply talking about the risk of unFriendly AI will somehow make unFriendly AI more likely?
Ideally, DOOM SOON should sharpen our wits, and make us more vigilant and secure. However, the opposite response seems quite likely: DOOM SOON might make people feel despair, apathy, helplessness, futility and depression. Those things could then go on to cause a variety of problems. Most of them are not to do with intelligent machines—though the one I already mentioneddoes involve them.
Have the DOOM merchants looked into this kind of thing? Where are their reassurances that prophesying DOOM—and separating passing punters from their cash in the process—is a harmless pass-time, with no side effects?
If your standard is that they have to be clear there are no side effects, that’s a pretty high standard.
Sure. Doing more good than harm would be a nice start. I don’t know what the side effects of DOOM-mongering are—in detail, so it is hard to judge the scale of the side effects—besides the obvious financial losses among those involved. Probably, the most visible behaviour of the afflicted individuals is that they start flapping their hands and going on about DOOM—spreading the meme after being infected by it. To what extent this affects their relationships, work, etc. is not entirely clear. I would be interested in finding out, though.
My understanding is it means “the AI gets to a point where software improvements allow it to outpace us and trick us into doing anything it wants us to, and understand nanotechnology at a scale that it soon has unlimited material power.”
Instead of 1e-4 I’d probably put that at 1e-6 to 1e-9, but I have little experience accurately estimating very low probabilities.
(The sticking point of my interpretation is something that seems glossed over in the stuff I’ve read about it- that the AI only has complete access to software improvements. If it’s working on chips made of silicon, all it can do is tell us better chip designs (unless it’s hacked a factory, and is able to assemble itself somehow). Even if it’s as intelligent as EY imagines it can be, I don’t see how it could derive GR from a webcam quality picture; massive intelligence is no replacement for scant evidence. Those problems can be worked around- if it has access to the internet, it’s got a lot of evidence and a lot of power- but suggest that in some limited cases FOOM is very improbable.)
I am pretty sure that the “FOOM” term is an attempt to say something about the timescale of the growth of machine intelligence. So, I am sceptical about definitions which involve the concept of trickery. Surely rapid growth need not necessarily involve trickery. My FOOM sources don’t seem to mention trickery. Do you have any references relating to the point?
The bit about “trickery” was probably just referencing the weaknesses of AI boxing. You are correct that it’s not essential to the idea of hard takeoff.
The rational reasons to signal are outlined in the post Why Our Kind Can’t Cooperate, and there are more good articles with the Charity tag.
My personal reasons for supporting SIAI are outlined entirely in this comment.
Please inform me if anyone knows of a better charity.
As long as you presume that the SIAI saves a potential galactic civilization from extinction (i.e. from being created), and assign a high enough probably to that outcome, nobody is going to be able to inform you of a charity with an higher payoff. At least as long as no other organization is going to make similar claims (implicitly or explicitly).
If you don’t mind I would like you to state some numerical probability estimates:
The risk of human extinction by AI (irrespective of countermeasures).
Probability of the SIAI succeeding to implement an AI (see 3.) taking care of any risks thereafter.
Estimated trustworthiness of the SIAI (signaling common good (friendly AI/CEV) while following selfish objectives (unfriendly AI)).
I’d also like you to tackle some problems I see regarding the SIAI in its current form:
Transparency
How do you know that they are trying to deliver what they are selling? If you believe the premise of AI going FOOM and that the SIAI is trying to implement a binding policy based on which the first AGI is going to FOOM, then you believe that the SIAI is an organisation involved in shaping the future of the universe. If the stakes are this high there does exist a lot of incentive for deception. Can you conclude that because someone writes a lot of ethical correct articles and papers that that output is reflective of their true goals?
Agenda and Progress
The current agenda seems to be very broad and vague. Can the SIAI make effective progress given such an agenda compared to specialized charities and workshops focusing on more narrow sub-goals?
How do you estimate their progress?
What are they working on right now?
Are there other organisations working on some of the sub-goals that make better progress?
As multifoliaterose implied here, at the moment the task to recognize humans as distinguished beings already seems to be too broad a problem to tackle directly. Might it be more effective, at this point, to concentrate on supporting other causes leading towards the general goal of AI associated existential risk mitigation?
Third Party Review
Without being an expert and without any peer review, how sure can you be about the given premises (AI going FOOM etc.) and the effectiveness of their current agenda?
Also what conclusion should one draw from the fact that at least 2 people who have been working for the SIAI, or have been in close contact with it, do disagree with some of the stronger claims. Robin Hanson seems not to be convinced that donating to the SIAI is an effective way to mitigate risks from AI? Ben Goertzel does not believe into the scary idea. And Katja Grace thinks AI is no big threat.
More
What I would like the SIAI to publish
Should I believe what the SIAI claims?
My own estimations
AI going FOOM: 0.1%
AI going FOOM being an x-risk: 5%
AI going FOOM being an x-risk is prevented by the SIAI: 0.01%
That the SIAI is trustworthy of pursuing to create the best possible world for all human beings: 60%
Therefore that a donation to the SIAI does pay off: 0.0000003%
I consider the above form of futurism to be the “narrow view”. It considers too few possibilities over too short a timespan.
AI is not the only extinction risk we face.
AI is useful for a LOT more than just preventing extinction.
FOOM isn’t necessary for AI to cause extinction.
AI seems inevitable, assuming humans survive other risks.
Human extinction by AI doesn’t require the AI to swallow its light cone (Katja).
My interpretation of Ben’s article is that he’s saying SIAI is correct in everything except the probability that they can change the outcome.
You didn’t mention third parties who support SIAI, like Nick Bostrom, who I consider to be the preeminent analyst on these topics.
I’m not academic enough to provide the defense you’re looking for. Instead, I’ll do what I did at the end of the above linked thread, and say you should read more source material. And no, I don’t know what the best material is. And yes, this is SIAI’s problem. They really do suck at marketing. I think it’d be pretty funny if they failed because they didn’t have a catchy slogan...
I will give one probability estimate, since I already linked to it: SIAI fails in their mission AND all homo sapiens are extinct by the year 2100: 90 percent. I’m donating in the hopes of reducing that estimate as much as possible.
One of my main problems regarding risks from AI is that I do not see anything right now that would hint at the possibility of FOOM. I am aware that you can extrapolate from the chimpanzee-human bridge. But does the possibility of superchimpanzee-intelligence really imply superhuman-intelligence? Even if that was the case, which I consider sparse evidence to neglect other risks for, I do not see that it implies FOOM (e.g. vast amounts of recursive self-improvement). You might further argue that even human-level intelligence (EMS or AI) might pose a significant risk when speed-up or by means of brute-force. In any case, I do believe that the associated problems to create any such intelligence are vastly greater than the problem to limit an intelligence, its scope of action. I believe that it is reasonable to assume that there will be a gradual development with many small-impact mistakes that will lead to a thoroughly comprehension of intelligence and its risks before any superhuman-intelligence could pose an existential risk.
I see foom as a completely separate argument from FAI or AGI or extinction risks. Certainly it would make things more chaotic and difficult to handle, increasing risk and uncertainty, but it’s completely unnecessary for chaos, risk, and destruction to occur—humans are quite capable of that on their own.
Once an AGI is “out there” and starts getting copied (assuming no foom), I want to make sure they’re all pointed in the right direction, regardless of capabilities, just as I want that for nuclear and other weapons. I think there’s a possibility we’ll be arguing over the politics of enemy states getting an AGI. That doesn’t seem to be a promising future. FAI is arms control, and a whole lot more.
I do not see that. The first AGI will likely be orders of magnitude slower (not less intelligent) than a standard human and be running on some specialized computational substrate (supercomputer). If you remove FOOM from the equation then I see many other existential risks being as dangerous as AI associated risks.
Again, a point-in-time view. Maybe you’re just not playing it out in your head like I am? Because when you say, “the first AGI will likely be orders of magnitude slower”, I think to myself, uh, who cares? What about the one built three years later that’s 3x faster and runs on a microcomputer? Does the first one being slow somehow make that other one less dangerous? Or that no one else will build one? Or that AGI theory will stagnate after the first artificial mind goes online? (?!?!)
Why does it have to happen ‘in one day’ for it to be dangerous? It could take a hundred years, and still be orders of magnitude more dangerous than any other known existential risk.
Yes, because I believe that the development will be gradually enough to tackle any risks on the way to a superhuman AGI, if superhuman capability is possible at all. There are certain limitations. Shortly after the invention of rocket science people landed on the moon. But the development eventually halted or slowed down. We haven’t reached other star systems yet. By that metaphor I want highlight that I am not aware of good arguments or other kinds of evidence indicating that an AGI would likely result in a run-away risk at any point of its development. It is possible but I am not sure that because of its low-probability we can reasonable neglect other existential risks. I believe that once we know how to create artificial intelligence capable of learning on a human level our comprehension of its associated risks and ability to limit its scope will have increased dramatically as well.
You’re using a different definition of AI than me. I’m thinking of ‘a mind running on a computer’ and you’re apparently thinking of ‘a human-like mind running on a computer’, where ‘human-like’ includes a lot of baggage about ‘what it means to be a mind’ or ‘what it takes to have a mind’.
I think any AI built from scratch will be a complete alien, and we won’t know just how alien until it starts doing stuff for reasons we’re incapable of understanding. And history has proven that the more sophisticated and complex the program, the more bugs, and the more it goes wrong in weird, subtle ways. Most such programs don’t have will, intent, or the ability to converse with you, making them substantially less likely to run away.
And again, you’re positing that people will understand, accept, and put limits in place, where there’s substantial incentives to let it run as free and as fast as possible.
Sorry, I meant human-level learning capability when I said human like.
We have had quite a bit of self-improvement so far—according to my own:
http://www.alife.co.uk/essays/the_intelligence_explosion_is_happening_now/
Progress is accelerating in a sense—due to synergy between developments making progress easier.
...when progress is at its fastest, things might well get pretty interesting. Much in the way of “existential” risk seems pretty unlikely to me—but with speculative far-future events, it is hard to be certain.
What does look as though it will go up against the wall are unmodified human beings—and other multicellular DNA-based lifeforms. There is no way these can compete against engineering and intelligent design—and they look like an unsuitable foundation for building directly on top of.
Some will paint that as an apocalypse—though to me it looks like a sensible, obvious and practically inevitable move. The most reasonable hope for the continued existence of biological humans is in future equivalents of museums and historical simulations—IMO.
To restate my original question, is there anyone out there doing better than your estimated 0.0000003%? Even though the number is small, it could still be the highest.
None whose goal is to save humanity from an existential risk. Although asteroid surveillance might come close, I’m not sure. It is not my intention to claim that donating to the SIAI is worthless, I believe that the world does indeed need an organisation that does tackle the big picture. In other words, I am not saying that you shouldn’t be donating to the SIAI, I am happy someone does (if only because of LW). But the fervor in this thread seemed to me completely unjustified. One should seriously consider if there are other groups worthy of promotion or if there should be other groups doing the same as the SIAI or being dealing with one of its sub-goals.
My main problem is how far I should go to neglect other problems in favor of some high-impact low-probability event. If your number of possible beings of human descent is high enough, and you assign each being enough utility, you can outweigh any low probability. You could probably calculate not to help someone who is drowning because 1.) you’d risk your own life and all the money you could make to donate to the SIAI 2.) in that time you could tell 5 people about existential risks from AI. I am exaggerating to highlight my problem. I’m just not educated enough yet, I have to learn more math, especially probability. Right now I feel that it is unreasonable to donate my whole money (or a lot) to the SIAI.
It really saddens me to see how often LW perceives any critique of the SIAI as ill-intentioned. As if people want to destroy the world. There are some morons out there, but most really would like to save the world if possible. They just don’t see that the SIAI is a reasonable choice to do so.
I agree with SIAI’s goals. I don’t see it as “fervor”. I see it as: I can do something to make this world a better place (according to my own understanding, in a better way than any other possible), therefore I will do so.
I compartmentalize. Humans are self-contradictory in many ways. I can send my entire bank account to some charity in the hopes of increasing the odds of friendly AI, and I can buy a hundred dollar bottle of bourbon for my own personal enjoyment. Sometimes on the same day. I’m not ultra-rational or pure utilitarian. I’m a regular person with various drives and desires. I save frogs from my stairwell rather than driving straight to work and earning more money. I do what I can.
I have seriously considered it. I have looked for such groups, here and elsewhere, and no one has ever presented a contender. That’s why I made my question as simple and straightforward as possible: name something more important. No one’s named anything so far, and I still read for many hours each week on this and other such topics, so hopefully if one arises, I’ll know and be able to evaluate it.
I donate based on relative merit. As I said at the end of my original supporting post: so far, no one else seems to come close to SIAI. I’m comfortable with giving away a large portion of my income because I don’t have much use for it myself. I post it here because it encourages others to give of themselves. I think it’s the right thing to do.
I know it’s hard to see why. I wish they had better marketing materials. I was really hoping the last challenge, with projects like a landing page, a FAQ, etc., would make a difference. So far, I don’t see much in the way of results, which is upsetting.
I still think it’s the right place to put my money.
If you’re going to do this sort of explicit decomposition at all, it’s probably also worth thinking explicitly about the expected value of a donation. That is: how much does your .0001 estimate of SIAI’s chance of preventing a humanity-destroying AI go up or down based on an N$ change in its annual revenue?
Thanks, you are right. I’d actually do a lot more but I feel I am not yet ready to tackle this topic mathematically. I only started getting into math in 2009. I asked several times for an analysis with input variables I could use to come up with my own estimations of the expected value of a donation to the SIAI. I asked people who are convinced of the SIAI to provide a decision procedure on how they were convinced. I asked them to lay it open to public inspection so people could reassess the procedure and calculations to compute their own conclusion. In response they asked me to do so myself. I do not take it amiss, they do not have to convince me. I am not able to do so yet. But while learning math I try to encourage other people to think about it.
I feel that this deserves a direct answer. I think it is not just about money. The question would be, what would they do with it, would they actually hire experts? I will assume the best-case scenario here.
If the SIAI would be able to obtain a billion dollars I’d estimate the chance of the SIAI to prevent a FOOMing uFAI 10%.
This part is the one that seems the most different from my own probabilities:
So, do you think the default case is a friendly AI? Or at least innocuous AI? Or that friendly AI is easy enough so that whoever first makes a fooming AI will get the friendliness part right with no influence from the SIAI?
No, I do not believe that the default case is friendly AI. But I believe that AI going FOOM is, if possible at all, very hard to accomplish. Surely everyone agrees here. But at the moment I do not share the opinion that friendliness, that is to implement scope boundaries, is a very likely failure mode. I see it this way, if one can figure out how to create an AGI that FOOM’s (no I do not think AGI implies FOOM) then you have a thorough comprehension of intelligence and its associated risks. I just don’t see that a group of researchers (I don’t believe a mere group is enough anyway) will be smart enough to create an AGI that does FOOM but somehow fail to limit its scope. Please consider reading this comment where I cover this topic in more detail. That is why I believe that only 5% of all AI’s going FOOM will be an existential risk to all of humanity. That is my current estimation, I’ll of course update on new evidence (e.g. arguments).
I looked at http://lesswrong.com/lw/wf/hard_takeoff/
I was left pretty puzzled about what “AI go FOOM” was actually intended to mean. The page shies away from making any kind of quantifiable statement.
You seem to be assigning probabilities to this—as though it is a well defined idea—but what is it supposed to mean?
I know (I don’t), but since I asked Rain to assign probabilities to it I felt that I had to state my own as well. I asked him to do so because I read that some people are arguing in favor of making probability estimates, to say a number. But since I haven’t come across much analysis that actually does state numbers I thought I’d ask a donor who contributed the current balance of his bank account.
Well, bypassing the issue of FOOMingness, I am pretty sure that machine intelligence represents an upcoming issue that could go better or worse than average—and which humanity should try and steer in a positive direction—x-risk or no.
My concerns about the SIAI are mostly about their competence. It seems rather easy for me to imagine another organisation in the SIAI’s niche doing a much better job. Are 63 chapters of a Harry Potter fanfic really helping, for instance?
Also, if they think using fear of THE END OF THE WORLD is a good way to stimulate donations, I would be very interested to see information about the effect on society of such marketing. Will it produce a culture of fear? What about The risks of caution?
My general impression is that spreading the DOOM virus around is rarely very constructive. It may well be actively harmful. In financial markets, prophesying market crashes may actually help make them happen, since the whole system works like a big rumour mill—and if a crash is coming, it makes sense to cash in and buy gold—and, if everyone does that, then the crash happens. A case of the self-fulfilling prophesy. The prophet may look smug—but if only they had kept their mouth shut!
Have the DOOM merchants looked into this kind of thing? Where are their reassurances that prophesying DOOM—and separating passing punters from their cash in the process—is a harmless pass-time, with no side effects?
That isn’t an SIAI thing, that’s Eliezer’s thing. But if you really want to know, it seems from anecdotal evidence that HPMR is helping raise the general sanity waterline. Not only has it made more people be interested in LW in general, I can personally attest to it helping modify irrational beliefs that friends have had.
(Also, Tim, I know you are very fond of capitalizing “DOOM” and certain other phrases, but the rest of us find it distracting and disruptive. Could you please consider not doing it here?)
I’m not sure why you think they think that doomsday predictions are a good way to stimulate donations. They are simply being honest in their goals. Empirically, existential risk is not a great motivator for getting money. Look for example at how much trouble people concerned with asteroid impacts have getting money (although now that the WISE survey is complete we’re in much better shape in understanding and handling that risk.)
So should people not say what they are honestly thinking?
Yes, that can happen in markets. What is the analogy here? Is there a situation where simply talking about the risk of unFriendly AI will somehow make unFriendly AI more likely? (And note, improbable decision-theory basilisks don’t count.)
If your standard is that they have to be clear there are no side effects, that’s a pretty high standard. How certain do they need to be? To return to the asteroid example, thanks to the WISE mission we now are tracking about 95% of all asteroids that could pose an extinction threat if they impacted, and are tracking a much higher percentage of those that live in severely threatening orbits. But, whenever we spend any money it means we might be missing that small percentage. We’ll feel really stupid if our donations to any cause turn out not to matter because we missed another one. If a big asteroid hits the Earth tomorrow we’ll feel really dumb. By the same token, we’ll feel really stupid if tomorrow someone makes an approximation of AIXI devoted to playing WoW that goes foom. The fact that we have the asteroids charted won’t make any difference. No matter how good an estimate we do, there’s a chance we’ll be wrong. And no matter what happens there are side effects, simply due at minimum to the fact that we have a finite set of resources. And the more we talk about any issue the less we are focusing on others. And yes, obviously if fooming turns out to not be an issue, there will be negative side effects. So where is the line?
I haven’t looked—but it seems to be pretty amazing behaviour to me.
Using threats of the apocalypse is an ancient method, used by religions and cults for centuries.
Their smallish p(DOOM) values probably don’t help too much.
It is up to the people involved if they want to dabble in harmful self-fulfilling prophesies. Maybe society should reward them less and ignore them more, though. I figure, if we study the DOOM merchants more scientifically, we will have a better understanding of the risks and problems they cause—and what we should do about them.
Most people already have a pretty high barrier against END OF THE WORLD schemes. It is such an obvious and well-worn routine. However, it appears that not everyone has been immunised.
Ideally, DOOM SOON should sharpen our wits, and make us more vigilant and secure. However, the opposite response seems quite likely: DOOM SOON might make people feel despair, apathy, helplessness, futility and depression. Those things could then go on to cause a variety of problems. Most of them are not to do with intelligent machines—though the one I already mentioned does involve them.
Sure. Doing more good than harm would be a nice start. I don’t know what the side effects of DOOM-mongering are—in detail, so it is hard to judge the scale of the side effects—besides the obvious financial losses among those involved. Probably, the most visible behaviour of the afflicted individuals is that they start flapping their hands and going on about DOOM—spreading the meme after being infected by it. To what extent this affects their relationships, work, etc. is not entirely clear. I would be interested in finding out, though.
My understanding is it means “the AI gets to a point where software improvements allow it to outpace us and trick us into doing anything it wants us to, and understand nanotechnology at a scale that it soon has unlimited material power.”
Instead of 1e-4 I’d probably put that at 1e-6 to 1e-9, but I have little experience accurately estimating very low probabilities.
(The sticking point of my interpretation is something that seems glossed over in the stuff I’ve read about it- that the AI only has complete access to software improvements. If it’s working on chips made of silicon, all it can do is tell us better chip designs (unless it’s hacked a factory, and is able to assemble itself somehow). Even if it’s as intelligent as EY imagines it can be, I don’t see how it could derive GR from a webcam quality picture; massive intelligence is no replacement for scant evidence. Those problems can be worked around- if it has access to the internet, it’s got a lot of evidence and a lot of power- but suggest that in some limited cases FOOM is very improbable.)
I am pretty sure that the “FOOM” term is an attempt to say something about the timescale of the growth of machine intelligence. So, I am sceptical about definitions which involve the concept of trickery. Surely rapid growth need not necessarily involve trickery. My FOOM sources don’t seem to mention trickery. Do you have any references relating to the point?
The bit about “trickery” was probably just referencing the weaknesses of AI boxing. You are correct that it’s not essential to the idea of hard takeoff.