I’m definitely interested in funding an existential risk organization. SIAI would have to be a lot more transparent than it is now right now for me to be interested in funding SIAI. For me personally, it wouldn’t be enough for SIAI to just take measures to avoid poisoning the meme, I would need to see a lot more evidence that SIAI is systematically working to maximize its impact on existential risk reduction.
As things stand I prefer to hold out for a better organization. But if SIAI exhibited transparency and accountability of levels similar to those of GiveWell (welcoming and publically responding to criticism regularly, regularly posting detailed plans of action, seeking out feedback from subject matter specialists and making this public when possible, etc.) I would definitely fund SIAI and advocate that others do so as well.
“transparency”? I thought the point of your post was that SIAI members should refrain from making some of their beliefs easily available to the public?
I see, maybe I should have been more clear. The point of my post is that SIAI members should not express controversial views without substantiating them with abundant evidence. If SIAI provided compelling evidence that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing, then I would think Eliezer’s comment appropriate.
As things stand SIAI has not provided such evidence. Eliezer himself may have such evidence, but if so he’s either unwilling or unable to share it.
There are a lot of second and higher order effects in PR. You can always shape your public statements for one audience and end up driving away (or failing to convince) another one that’s more important. If Eliezer had shied away from stating some of the more “uncredible” ideas because there wasn’t enough evidence to convince a typical smart person, it would surely prompt questions of “what do you really think about this?” or fail to attract people who are currently interested in SIAI because of those ideas.
If SIAI provided compelling evidence that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing, then I would think Eliezer’s comment appropriate.
Suppose Eliezer hadn’t made that claim, and somebody asks him, “do you think the work SIAI is doing has higher expected value to humanity than what everybody else is doing?”, which somebody is bound to, given that Eliezer is asking for donations from rationalists. What is he supposed to say? “I can’t give you the answer because I don’t have enough evidence to convince a typical smart person?”
I think you make a good point that it’s important to think about PR, but I’m not at all convinced that the specific advice you give are the right ones.
You can always shape your public statements for one audience and end up driving away (or failing to convince) another one that’s more important.
This is of course true. I myself am fairly certain that SIAI’s public statements are driving away the people who it’s most important to interest in existential risk.
Suppose Eliezer hadn’t made that claim, and somebody asks him, “do you think the work SIAI is doing has higher expected value to humanity than what everybody else is doing?”, which somebody is bound to, given that Eliezer is asking for donations from rationalists. What is he supposed to say? “I can’t give you the answer because I don’t have enough evidence to convince a typical smart person?”
•It’s standard public relations practice to reveal certain information only if asked.
•An organization that has the strongest case for room for more funding need not be an organization that’s doing something of higher expected value to humanity than what everybody else is doing. In particular, I simultaneously believe that there are politicians who have higher expected value to humanity than all existential risk researchers alive and that the cause of existential risk has the greatest room for more funding.
•One need not be confident in one’s belief that funding one’s organization has highest expected value to humanity to believe that funding one’s organization has highest expected to humanity. A major issue that I have with Eliezer’s rhetoric is that he projects what I perceive to be an unreasonably high degree of confidence in his beliefs.
•Another major issue with Eliezer’s rhetoric that I have is that even putting issues of PR aside, I personally believe that funding SIAI does not have anywhere near the highest expected value to humanity out of all possible uses of money. So from my point of view, I see no upside to Eliezer making extreme claims of the sort that he has—it looks to me as though Eliezer is making false claims and damaging public relations for existential risk as a result.
I will be detailing my reasons for thinking that SIAI’s research does not have high expected value in a future post.
The level of certainty is not up for grabs. You are as confident as you happen to be, this can’t be changed. You can change the appearance, but not your actual level of confidence. And changing the apparent level of confidence is equivalent to lying.
But it isn’t perceived as so by the general public—it seems to me that the usual perception of “confidence” has more to do with status than with probability estimates.
The non-technical people I work with often say that I use “maybe” and “probably” too much (I’m a programmer—“it’ll probably work” is a good description of how often it does work in practice) - as if having confidence in one’s statements was a sign of moral fibre, and not a sign of miscalibration.
Actually, making statements with high confidence is a positive trait, but most people address this by increasing the confidence they express, not by increasing their knowledge until they can honestly make high-confidence statements. And our culture doesn’t correct for that, because errors of calibration are not immediatly obvious (as they would be if, say, we had a widespread habit of betting on various things).
higher expected value to humanity than what virtually everybody else is doing,
For what definitions of “value to humanity” and “virtually everybody else”?
If “value to humanity” is assessed as in Bostrom’s Astronomical Waste paper, that hugely favors effects on existential risk vs alleviating current suffering or increasing present welfare (as such, those also have existential risk effects). Most people don’t agree with that view, so asserting that as a privileged frame can be seen as a hostile move (attacking the value systems of others in favor of a value system according to which one’s area of focus is especially important). Think of the anger directed at vegetarians, or those who guilt-trip others about not saving African lives. And of course, it’s easier to do well on a metric that others are mostly not focused on optimizing.
Dispute about what best reduces existential risk, and annoyance at overly confident statements there is a further issue, but I think that asserting uncommon moral principles (which happen to rank one’s activities as much more valuable than most people would rank them) is a big factor on its own.
In case my previous comment was ambiguous, I should say that I agree with you completely on this point. I’ve been wanting to make a top level post about this general topic for a while. Not sure when I’ll get a chance to do so.
Eliezer himself may have such evidence [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing], but if so he’s either unwilling or unable to share it.
Now that is unfair.
Since 1997, Eliezer has published (mostly on mailing lists and blogs but also in monographs) an enormous amount (at least ten novels worth unless I am very mistaken) of writings supporting exactly that point. Of course most of this material is technical, but unlike the vast majority of technical prose, it is accessible to non-specialists and non-initiates with enough intelligence, a solid undergraduate education as a “scientific generalist” and a lot of free time on their hands because in his writings Eliezer is constantly “watching out for” the reader who does not yet know what he knows. (In other words, it is uncommonly good technical exposition.)
So my impression has been that the situation is that
(i) Eliezer’s writings contain a great deal of insightful material.
(ii) These writings do not substantiate the idea that [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing].
I say this having read perhaps around a thousand pages of what Eliezer has written. I consider the amount of reading that I’ve done to be a good “probabilistic proof” that the points (i) and (ii) apply to the portion of his writings that I haven’t read.
That being said, if there are any particular documents that you would point me to which you feel do provide a satisfactory evidence for the idea [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing], I would be happy to examine them.
I’m unwilling to read the whole of his opus given how much of it I’ve already read without being convinced. I feel that the time that I put into reducing existential risk can be used to better effect in other ways.
It would help to know what steps in the probabilistic proof don’t have high probability for you.
For example, you might think that the singularity has a good probability of being relatively smooth and some kind of friendly, even without FAI. or you might think that other existential risks may still be a bigger threat, or you may think that Eliezer isn’t putting a dent in the FAI problem.
This might be a convenient place to collect a variety of reasons why people are FOOM denialists. From my POV:
I am skeptical that safeguards against UFAI (unFAI) will not work. In part because:
I doubt that the “takeoff” will be “hard”. Because:
I am pretty sure the takeoff will require repeatedly doubling and quadrupling hardware, not just autorewriting software.
And hence an effective safeguard would be to simply not give the machine its own credit card!
And in any case, the Moore’s law curve for electronics does not arise from delays in thinking up clever ideas, it arises from delays in building machines to incredibly high tolerances.
Furthermore, even after the machine has more hardware, it doesn’t yet have higher intelligence until it reads lots more encyclopedias and proves for itself many more theorems. These things take time.
And finally, I have yet to see the argument that an FAI protects us from a future UFAI. That is, how does the SIAI help us?
Oh, and I do think that the other existential risks, particularly war and economic collapse, put the UFAI risk pretty far down the priority list. Sure, those other risks may not be quite so existential, but if they don’t kill us, they will at least prevent an early singularity.
Edit added two days later: Since writing this, I thought about it some more, shut up for a moment, and did the math. I still think that it is unlikely that the first takeoff will be a hard one; so hard that it gets out of control. But I now estimate something like a 10% chance that the first takeoff will be hard, and I estimate something like a 30% chance that at least one of the first couple dozen takeoffs will be hard. Multiply that by an estimated 10% chance that a hard takeoff will take place without adequate safeguards in place, and another 10% chance that a safeguardless hard takeoff will go rogue, and you get something like a 0.3% chance of a disaster of Forbin Project magnitude. Completely unacceptable.
Originally, I had discounted the chance that a simple software change could cause the takeoff; I assumed you would need to double and redouble the hardware capability. What I failed to notice was that a simple “tuning” change to the (soft) network connectivity parameters—changing the maximum number of inputs per “neuron” from
8 to 7, say, could have an (unexpected) effect on performance of several orders of magnitude simply by suppressing wasteful thrashing or some such thing.
I am pretty sure the takeoff will require repeatedly doubling and quadrupling hardware, not just autorewriting software.
Do you think that progress in AI is limited primarily by hardware? If hardware is the limiting factor, then you should think AI soon relatively plausible. If software is the limiting factor (the majority view, and the reason most AI folk reject claims such as those of Moravec), such that we won’t get AI until well beyond the minimum computational requirements, then either early AIs should be able to run fast or with numerous copies cheaply, or there will be a lot of room to reduce bloated hardware demands through software improvements.
Thinking that AI will take a long time (during which hardware will advance mightily towards physical limits) but also be sharply and stably hardware-limited when created is a hard view to defend.
I am imagining that it will work something like the human brain (but not by ‘scan and emulate’). We need to create hardware modules comparable to neurons, we need to have some kind of geometric organization which permits individual hardware modules to establish physical connections to a handful of nearby modules, and we need a ‘program’ (corresponding to human embryonic development) which establishes a few starting connections, and finally we need a training period (like training a neural net, and comparable to what the human brain experiences from the first neural activity in the womb through graduate school) which adds many more physical connections. I’m not sure whether to call these connections hardware or software. Actually, they are a hybrid of both—like PLAs (yeah, I way out of date on technology).
So I’m imagining a lot of theoretical work needed to come up with a good ‘neuron’ design (probably several dozen different kinds of neurons), more theoretical work to come up with a good ‘program’ to correspond to the embryonic interconnect, and someone willing to pay for lots and lots of neurons.
So, yeah, I’m thinking that the program will be relatively simple (equivalent to a few million lines of code), but it will take us a long time to find it. Not the 500 million years that it took evolution to come up with that program—apparently 500 million years after it had already invented the neuron. But for human designers, at least a few decades to find and write the program. I hope this explanation helps to make my position seem less weird.
4 . And hence an effective safeguard would be to simply not give the machine its own credit card!
(Powerful) optimization processes can find such ways of solving problems by exploiting every possible shortcut that it is hard to predict those ways in advance. Recently here was an example of that. Genetic algorithm found unexpected solution of a problem exploiting analog properties of particular FPGA chip.
3 and 4: hardware, sure—that is improving too—just not as fast, sometimes. A machine may find a way to obtain a credit card—or it will get a human to buy whatever it needs—as happens in companies today.
6: how much time? Surely a better example would be: “perform experiments”—and experiments that caan’t be minaturised and executed at high speeds—such as those done in the LHC.
7: AltaVista didn’t protect us from Google—nor did Friendster protect against MySpace. However, so far Google has mostly successfully crushed its rivals.
8: no way, IMO—e.g. see Matt Ridley. That is probably good advice for all DOOMsters, actually.
Some of the most obvious safeguards are likely to be self-imposed ones:
Can you be more specific than “it’s somewhere beneath an enormous amount of 13 years of material from the very same person whose arguments are scrutinized for evidence”?
This is not sufficient to scare people up to the point of having nightmares and ask them for most of their money.
Do you want me to repeat the links people gave you 24 hours ago?
The person who was scared to the point of having nightmares was almost certainly on a weeks-long or months-long visit to the big house in California where people come to discuss extremely powerful technologies and the far future and to learn from experts on these subjects. That environment would tend to cause a person to take certain ideas more seriously than a person usually would.
Also, are we really discrediting people because they were foolish enough to talk about their deranged sleep-thoughts? I’d sound pretty stupid too if I remembered and advertised every bit of nonsense I experienced while sleeping.
It was more than one person. Anyway, I haven’t read all of the comments yet so I might have missed some specific links. If you are talking about links to articles written by EY himself where he argues about AI going FOOM, I commented on one of them.
Here is an example of the kind of transparency in the form of strict calculations, references and evidence I expect.
As I said, I’m not sure what other links you are talking about. But if you mean the kind of LW posts dealing with antipredictions, I’m not impressed. Predicting superhuman AI to be a possible outcome of AI research is not sufficient. Where is the difference between claiming the LHC will go FOOM? I’m sure someone like EY would be able to write a thousand posts around such a scenario telling me that the high risk associated with the LHC going FOOM does outweigh its low probability. There might be sound arguments to support this conclusion. But it is a conclusion and a framework of arguments based on a assumption that is itself of unknown credibility. So is it too much to ask for some transparet evidence to fortify this basic premise? Evidence that is not somewhere to be found within hundreds of posts not directly concerned with the evidence in question but rather arguing based on the very assumption it is trying to justify?
Asteroids really are an easier problem: celestial mechanics in vacuum are pretty stable, we have the Moon providing a record of past cratering to calibrate on, etc. There’s still uncertainty about the technology of asteroid deflection (e.g. its potential for military use, or to incite conflict), but overall it’s perhaps the most tractable risk for analysis since the asteroids themselves don’t depend on recent events (save for some smallish anthropic shadow effects).
An analysis for engineered pathogens, where we have a lot of uncertainty about the difficulty of engineering various of diseases for maximum damage, and how the technology for detection, treatment and prevention will keep pace. We can make generalizations based on existing diseases and their evolutionary dynamics (selection for lower virulence over time with person-to-person transmission, etc), current public health measures, etc, the rarity of the relevant motivations, etc, but you’re still left with many more places where you can’t just plug in well-established numbers and crank forward.
You can still give probability estimates, and plug in well-understood past data where you can, but you can’t get asteroid-level exactitude.
The difference is that we understand both asteroids and particle physics far better than we do intelligence, and there is precedence for both asteroid impacts and high energy particle collisions (natural ones at far higher energy than in the LHC) while there is none for an engineered human level intelligence with access to its own source code.
So calculations of the kind you seem to be asking for just aren’t possible at this point (and calculations with exactly that level of evidence won’t be possible right up until it’s too late), while refutations of the kind LHC panic gets aren’t possible either. You should also note that Eliezer takes LHC panic more serious than most non-innumerate people.
But if you want some calculation anyway: Let’s assume there is a 1% chance of extinction by uFAI within the next 100 years. Let’s also assume that spending $10 million per year (in 2010 dollars, adjusting for inflation) allows us to reduce that risk by 10%, just by the dangers of uFAI being in the public eye and people being somewhat more cautious, and taking the right sort of caution instead of worrying about Skynet or homicidal robots. So $1 billion saves about an expected 1 million lives, a cost of $ 1000 per life, which is about the level of the most efficient conventional charities. And that’s with Robins low-balling estimate (which was for a more specific case, not uFAI extinction in general, so even Robin would likely estimate a higher chance in the case considered) and assuming that FAI research won’t succeed.
So calculations of the kind you seem to be asking for just aren’t possible at this point …
I’m asking for whatever calculations should lead people to donate most of their money to the SIAI or get nightmares from stories of distant FAI’s. Surely there must be something to outweigh the lack of evidence, or on what basis has anyone decided to take things serious?
I really don’t want to anger you but the “let’s assume X” attitude is what I have my problems with here. A 1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
I wonder why it took so long until you came along posting this comment.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
Have you yet bothered to read e.g. this synopsis of SIAI’s position:
“Many AIs will converge toward being optimizing systems, in the sense that, after self-modification, they will act to maximize some goal. For instance, AIs developed under evolutionary pressures would be selected for values that maximized reproductive fitness, and would prefer to allocate resources to reproduction rather than supporting humans. Such unsafe AIs might actively mimic safe benevolence until they became powerful, since being destroyed would prevent them from working toward their goals. Thus, a broad range of AI designs may initially appear safe, but if developed to the point of a Singularity could cause human extinction in the course of optimizing the Earth for their goals.”
Personally, I think that presents a very weak case for there being risk. It argues that there could be risk if we built these machines wrong, and the bad machines became powerful somehow. That is true—but the reader is inclined to respond “so what”. A dam can be dangerous if you build it wrong too. Such observations don’t say very much about the actual risk.
I am very sceptical about that being true for those alive now:
We have been looking for things that might hit us for a long while now—and we can see much more clearly what the chances are for that period than by looking at the historical record. Also, that is apparently assuming no mitigation attempts—which also seems totally unrealistic.
...gives 700 deaths/year for aircraft—and 1,400 deaths/year for 2km impacts—based on assumption that one quarter of the human population would perish in such an impact.
I’m planning to fund FHI rather than SIAI, when I have a stable income (although my preference is for a different organisation that doesn’t exist)
My position is roughly this.
The nature of intelligence (and its capability for FOOMing) is poorly understood
The correct actions to take depend upon the nature of intelligence.
As such I would prefer to fund an institute that questioned the nature of intelligence, rather than one that has made up its mind that a singularity is the way forward. And it is not just the name that makes me think that SIAI has settled upon this view.
And because the nature of intelligence is the largest wild card in the future of humanity, I would prefer FHI to concentrate on that. Rather than longevity etc.
When I read good popular science books the people will tend to come up with some idea. Then they will test the idea to destruction. Poking and prodding at the idea until it really can’t be anything but what they say it is.
I want to get the same feeling off the group studying intelligence as I do from that type of research. They don’t need to be running foomable AIs, but truth is entangled so they should be able to figure out the nature of intelligence from other facets of the world, including physics and the biological examples.
Questions I hope they would be asking:
Is the g factor related to ability to absorb cultural information? I.e. is peoples increased ability to solve problems if they have a high g due to them being able to get more information about solving problems from cultural information sources?
If it wasn’t then it would be further evidence for .something special in one intelligence over another and it might make sense to call one more intelligent, rather than just having different initial skill sets.
If SIAI had the ethos I’d like, we’d be going over and kicking every one of the supporting arguments for the likelihood of fooming and the nature of intelligence to make sure they were sound. Performing experiments where necessary. However people have forgotten them and moved on to decision theory and the like.
Interesting points. Speaking only for myself, it doesn’t feel as though most of my problem solving or idea generating approaches were picked up from the culture, but I could be kidding myself.
For a different angle, here’s an old theory of Michael Vassar’s—I don’t know whether he still holds it. Talent consists of happening to have a reward system which happens to make doing the right thing feel good.
Talent consists of happening to have a reward system which happens to make doing the right thing feel good.
Definitely not just that. Knowing what the right thing is, and being able to do it before it’s too late, are also required. And talent implies a greater innate capacity for learning to do so. (I’m sure he meant in prospect, not retrospect).
It’s fair to say that some of what we identify as “talent” in people is actually in their motivations as well as their talent-requisite abilities.
If SIAI had the ethos I’d like, we’d be going over and kicking every one of the supporting arguments for the likelihood of fooming and the nature of intelligence to make sure they were sound.
And then, hypothetically, if they found that fooming is not likely at all, and that dangerous fooming can be rendered nearly impossible by some easily enforced precautions/regulations, what then? If they found that the SIAI has no particular unique expertise to contribute to the development of FAI? An organization with an ethos you would like: what would it do then? To make it a bit more interesting, suppose they find themselves sitting on a substantial endowment when they reason their way to their own obsolescence?
How often in human history have organizations announced, “Mission accomplished—now we will release our employees to go out and do something else”?
It doesn’t seem likely. The paranoid can usually find something scary to worry about. If something turns out to be not really-frightening, fear mongers can just go on to the next-most frightening thing in line. People have been concerned about losing their jobs to machines for over a century now. Machines are a big and scary enough domain to keep generating fear for a long time.
I think that what SIAI works on is real and urgent, but if I’m wrong and what you describe here does come to pass, the world gets yet another organisation campaigning about something no-one sane should care about. It doesn’t seem like a disastrous outcome.
From a less cynical angle, building organizations is hard. If an organization has fulfilled its purpose, or that purpose turns out to be a mistake, it isn’t awful to look for something useful for the organization to do rather than dissolving it.
The American charity organization, The March of Dimes was originally created to combat polio. Now they are involved with birth defects and other infant health issues.
Since they are the one case I know of (other than ad hoc disaster relief efforts) in which an organized charity accomplished its mission, I don’t begrudge them a few additional decades of corporate existence.
Will you do this?
I’m definitely interested in funding an existential risk organization. SIAI would have to be a lot more transparent than it is now right now for me to be interested in funding SIAI. For me personally, it wouldn’t be enough for SIAI to just take measures to avoid poisoning the meme, I would need to see a lot more evidence that SIAI is systematically working to maximize its impact on existential risk reduction.
As things stand I prefer to hold out for a better organization. But if SIAI exhibited transparency and accountability of levels similar to those of GiveWell (welcoming and publically responding to criticism regularly, regularly posting detailed plans of action, seeking out feedback from subject matter specialists and making this public when possible, etc.) I would definitely fund SIAI and advocate that others do so as well.
“transparency”? I thought the point of your post was that SIAI members should refrain from making some of their beliefs easily available to the public?
I see, maybe I should have been more clear. The point of my post is that SIAI members should not express controversial views without substantiating them with abundant evidence. If SIAI provided compelling evidence that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing, then I would think Eliezer’s comment appropriate.
As things stand SIAI has not provided such evidence. Eliezer himself may have such evidence, but if so he’s either unwilling or unable to share it.
There are a lot of second and higher order effects in PR. You can always shape your public statements for one audience and end up driving away (or failing to convince) another one that’s more important. If Eliezer had shied away from stating some of the more “uncredible” ideas because there wasn’t enough evidence to convince a typical smart person, it would surely prompt questions of “what do you really think about this?” or fail to attract people who are currently interested in SIAI because of those ideas.
Suppose Eliezer hadn’t made that claim, and somebody asks him, “do you think the work SIAI is doing has higher expected value to humanity than what everybody else is doing?”, which somebody is bound to, given that Eliezer is asking for donations from rationalists. What is he supposed to say? “I can’t give you the answer because I don’t have enough evidence to convince a typical smart person?”
I think you make a good point that it’s important to think about PR, but I’m not at all convinced that the specific advice you give are the right ones.
Thanks for your feedback. Several remarks:
This is of course true. I myself am fairly certain that SIAI’s public statements are driving away the people who it’s most important to interest in existential risk.
•It’s standard public relations practice to reveal certain information only if asked.
•An organization that has the strongest case for room for more funding need not be an organization that’s doing something of higher expected value to humanity than what everybody else is doing. In particular, I simultaneously believe that there are politicians who have higher expected value to humanity than all existential risk researchers alive and that the cause of existential risk has the greatest room for more funding.
•One need not be confident in one’s belief that funding one’s organization has highest expected value to humanity to believe that funding one’s organization has highest expected to humanity. A major issue that I have with Eliezer’s rhetoric is that he projects what I perceive to be an unreasonably high degree of confidence in his beliefs.
•Another major issue with Eliezer’s rhetoric that I have is that even putting issues of PR aside, I personally believe that funding SIAI does not have anywhere near the highest expected value to humanity out of all possible uses of money. So from my point of view, I see no upside to Eliezer making extreme claims of the sort that he has—it looks to me as though Eliezer is making false claims and damaging public relations for existential risk as a result.
I will be detailing my reasons for thinking that SIAI’s research does not have high expected value in a future post.
The level of certainty is not up for grabs. You are as confident as you happen to be, this can’t be changed. You can change the appearance, but not your actual level of confidence. And changing the apparent level of confidence is equivalent to lying.
But it isn’t perceived as so by the general public—it seems to me that the usual perception of “confidence” has more to do with status than with probability estimates.
The non-technical people I work with often say that I use “maybe” and “probably” too much (I’m a programmer—“it’ll probably work” is a good description of how often it does work in practice) - as if having confidence in one’s statements was a sign of moral fibre, and not a sign of miscalibration.
Actually, making statements with high confidence is a positive trait, but most people address this by increasing the confidence they express, not by increasing their knowledge until they can honestly make high-confidence statements. And our culture doesn’t correct for that, because errors of calibration are not immediatly obvious (as they would be if, say, we had a widespread habit of betting on various things).
That a lie is likely to be misinterpreted or not noticed doesn’t make it not a lie, and conversely.
Oh, I fully agree with your point; it’s a pity that high confidence on unusual topics is interpreted as arrogance.
Try this: I prefer my leaders to be confident. I prefer my subordinates to be truthful.
For what definitions of “value to humanity” and “virtually everybody else”?
If “value to humanity” is assessed as in Bostrom’s Astronomical Waste paper, that hugely favors effects on existential risk vs alleviating current suffering or increasing present welfare (as such, those also have existential risk effects). Most people don’t agree with that view, so asserting that as a privileged frame can be seen as a hostile move (attacking the value systems of others in favor of a value system according to which one’s area of focus is especially important). Think of the anger directed at vegetarians, or those who guilt-trip others about not saving African lives. And of course, it’s easier to do well on a metric that others are mostly not focused on optimizing.
Dispute about what best reduces existential risk, and annoyance at overly confident statements there is a further issue, but I think that asserting uncommon moral principles (which happen to rank one’s activities as much more valuable than most people would rank them) is a big factor on its own.
In case my previous comment was ambiguous, I should say that I agree with you completely on this point. I’ve been wanting to make a top level post about this general topic for a while. Not sure when I’ll get a chance to do so.
Now that is unfair.
Since 1997, Eliezer has published (mostly on mailing lists and blogs but also in monographs) an enormous amount (at least ten novels worth unless I am very mistaken) of writings supporting exactly that point. Of course most of this material is technical, but unlike the vast majority of technical prose, it is accessible to non-specialists and non-initiates with enough intelligence, a solid undergraduate education as a “scientific generalist” and a lot of free time on their hands because in his writings Eliezer is constantly “watching out for” the reader who does not yet know what he knows. (In other words, it is uncommonly good technical exposition.)
So my impression has been that the situation is that
(i) Eliezer’s writings contain a great deal of insightful material.
(ii) These writings do not substantiate the idea that [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing].
I say this having read perhaps around a thousand pages of what Eliezer has written. I consider the amount of reading that I’ve done to be a good “probabilistic proof” that the points (i) and (ii) apply to the portion of his writings that I haven’t read.
That being said, if there are any particular documents that you would point me to which you feel do provide a satisfactory evidence for the idea [that Eliezer’s work has higher expected value to humanity than what virtually everybody else is doing], I would be happy to examine them.
I’m unwilling to read the whole of his opus given how much of it I’ve already read without being convinced. I feel that the time that I put into reducing existential risk can be used to better effect in other ways.
It would help to know what steps in the probabilistic proof don’t have high probability for you.
For example, you might think that the singularity has a good probability of being relatively smooth and some kind of friendly, even without FAI. or you might think that other existential risks may still be a bigger threat, or you may think that Eliezer isn’t putting a dent in the FAI problem.
Or some combination of these and others.
Yes, I agree with you. I plan on making my detailed thoughts on these points explicit. I expect to be able to do so within a month.
But for a short answer, I would say that the situation is mostly that I think that:
This might be a convenient place to collect a variety of reasons why people are FOOM denialists. From my POV:
I am skeptical that safeguards against UFAI (unFAI) will not work. In part because:
I doubt that the “takeoff” will be “hard”. Because:
I am pretty sure the takeoff will require repeatedly doubling and quadrupling hardware, not just autorewriting software.
And hence an effective safeguard would be to simply not give the machine its own credit card!
And in any case, the Moore’s law curve for electronics does not arise from delays in thinking up clever ideas, it arises from delays in building machines to incredibly high tolerances.
Furthermore, even after the machine has more hardware, it doesn’t yet have higher intelligence until it reads lots more encyclopedias and proves for itself many more theorems. These things take time.
And finally, I have yet to see the argument that an FAI protects us from a future UFAI. That is, how does the SIAI help us?
Oh, and I do think that the other existential risks, particularly war and economic collapse, put the UFAI risk pretty far down the priority list. Sure, those other risks may not be quite so existential, but if they don’t kill us, they will at least prevent an early singularity.
Edit added two days later: Since writing this, I thought about it some more, shut up for a moment, and did the math. I still think that it is unlikely that the first takeoff will be a hard one; so hard that it gets out of control. But I now estimate something like a 10% chance that the first takeoff will be hard, and I estimate something like a 30% chance that at least one of the first couple dozen takeoffs will be hard. Multiply that by an estimated 10% chance that a hard takeoff will take place without adequate safeguards in place, and another 10% chance that a safeguardless hard takeoff will go rogue, and you get something like a 0.3% chance of a disaster of Forbin Project magnitude. Completely unacceptable.
Originally, I had discounted the chance that a simple software change could cause the takeoff; I assumed you would need to double and redouble the hardware capability. What I failed to notice was that a simple “tuning” change to the (soft) network connectivity parameters—changing the maximum number of inputs per “neuron” from 8 to 7, say, could have an (unexpected) effect on performance of several orders of magnitude simply by suppressing wasteful thrashing or some such thing.
Do you think that progress in AI is limited primarily by hardware? If hardware is the limiting factor, then you should think AI soon relatively plausible. If software is the limiting factor (the majority view, and the reason most AI folk reject claims such as those of Moravec), such that we won’t get AI until well beyond the minimum computational requirements, then either early AIs should be able to run fast or with numerous copies cheaply, or there will be a lot of room to reduce bloated hardware demands through software improvements.
Thinking that AI will take a long time (during which hardware will advance mightily towards physical limits) but also be sharply and stably hardware-limited when created is a hard view to defend.
I am imagining that it will work something like the human brain (but not by ‘scan and emulate’). We need to create hardware modules comparable to neurons, we need to have some kind of geometric organization which permits individual hardware modules to establish physical connections to a handful of nearby modules, and we need a ‘program’ (corresponding to human embryonic development) which establishes a few starting connections, and finally we need a training period (like training a neural net, and comparable to what the human brain experiences from the first neural activity in the womb through graduate school) which adds many more physical connections. I’m not sure whether to call these connections hardware or software. Actually, they are a hybrid of both—like PLAs (yeah, I way out of date on technology).
So I’m imagining a lot of theoretical work needed to come up with a good ‘neuron’ design (probably several dozen different kinds of neurons), more theoretical work to come up with a good ‘program’ to correspond to the embryonic interconnect, and someone willing to pay for lots and lots of neurons.
So, yeah, I’m thinking that the program will be relatively simple (equivalent to a few million lines of code), but it will take us a long time to find it. Not the 500 million years that it took evolution to come up with that program—apparently 500 million years after it had already invented the neuron. But for human designers, at least a few decades to find and write the program. I hope this explanation helps to make my position seem less weird.
(Powerful) optimization processes can find such ways of solving problems by exploiting every possible shortcut that it is hard to predict those ways in advance. Recently here was an example of that. Genetic algorithm found unexpected solution of a problem exploiting analog properties of particular FPGA chip.
7-8 aren’t hard-takeoff-denialist ideas; they’re SIAI noncontribution arguments. Good summary, though.
Phew! First, my material on the topic:
http://alife.co.uk/essays/the_singularity_is_nonsense/
http://alife.co.uk/essays/the_intelligence_explosion_is_happening_now/
Then a few points—which I may add to later.
3 and 4: hardware, sure—that is improving too—just not as fast, sometimes. A machine may find a way to obtain a credit card—or it will get a human to buy whatever it needs—as happens in companies today.
6: how much time? Surely a better example would be: “perform experiments”—and experiments that caan’t be minaturised and executed at high speeds—such as those done in the LHC.
7: AltaVista didn’t protect us from Google—nor did Friendster protect against MySpace. However, so far Google has mostly successfully crushed its rivals.
8: no way, IMO—e.g. see Matt Ridley. That is probably good advice for all DOOMsters, actually.
Some of the most obvious safeguards are likely to be self-imposed ones:
http://alife.co.uk/essays/stopping_superintelligence/
...though a resiliant infrastructure would help too. We see rogue agents (botnets) “eating” the internet today—and it is not very much fun!
Incidentally, a much better place for this kind of comment on this site would be:
http://lesswrong.com/lw/wf/hard_takeoff/
Can you be more specific than “it’s somewhere beneath an enormous amount of 13 years of material from the very same person whose arguments are scrutinized for evidence”?
This is not sufficient to scare people up to the point of having nightmares and ask them for most of their money.
Do you want me to repeat the links people gave you 24 hours ago?
The person who was scared to the point of having nightmares was almost certainly on a weeks-long or months-long visit to the big house in California where people come to discuss extremely powerful technologies and the far future and to learn from experts on these subjects. That environment would tend to cause a person to take certain ideas more seriously than a person usually would.
Also, are we really discrediting people because they were foolish enough to talk about their deranged sleep-thoughts? I’d sound pretty stupid too if I remembered and advertised every bit of nonsense I experienced while sleeping.
It was more than one person. Anyway, I haven’t read all of the comments yet so I might have missed some specific links. If you are talking about links to articles written by EY himself where he argues about AI going FOOM, I commented on one of them.
Here is an example of the kind of transparency in the form of strict calculations, references and evidence I expect.
As I said, I’m not sure what other links you are talking about. But if you mean the kind of LW posts dealing with antipredictions, I’m not impressed. Predicting superhuman AI to be a possible outcome of AI research is not sufficient. Where is the difference between claiming the LHC will go FOOM? I’m sure someone like EY would be able to write a thousand posts around such a scenario telling me that the high risk associated with the LHC going FOOM does outweigh its low probability. There might be sound arguments to support this conclusion. But it is a conclusion and a framework of arguments based on a assumption that is itself of unknown credibility. So is it too much to ask for some transparet evidence to fortify this basic premise? Evidence that is not somewhere to be found within hundreds of posts not directly concerned with the evidence in question but rather arguing based on the very assumption it is trying to justify?
Asteroids really are an easier problem: celestial mechanics in vacuum are pretty stable, we have the Moon providing a record of past cratering to calibrate on, etc. There’s still uncertainty about the technology of asteroid deflection (e.g. its potential for military use, or to incite conflict), but overall it’s perhaps the most tractable risk for analysis since the asteroids themselves don’t depend on recent events (save for some smallish anthropic shadow effects).
An analysis for engineered pathogens, where we have a lot of uncertainty about the difficulty of engineering various of diseases for maximum damage, and how the technology for detection, treatment and prevention will keep pace. We can make generalizations based on existing diseases and their evolutionary dynamics (selection for lower virulence over time with person-to-person transmission, etc), current public health measures, etc, the rarity of the relevant motivations, etc, but you’re still left with many more places where you can’t just plug in well-established numbers and crank forward.
You can still give probability estimates, and plug in well-understood past data where you can, but you can’t get asteroid-level exactitude.
The difference is that we understand both asteroids and particle physics far better than we do intelligence, and there is precedence for both asteroid impacts and high energy particle collisions (natural ones at far higher energy than in the LHC) while there is none for an engineered human level intelligence with access to its own source code.
So calculations of the kind you seem to be asking for just aren’t possible at this point (and calculations with exactly that level of evidence won’t be possible right up until it’s too late), while refutations of the kind LHC panic gets aren’t possible either. You should also note that Eliezer takes LHC panic more serious than most non-innumerate people.
But if you want some calculation anyway: Let’s assume there is a 1% chance of extinction by uFAI within the next 100 years. Let’s also assume that spending $10 million per year (in 2010 dollars, adjusting for inflation) allows us to reduce that risk by 10%, just by the dangers of uFAI being in the public eye and people being somewhat more cautious, and taking the right sort of caution instead of worrying about Skynet or homicidal robots. So $1 billion saves about an expected 1 million lives, a cost of $ 1000 per life, which is about the level of the most efficient conventional charities. And that’s with Robins low-balling estimate (which was for a more specific case, not uFAI extinction in general, so even Robin would likely estimate a higher chance in the case considered) and assuming that FAI research won’t succeed.
I’m asking for whatever calculations should lead people to donate most of their money to the SIAI or get nightmares from stories of distant FAI’s. Surely there must be something to outweigh the lack of evidence, or on what basis has anyone decided to take things serious?
I really don’t want to anger you but the “let’s assume X” attitude is what I have my problems with here. A 1% chance of extinction by uFAI? I just don’t see this, sorry. I can’t pull this out of my hat to make me believe either. I’m not saying this is wrong but I ask why there isn’t a detailed synopsis of this kind of estimations available? I think this is crucial.
So what’s the alternative?
You became aware of a possible danger. You didn’t think it up at random, so you can’t the heuristic that most complex hypotheses generated at random are wrong. There is no observational evidence, but the hypothesis doesn’t predict any observational evidence yet, so lack of evidence is no evidence against (like e.g. the lack of observation is against the danger of vampires). The best arguments for and against are about equally good (at least no order of magnitude differences). There seems to be a way to do something against the danger, but only before it manifests, that is before there can be any observational evidence either way. What do you do? Just assume that the danger is zero because that’s the default? Even though there is no particular reason to assume that’s a good heuristic in this particular case? (or do you think there are good reasons in this case? You mentioned the thought that it might be a scam, but it’s not like Eliezer invented the concept of hostile AIs).
The Bayesian way to deal with it would be to just use your prior (+ whatever evidence the arguments encountered provide, but the result probably mostly depends on your priors in this case). So this is a case where it’s OK to “just make numbers up”. It’s just that you should should make them up yourself, or rather base them on what you actually believe (if you can’t have experts you trust assess the issue and supply you with their priors). No one else can tell you what your priors are. The alternative to “just assuming” is “just assuming” zero, or one, or similar (or arbitrarily decide that everything that predicts observations that would be only 5% likely if it was false is true and everything without such observations is false, regardless of how many observations were actually made), purely based on context and how the questions are posed.
This is the kind of summary of a decision procedure I have been complaining about to be missing, or hidden within enormous amounts of content. I wish someone with enough skill could write a top-level post about it demanding that the SIAI creates an introductory paper exemplifying how to reach the conclusion that (1) the risks are to be taken seriously (2) you should donate to the SIAI to reduce the risks. There could either a be a few papers for different people with different backgrounds or one with different levels of detail. It should feature detailed references to what knowledge is necessary to understand the paper itself. Further it should feature the formulas, variables and decision procedures you have to follow to estimate the risks posed by and incentive to alleviate ufriendly AI. It should also include references to further information from people not associated with the SIAI.
This would allow for the transparency that is required by claims of this magnitude and calls for action, including donations.
I wonder why it took so long until you came along posting this comment.
You didn’t succeed in communicating your problem, otherwise someone else would have explained earlier. I had been reading your posts on the issue and didn’t have even the tiniest hint of an idea that the piece you were missing was an explanation of bayesian reasoning until just before writing that comment, and even then was less optimistic about the comment doing anything for you than I had been for earlier comments. I’m still puzzled and unsure whether it actually was Bayesian reasoning or something else in the comment that apparently helped you. if it was you should read http://yudkowsky.net/rational/bayes and some of the post here tagged “bayesian”.
Because thinking is work, and it’s not always obvious what question needs to be answered.
More generally (and this is something I’m still working on grasping fully). what’s obvious to you is not necessarily obvious to other people, even if you think you have enough in common with them that it’s hard to believe that they could have missed it.
I wouldn’t have said so even a week ago, but I’m now inclined to think that your short attention span is asset to LW.
Just as Eliezer has said (can someone remember the link?) that science as conventionally set up to be too leisurely (not enough thought put into coming up with good hypotheses), LW is set up on the assumption that people have a lot of time to put into the sequences and ability to remember what’s in them.
This isn’t quite what you’re talking about, but a relatively accessible intro doc:
http://singinst.org/riskintro/index.html
This seems like a summary of the idea of there being significant risk:
Anna Salamon at Singularity Summit 2009 - “Shaping the Intelligence Explosion”
http://www.vimeo.com/7318055
Good comment.
However,
This was hard to parse. I would have named “p-value” directly. My understanding is that a stated “p-value” will indeed depend on the number of observations, and that in practice meta-analyses pool the observations from many experiments. I agree that we should not use a hard p-value cutoff for publishing experimental results.
I should have said “a set of observations” and “sets of observations”. I meant things like that if you and other groups test lots of slightly different bogus hypotheses 5% of them will be “confirmed” with statistically significant relations.
Got it, and agreed. This is one of the most pernicious forms of dishonesty by professional researchers (lying about how many hypotheses were generated), and is far more common than merely faking everything.
Have you yet bothered to read e.g. this synopsis of SIAI’s position:
http://singinst.org/riskintro/index.html
I’d also strongly recommend this from Bostrom:
http://www.nickbostrom.com/fut/evolution.html
(Then of course there are longer and more comprehensive texts, which I won’t recommend because you would just continue to ignore them.)
The core of:
http://singinst.org/riskintro/
...that talks about risk appears to be:
“Many AIs will converge toward being optimizing systems, in the sense that, after self-modification, they will act to maximize some goal. For instance, AIs developed under evolutionary pressures would be selected for values that maximized reproductive fitness, and would prefer to allocate resources to reproduction rather than supporting humans. Such unsafe AIs might actively mimic safe benevolence until they became powerful, since being destroyed would prevent them from working toward their goals. Thus, a broad range of AI designs may initially appear safe, but if developed to the point of a Singularity could cause human extinction in the course of optimizing the Earth for their goals.”
Personally, I think that presents a very weak case for there being risk. It argues that there could be risk if we built these machines wrong, and the bad machines became powerful somehow. That is true—but the reader is inclined to respond “so what”. A dam can be dangerous if you build it wrong too. Such observations don’t say very much about the actual risk.
This calculation places no value on the future generations whose birth depends on averting existential risk. That’s not how I see things.
That claims that “that the lifetime risk of dying from an asteroid strike is about the same as the risk of dying in a commercial airplane crash”.
It cites:
Impacts on the Earth by asteroids and comets: assessing the hazard:
http://www.nature.com/nature/journal/v367/n6458/abs/367033a0.html
I am very sceptical about that being true for those alive now:
We have been looking for things that might hit us for a long while now—and we can see much more clearly what the chances are for that period than by looking at the historical record. Also, that is apparently assuming no mitigation attempts—which also seems totally unrealistic.
Looking further:
http://users.tpg.com.au/users/tps-seti/spacegd7.html
...gives 700 deaths/year for aircraft—and 1,400 deaths/year for 2km impacts—based on assumption that one quarter of the human population would perish in such an impact.
Yet, does the SIAI provide evidence on par with the paper I linked to?
What—about the chances of superintelligence causing THE END OF THE WORLD?!?
Of course not! How could they be expected to do that?
If there really was “abundant evidence” there probably wouldn’t be much of a controversy.
With machine intelligence, you probably want to be on the winning side—if that is possible.
Until it is clearer who that is going to be, many will want to hedge.
I’m planning to fund FHI rather than SIAI, when I have a stable income (although my preference is for a different organisation that doesn’t exist)
My position is roughly this.
The nature of intelligence (and its capability for FOOMing) is poorly understood
The correct actions to take depend upon the nature of intelligence.
As such I would prefer to fund an institute that questioned the nature of intelligence, rather than one that has made up its mind that a singularity is the way forward. And it is not just the name that makes me think that SIAI has settled upon this view.
And because the nature of intelligence is the largest wild card in the future of humanity, I would prefer FHI to concentrate on that. Rather than longevity etc.
What would the charity you’d like to contribute to look like?
When I read good popular science books the people will tend to come up with some idea. Then they will test the idea to destruction. Poking and prodding at the idea until it really can’t be anything but what they say it is.
I want to get the same feeling off the group studying intelligence as I do from that type of research. They don’t need to be running foomable AIs, but truth is entangled so they should be able to figure out the nature of intelligence from other facets of the world, including physics and the biological examples.
Questions I hope they would be asking:
Is the g factor related to ability to absorb cultural information? I.e. is peoples increased ability to solve problems if they have a high g due to them being able to get more information about solving problems from cultural information sources?
If it wasn’t then it would be further evidence for .something special in one intelligence over another and it might make sense to call one more intelligent, rather than just having different initial skill sets.
If SIAI had the ethos I’d like, we’d be going over and kicking every one of the supporting arguments for the likelihood of fooming and the nature of intelligence to make sure they were sound. Performing experiments where necessary. However people have forgotten them and moved on to decision theory and the like.
Interesting points. Speaking only for myself, it doesn’t feel as though most of my problem solving or idea generating approaches were picked up from the culture, but I could be kidding myself.
For a different angle, here’s an old theory of Michael Vassar’s—I don’t know whether he still holds it. Talent consists of happening to have a reward system which happens to make doing the right thing feel good.
Definitely not just that. Knowing what the right thing is, and being able to do it before it’s too late, are also required. And talent implies a greater innate capacity for learning to do so. (I’m sure he meant in prospect, not retrospect).
It’s fair to say that some of what we identify as “talent” in people is actually in their motivations as well as their talent-requisite abilities.
And then, hypothetically, if they found that fooming is not likely at all, and that dangerous fooming can be rendered nearly impossible by some easily enforced precautions/regulations, what then? If they found that the SIAI has no particular unique expertise to contribute to the development of FAI? An organization with an ethos you would like: what would it do then? To make it a bit more interesting, suppose they find themselves sitting on a substantial endowment when they reason their way to their own obsolescence?
How often in human history have organizations announced, “Mission accomplished—now we will release our employees to go out and do something else”?
It doesn’t seem likely. The paranoid can usually find something scary to worry about. If something turns out to be not really-frightening, fear mongers can just go on to the next-most frightening thing in line. People have been concerned about losing their jobs to machines for over a century now. Machines are a big and scary enough domain to keep generating fear for a long time.
I think that what SIAI works on is real and urgent, but if I’m wrong and what you describe here does come to pass, the world gets yet another organisation campaigning about something no-one sane should care about. It doesn’t seem like a disastrous outcome.
From a less cynical angle, building organizations is hard. If an organization has fulfilled its purpose, or that purpose turns out to be a mistake, it isn’t awful to look for something useful for the organization to do rather than dissolving it.
The American charity organization, The March of Dimes was originally created to combat polio. Now they are involved with birth defects and other infant health issues.
Since they are the one case I know of (other than ad hoc disaster relief efforts) in which an organized charity accomplished its mission, I don’t begrudge them a few additional decades of corporate existence.
I like this concept.
Assume your theory will fail in some places, and keep pressing it until it does, or you run out of ways to test it.
FHI?
The Future of Humanity Institute.
Nick Bostrom’s personal website probably gives you the best idea of what they produce.
A little too philosophical for my liking, but still interesting.