These two excerpts summarize where I disagree with SIAI:
Our needs and opportunities could change in a big way in the future. Right now we are still trying to lay the basic groundwork for a project to build an FAI. At the point where we had the right groundwork and the right team available, that project could cost several million dollars per year.
As to patents and commercially viable innovations—we’re not as sure about these. Our mission is ultimately to ensure that FAI gets built before UFAI; putting knowledge out there with general applicability for building AGI could therefore be dangerous and work directly against our mission.
So, SIAI plans to develop an AI that will take over the world, keeping their techniques secret, and therefore not getting critiques from the rest of the world.
This is WRONG. Horrendously, terrifyingly, irrationally wrong.
There are two major risks here. One is the risk of an arbitrarily-built AI, made not with Yudkowskian methodologies, whatever they will be, but with due diligence and precautions taken by the creators to not build something that will kill everybody.
The other is the risk of building a “FAI” that works, and then successfully becomes dictator of the universe for the rest of time, and this turns out more poorly than we had hoped.
I’m more afraid of the second than of the first. I find it implausible that it is harder to build an AI that doesn’t kill or enslave everybody, than to build an AI that does enslave everybody, in a way that wiser beings than us would agree was beneficial.
And I find it even more implausible, if the people building the one AI can get advice from everyone else in the world, while the people building the FAI do not.
An AI that is successfully “Friendly” poses an extistential risk of a kind that other AIs don’t pose. The main risk from an unfriendly AI is that it will kill all humans. That isn’t much of a risk; humans are on the way out in any case. Whereas the main risk from a “friendly” AI is that it will successfully impose a single set of values, defined by hairless monkeys, on the entire Universe until the end of time.
And, if you are afraid of unfriendly AI because you’re afraid it will kill you—why do you think that a “Friendly” AI is less likely to kill you? An “unfriendly” AI is following goals that probably appear random to us. There are arguments that it will inevitably take resources away from humans, but these are just that—arguments. Whereas a “friendly” AI will be designed to try to seize absolute power, and take every possible measure to prevent humans from creating another AI. If your name appears on this website, you’re already on its list of people whose continued existence will be risky.
(Also, all these numbers seem to be pulled out of thin air.)
I see no reason an AI with any other expansionist value system will not exhibit the exact same behaviour, except towards a different goal. There’s nothing so special about human values (except that they’re, y’know, good, but that’s a different issue).
You’re using a different definition of “friendly” than I am. An 80% chance SIAI’s AI is Unfriendly already contains all of your “takes over but messes everything up in unpredictable ways” scenarios.
The numbers were exaggerated for effect, to show contrast and my thought process. It seems to me that you think the probabilities are reversed.
The term “Friendly AI” refers to the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.
See the “non-human-harming” bit. Regarding:
If your name appears on this website, you’re already on its list of people whose continued existence will be risky.
Yes, one of their PR problems is that they are implicitly threatening their rivals. In the case of Ben Goertzel some of the threats are appearing IRL. Let us hear the tale of how threats and nastiness will be avoided. No plan is not a good plan, in this particular case.
An AI that is successfully “Friendly” poses an extistential risk of a kind that other AIs don’t pose. The main risk from an unfriendly AI is that it will kill all humans. That isn’t much of a risk
What do you mean by existential risk, then? I thought things that killed all humans were, by definition, existential risks.
humans are on the way out in any case.
What, if anything, do you value that you expect to exist in the long term?
There are arguments that [an UFAI] will inevitably take resources away from humans, but these are just that—arguments.
Pretty compelling arguments, IMO. It’s simple—the vast majority of goals can be achieved more easily if one has more resources, and humans control resources, so an entity that is able to self-improve will tend to seize control of all the resources and therefore take control of those resources from the humans.
Do you have a counterargument, or something relevant to the issue that isn’t just an argument?
AI will be designed to try to seize absolute power, and take every possible measure to prevent humans from creating another AI. If your name appears on this website, you’re already on its list of people whose continued existence will be risky.
Not much risk. Hunting down irrelevant blog commenters is a greater risk than leaving them be. There isn’t much of a window during which any human is a slightest threat and during that window going around killing people is just going to enhance the risk to it.
The window is presumably between now and when the winner is obvious—assuming we make it that far.
IMO, there’s plenty of scope for paranoia in the interim. Looking at the logic so far some teams will reason that unless their chosen values get implemented, much of value is likely to be lost. They will then mulitiply that by a billion years and a billion planets—and conclude that their competitors might really matter.
Killing people might indeed backfire—but that still leaves plenty of scope for dirty play.
Uh huh. So: world view difference. Corps and orgs will most likely go from 90% human to 90% machine through the well-known and gradual process of automation, gaining power as they go—and the threats from bad organisations are unlikely to be something that will appear suddenly at some point.
If we take those probabilities as a given, they strongly encourage a strategy that increases the chance that the first seed AI is Friendly.
jsalvatier already had a suggestion along those lines:
I wonder if SIAI could publicly discuss the values part of the AI without discussing the optimization part.
A public Friendly design could draw funding, benefit from technical collaboration, and hopefully end up used in whichever seed AI wins. Unfortunately, you’d have to decouple the F and AI parts, which is impossible.
I’m talking about publishing a technical design of Friendliness that’s conserved under self-improving optimization without also publishing (in math and code) exactly what is meant by self-improving optimization. CEV is a good first step, but a programmatically reusable solution it is not.
Before you the terrible blank wall stretches up and up and up, unimaginably far out of reach. And there is also the need to solve it, really solve it, not “try your best”.
I wonder if SIAI could publicly discuss the values part of the AI without discussing the optimization part. The values part seems to me (and from what I can tell, you too) where the most good would be done by public discussion while the optimization part seems to me where the danger lies if the information gets out.
I wonder if SIAI could publicly discuss the values part of the AI without discussing the optimization part. The values part seems to me (and from what I can tell, you too) where the most good would be done by public discussion while the optimization part seems to me where the danger lies if the information gets out.
Not honestly. When discussing values publicly you more or less have to spin bullshit. I would expect any public discussion the SIAI engaged in to be downright sickening to read and any interesting parts quickly censored. I’d much prefer no discussion at all—or discussion done by other people outside the influence or direct affiliation with the SIAI. That way the SIAI would not be obliged to distort or cripple the conversation for the sake of PR nor able to even if it wanted to.
CEV is one of the things which, if actually explored thoroughly, would definitely fit this description. As it is it is at the ‘bullshit border’. That is, a point at which you don’t yet have to trade off epistemic considerations in favor of signalling to the lowest common denominator. Because it is still credible that the not-superficially-nice parts just haven’t been covered yet—rather than being outright lied about.
I agree entirely with both of wedifrid’s comments above. Just read the CEV document, and ask, “If you were tasked with implementing this, how would you do it?” I tried unsuccessfully many times to elicit details from Eliezer on several points back on Overcoming Bias, until I concluded he did not want to go into those details.
One obvious question is, “The expected value calculations that I make from your stated beliefs indicate that your Friendly AI should prefer killing a billion people over taking a 10% chance that one of them is developing an AI; do you agree?” (If the answer is “no”, I suspect that is only due to time discounting of utility.)
Surely though if the FAI is in a position to be able to execute that action, it is in a position where it is so far ahead of an AI someone could be developing that it would have little fear of that possibility as a threat to CEV?
It won’t be very far ahead of an AI in realtime. The idea that the FAI can get far ahead, is based on the idea that it can develop very far in a “small” amount of time. Well, so can the new AI—and who’s to say it can’t develop 10 times as quickly as the FAI? So, how can a one-year-old FAI be certain that there isn’t an AI project that has been developed secretly 6 months ago and is about to overtake it in itelligence?
It is a somewhat complex issue, best understood by following what is (and isn’t) said in conversations along the lines of CEV (and sometimes metaethics) when the subject comes up. I believe the last time was a month or two ago in one of lukeprog’s posts.
Mind you this is a subject that would take a couple of posts to properly explore.
Because it is still credible that the not-superficially-nice parts just haven’t been covered yet—rather than being outright lied about.
Isn’t exploring the consequences of something like CEV pretty boring? Naively, the default scenario conditional on a large amount of background assumptions about relative optimization possible from various simulation scenarios et cetera is that the FAI fooms along possibly metaphysical spatiotemporal dimensions turning everything into acausal economic goodness. Once you get past the ‘oh no that means it kills everything I love’ part it’s basically a dead end. No? Note: the publicly acknowledged default scenario for a lot of smart people is a lot more PC than this. It’s probably not default for many people at all. I’m not confident in it.
The values part seems to me (and from what I can tell, you too) where the most good would be done by public discussion while the optimization part seems to me where the danger lies if the information gets out.
The problem is if one organisation with dubious values gets far ahead of everyone else. That situation is likely to be result of keeping secrets in this area.
Openness seems more likely to create a level playing field where the good guys have an excellent chance of winning. Those promoting secrecy are part of the problem here, IMO. I think we should leave the secret projects to the NSA and IARPA.
The history of IT shows many cases where use of closed solutions led to monopolies and problems. I think history shows that closed source solutions are mostly good for those selling them, but bad for the rest of society. IMO, we really don’t want machine intelligence to be like that.
The problem is if one organisation with dubious values gets far ahead of everyone else. That situation is likely to be result of keeping secrets in this area.
It’s likely to be the result of organizations with dubious values keeping secrets in this area. The good guys being open doesn’t make it better, it makes it worse, by giving the bad guys an asymmetric advantage.
The good guys want to form a large cooperatve network with each other, to help ensure they reach the goal first. Sharing is one of the primary ways they have of signalling to each other that they are good guys. Signalling must be expensive to be credible, and this is a nice, relevant, expensive signal. Being secretive—and failing to share—self-identifies yourself as a selfish bad guy—in the eyes of the sharers.
It is not an advantage to be recognised by good guys as a probable bad guy. For one thing, it most likey means you get no technical support.
A large cooperative good-guy network is a major win in terms of risk—compared to the scenario where everyone is secretive. The bad guys get some shared source code—but that in no way makes up for how much worse their position is overall.
To get ahead, the bad guys have to pretend to be good guys. To convince others of this—in the face of the innate human lie-detector abilities—they may even need to convince themselves they are good guys...
Personally, I think the benefits of openness win out in this case too.
That is especially true for the “inductive inference” side of things—which I estimate to be about 80% of the technical problem of machine intelligence. Keeping that secret is just a fantasy. Versions of that are going to be embedded in every library in every mobile computing device on the planet—doing input prediction, compression, and pattern completion. It is core infrastructure. You can’t hide things like that.
Essentially, you will have to learn to live with the possibility of bad guys using machine intelligence to help themselves. You can’t really stop that—so, don’t think that you can—and instead look into affecting what you can change—for example, reducing the opportunities for them to win, limiting the resulting damage, etc.
In this case, I’m less afraid of “bad guys” than I am of “good guys” who make mistakes. The bad guys just want to rule the Earth for a little while. The good guys want to define the Universe’s utility function.
I’m less afraid of “bad guys” than I am of “good guys” who make mistakes.
Looking at history of accidents with machines, they seem to be mostly automobile accidents. Medical accidents are number two, I think.
In both cases, technology that proved dangerous was used deliberately—before the relevant safety features could be added—due to the benefits it gave in the mean time. It seems likely that we will see more of that—in conjunction with the overall trend towards increased safety.
My position on this is the opposite of yours. I think there are probably greater individual risks from a machine intelligence working properly for someone else than there are from an accident. Both positions are players, though.
I find it implausible that it is harder to build an AI that doesn’t kill or enslave everybody, than to build an AI that does enslave everybody, in a way that wiser beings than us would agree was beneficial.
Why?
The SIAI claims they want to build an AI that asks what wiser beings than us would want (where the definition includes our values right before the AI gets the ability to alter our brains). They say it would look at you just as much as it looks at Eliezer in defining “wise”. And we don’t actually know it would “enslave everybody”. You think it would because you think a superhumanly bright AI that only cares about ‘wisdom’ so defined would do so, and this seems unwise to you. What do you mean by “wiser” that makes this seem logically coherent?
Those considerations obviously ignore the risk of bugs or errors in execution. But to this layman, bugs seem far more likely to kill us or simply break the AI than to hit that sweet spot (sour spot?) which keeps us alive in a way we don’t want. Which may or may not address your actual point, but certainly addresses the quote.
So, SIAI plans to develop an AI that will take over the world, keeping their techniques secret, and therefore not getting critiques from the rest of the world.
This is WRONG. Horrendously, terrifyingly, irrationally wrong.
It reminds me of this:
if we can make it all the way to Singularity without it ever becoming a “public policy” issue, I think maybe we should.
“Plan to Singularity” dates back to 2000. Other parties are now murmuring—but I wouldn’t say machine intelligence had yet become a “public policy” issue. I think it will, in due course though. So, I don’t think the original plan is very likely to pan out.
This is a good discussion. I see this whole issue as a power struggle, and I don’t consider the Singularity Institute to be more benevolent than anyone else just because Eliezer Yudkowsky has written a paper about “CEV” (whatever that is—I kept falling asleep when I tried to read it, and couldn’t make heads or tails of it in any case).
The megalomania of the SIAI crowd in claiming that they are the world-savers would worry me if I thought they might actually pull something off. For the sake of my peace of mind, I have formed an organization which is pursuing an AI world domination agenda of our own. At some point we might even write a paper explaining why our approach is the only ethically defensible means to save humanity from extermination. My working hypothesis is that AGI will be similar to nuclear weapons, in that it will be the culmination of a global power struggle (which has already started). Crazy old world, isn’t it?
The megalomania of the SIAI crowd in claiming that they are the world-savers would worry me if I thought they might actually pull something off.
I also think they look rather ineffectual from the outside. On the other hand they apparently keep much of their actual research secret—reputedly for fears that it will be used to do bad things—which makes them something of an unknown quantity.
I am pretty sceptical about them getting very far with their projects—but they are certainly making an interesting sociological phenomenon in the mean time!
These two excerpts summarize where I disagree with SIAI:
So, SIAI plans to develop an AI that will take over the world, keeping their techniques secret, and therefore not getting critiques from the rest of the world.
This is WRONG. Horrendously, terrifyingly, irrationally wrong.
There are two major risks here. One is the risk of an arbitrarily-built AI, made not with Yudkowskian methodologies, whatever they will be, but with due diligence and precautions taken by the creators to not build something that will kill everybody.
The other is the risk of building a “FAI” that works, and then successfully becomes dictator of the universe for the rest of time, and this turns out more poorly than we had hoped.
I’m more afraid of the second than of the first. I find it implausible that it is harder to build an AI that doesn’t kill or enslave everybody, than to build an AI that does enslave everybody, in a way that wiser beings than us would agree was beneficial.
And I find it even more implausible, if the people building the one AI can get advice from everyone else in the world, while the people building the FAI do not.
I think of it this way:
Chance SIAI’s AI is Unfriendly: 80%
Chance anyone else’s AI is Unfriendly: >99%
Chance SIAI builds their AI first: 10%
Chance SIAI builds their AI first while making all their designs public: <1% (no change to other probabilities)
An AI that is successfully “Friendly” poses an extistential risk of a kind that other AIs don’t pose. The main risk from an unfriendly AI is that it will kill all humans. That isn’t much of a risk; humans are on the way out in any case. Whereas the main risk from a “friendly” AI is that it will successfully impose a single set of values, defined by hairless monkeys, on the entire Universe until the end of time.
And, if you are afraid of unfriendly AI because you’re afraid it will kill you—why do you think that a “Friendly” AI is less likely to kill you? An “unfriendly” AI is following goals that probably appear random to us. There are arguments that it will inevitably take resources away from humans, but these are just that—arguments. Whereas a “friendly” AI will be designed to try to seize absolute power, and take every possible measure to prevent humans from creating another AI. If your name appears on this website, you’re already on its list of people whose continued existence will be risky.
(Also, all these numbers seem to be pulled out of thin air.)
I see no reason an AI with any other expansionist value system will not exhibit the exact same behaviour, except towards a different goal. There’s nothing so special about human values (except that they’re, y’know, good, but that’s a different issue).
You’re using a different definition of “friendly” than I am. An 80% chance SIAI’s AI is Unfriendly already contains all of your “takes over but messes everything up in unpredictable ways” scenarios.
The numbers were exaggerated for effect, to show contrast and my thought process. It seems to me that you think the probabilities are reversed.
One definition of the term explains:
See the “non-human-harming” bit. Regarding:
Yes, one of their PR problems is that they are implicitly threatening their rivals. In the case of Ben Goertzel some of the threats are appearing IRL. Let us hear the tale of how threats and nastiness will be avoided. No plan is not a good plan, in this particular case.
What do you mean by existential risk, then? I thought things that killed all humans were, by definition, existential risks.
What, if anything, do you value that you expect to exist in the long term?
Pretty compelling arguments, IMO. It’s simple—the vast majority of goals can be achieved more easily if one has more resources, and humans control resources, so an entity that is able to self-improve will tend to seize control of all the resources and therefore take control of those resources from the humans.
Do you have a counterargument, or something relevant to the issue that isn’t just an argument?
Not much risk. Hunting down irrelevant blog commenters is a greater risk than leaving them be. There isn’t much of a window during which any human is a slightest threat and during that window going around killing people is just going to enhance the risk to it.
The window is presumably between now and when the winner is obvious—assuming we make it that far.
IMO, there’s plenty of scope for paranoia in the interim. Looking at the logic so far some teams will reason that unless their chosen values get implemented, much of value is likely to be lost. They will then mulitiply that by a billion years and a billion planets—and conclude that their competitors might really matter.
Killing people might indeed backfire—but that still leaves plenty of scope for dirty play.
No. Reread the context. This is the threat from “F”AI, not from designers. The window opens when someone clicks ‘run’.
Uh huh. So: world view difference. Corps and orgs will most likely go from 90% human to 90% machine through the well-known and gradual process of automation, gaining power as they go—and the threats from bad organisations are unlikely to be something that will appear suddenly at some point.
If we take those probabilities as a given, they strongly encourage a strategy that increases the chance that the first seed AI is Friendly.
jsalvatier already had a suggestion along those lines:
A public Friendly design could draw funding, benefit from technical collaboration, and hopefully end up used in whichever seed AI wins. Unfortunately, you’d have to decouple the F and AI parts, which is impossible.
Isn’t CEV an attempt to separate F and AI parts?
It’s half of the F. Between the CEV and the AGI is the ‘goal stability under recursion’ part.
It’s a good first step.
I don’t understand your impossibility comment, then.
I’m talking about publishing a technical design of Friendliness that’s conserved under self-improving optimization without also publishing (in math and code) exactly what is meant by self-improving optimization. CEV is a good first step, but a programmatically reusable solution it is not.
On doing the impossible:
OK, I understand that much better now. Great point.
I wonder if SIAI could publicly discuss the values part of the AI without discussing the optimization part. The values part seems to me (and from what I can tell, you too) where the most good would be done by public discussion while the optimization part seems to me where the danger lies if the information gets out.
Not honestly. When discussing values publicly you more or less have to spin bullshit. I would expect any public discussion the SIAI engaged in to be downright sickening to read and any interesting parts quickly censored. I’d much prefer no discussion at all—or discussion done by other people outside the influence or direct affiliation with the SIAI. That way the SIAI would not be obliged to distort or cripple the conversation for the sake of PR nor able to even if it wanted to.
CEV doesn’t seem to fit this description.
CEV is one of the things which, if actually explored thoroughly, would definitely fit this description. As it is it is at the ‘bullshit border’. That is, a point at which you don’t yet have to trade off epistemic considerations in favor of signalling to the lowest common denominator. Because it is still credible that the not-superficially-nice parts just haven’t been covered yet—rather than being outright lied about.
Do you have evidence for this proposition?
I agree entirely with both of wedifrid’s comments above. Just read the CEV document, and ask, “If you were tasked with implementing this, how would you do it?” I tried unsuccessfully many times to elicit details from Eliezer on several points back on Overcoming Bias, until I concluded he did not want to go into those details.
One obvious question is, “The expected value calculations that I make from your stated beliefs indicate that your Friendly AI should prefer killing a billion people over taking a 10% chance that one of them is developing an AI; do you agree?” (If the answer is “no”, I suspect that is only due to time discounting of utility.)
Surely though if the FAI is in a position to be able to execute that action, it is in a position where it is so far ahead of an AI someone could be developing that it would have little fear of that possibility as a threat to CEV?
It won’t be very far ahead of an AI in realtime. The idea that the FAI can get far ahead, is based on the idea that it can develop very far in a “small” amount of time. Well, so can the new AI—and who’s to say it can’t develop 10 times as quickly as the FAI? So, how can a one-year-old FAI be certain that there isn’t an AI project that has been developed secretly 6 months ago and is about to overtake it in itelligence?
It is a somewhat complex issue, best understood by following what is (and isn’t) said in conversations along the lines of CEV (and sometimes metaethics) when the subject comes up. I believe the last time was a month or two ago in one of lukeprog’s posts.
Mind you this is a subject that would take a couple of posts to properly explore.
Isn’t exploring the consequences of something like CEV pretty boring? Naively, the default scenario conditional on a large amount of background assumptions about relative optimization possible from various simulation scenarios et cetera is that the FAI fooms along possibly metaphysical spatiotemporal dimensions turning everything into acausal economic goodness. Once you get past the ‘oh no that means it kills everything I love’ part it’s basically a dead end. No? Note: the publicly acknowledged default scenario for a lot of smart people is a lot more PC than this. It’s probably not default for many people at all. I’m not confident in it.
I don’t really understand what this means, so I don’t see why the next bit follows. Could you break this down, preferably using simpler terms?
The problem is if one organisation with dubious values gets far ahead of everyone else. That situation is likely to be result of keeping secrets in this area.
Openness seems more likely to create a level playing field where the good guys have an excellent chance of winning. Those promoting secrecy are part of the problem here, IMO. I think we should leave the secret projects to the NSA and IARPA.
The history of IT shows many cases where use of closed solutions led to monopolies and problems. I think history shows that closed source solutions are mostly good for those selling them, but bad for the rest of society. IMO, we really don’t want machine intelligence to be like that.
Many governments realise the significance of open source software these days—e.g. see: The government gets really serious about open source.
It’s likely to be the result of organizations with dubious values keeping secrets in this area. The good guys being open doesn’t make it better, it makes it worse, by giving the bad guys an asymmetric advantage.
We discussed this very recently.
The good guys want to form a large cooperatve network with each other, to help ensure they reach the goal first. Sharing is one of the primary ways they have of signalling to each other that they are good guys. Signalling must be expensive to be credible, and this is a nice, relevant, expensive signal. Being secretive—and failing to share—self-identifies yourself as a selfish bad guy—in the eyes of the sharers.
It is not an advantage to be recognised by good guys as a probable bad guy. For one thing, it most likey means you get no technical support.
A large cooperative good-guy network is a major win in terms of risk—compared to the scenario where everyone is secretive. The bad guys get some shared source code—but that in no way makes up for how much worse their position is overall.
To get ahead, the bad guys have to pretend to be good guys. To convince others of this—in the face of the innate human lie-detector abilities—they may even need to convince themselves they are good guys...
You never did address the issue I raised in the linked comment. As far as I can tell, it’s a showstopper for open-access development models of AI.
You gave some disadvantages of openness—I responded with a list of advantages of openness. Why you concluded this was not responsive is not clear.
Conventional wisdom about open source and security is that it helps—e.g. see Bruce Schneier on the topic.
Personally, I think the benefits of openness win out in this case too.
That is especially true for the “inductive inference” side of things—which I estimate to be about 80% of the technical problem of machine intelligence. Keeping that secret is just a fantasy. Versions of that are going to be embedded in every library in every mobile computing device on the planet—doing input prediction, compression, and pattern completion. It is core infrastructure. You can’t hide things like that.
Essentially, you will have to learn to live with the possibility of bad guys using machine intelligence to help themselves. You can’t really stop that—so, don’t think that you can—and instead look into affecting what you can change—for example, reducing the opportunities for them to win, limiting the resulting damage, etc.
What linked comment?
The first comment here, I believe.
In this case, I’m less afraid of “bad guys” than I am of “good guys” who make mistakes. The bad guys just want to rule the Earth for a little while. The good guys want to define the Universe’s utility function.
Looking at history of accidents with machines, they seem to be mostly automobile accidents. Medical accidents are number two, I think.
In both cases, technology that proved dangerous was used deliberately—before the relevant safety features could be added—due to the benefits it gave in the mean time. It seems likely that we will see more of that—in conjunction with the overall trend towards increased safety.
My position on this is the opposite of yours. I think there are probably greater individual risks from a machine intelligence working properly for someone else than there are from an accident. Both positions are players, though.
Now I’m confused again. Who do you worry about if not the NSA?
I’m having a hard time parsing what that last clause refers to; what is supposed to be better, enslaving or not enslaving?
Why?
The SIAI claims they want to build an AI that asks what wiser beings than us would want (where the definition includes our values right before the AI gets the ability to alter our brains). They say it would look at you just as much as it looks at Eliezer in defining “wise”. And we don’t actually know it would “enslave everybody”. You think it would because you think a superhumanly bright AI that only cares about ‘wisdom’ so defined would do so, and this seems unwise to you. What do you mean by “wiser” that makes this seem logically coherent?
Those considerations obviously ignore the risk of bugs or errors in execution. But to this layman, bugs seem far more likely to kill us or simply break the AI than to hit that sweet spot (sour spot?) which keeps us alive in a way we don’t want. Which may or may not address your actual point, but certainly addresses the quote.
It reminds me of this:
http://yudkowsky.net/obsolete/plan.html
The plan to steal the singularity.
Any other plan would be insane! (Or, at least, only sane as a second choice when stealing seems impractical.)
Uh huh. You don’t think some other parties might prefer to be consulted?
A plan to pull this off before the other parties wake up may set off some alarm bells.
… The kind of thing that makes ‘just do it’ seem impractical?
“Plan to Singularity” dates back to 2000. Other parties are now murmuring—but I wouldn’t say machine intelligence had yet become a “public policy” issue. I think it will, in due course though. So, I don’t think the original plan is very likely to pan out.
This is a good discussion. I see this whole issue as a power struggle, and I don’t consider the Singularity Institute to be more benevolent than anyone else just because Eliezer Yudkowsky has written a paper about “CEV” (whatever that is—I kept falling asleep when I tried to read it, and couldn’t make heads or tails of it in any case).
The megalomania of the SIAI crowd in claiming that they are the world-savers would worry me if I thought they might actually pull something off. For the sake of my peace of mind, I have formed an organization which is pursuing an AI world domination agenda of our own. At some point we might even write a paper explaining why our approach is the only ethically defensible means to save humanity from extermination. My working hypothesis is that AGI will be similar to nuclear weapons, in that it will be the culmination of a global power struggle (which has already started). Crazy old world, isn’t it?
I also think they look rather ineffectual from the outside. On the other hand they apparently keep much of their actual research secret—reputedly for fears that it will be used to do bad things—which makes them something of an unknown quantity.
I am pretty sceptical about them getting very far with their projects—but they are certainly making an interesting sociological phenomenon in the mean time!