Ask yourself, what difference would you expect to see if Dr. Evil would disguise as Eliezer Yudkowsky? Why wouldn’t he write the sequences, why wouldn’t he claim to be implementing CEV?
In that case, I would expect a stupid Eliezer Yudkowsky. But one shouldn’t actually reason this way, the question is, what do you anticipate, given observations actually made; not how plausible are the observations actually made, given an uncaused hypothesis.
In that case, I would expect a stupid Eliezer Yudkowsky
Why is evil stupid and what evidence is there that Yudkowsky is smart enough not to be evil?
But one shouldn’t actually reason this way, the question is, what do you anticipate, given observations actually made; not how plausible are the observations actually made, given an uncaused hypothesis.
If you got someone working on friendly AI you better ask if the person is friendly in the first place. You also shouldn’t make conclusions based on the output of the subject of your conclusions. If Yudkowsky states what is right and states that he will do what is right that provides no evidence about the rightness and honesty of those statements. Besides, the most advanced statements about Yudkowsky’s intentions are CEV and the meta-ethics sequence. Both are either criticized or not understood.
The question should be, what is the worst-case scenario regarding Yudkowsky and the SIAI and how can we discern it from what he is signaling? If the answer isn’t clear, one should ask for transparency and oversight.
You seem to be under the impression that Eliezer is going to create an artificial general intelligence, and oversight is necessary to ensure that he doesn’t create one which places his goals over humanity’s interests. It is important, you say, that he is not allowed unchecked power. This is all fine, except for one very important fact that you’ve missed.
Eliezer Yudkowsky can’t program. He’s never published a nontrivial piece of software, and doesn’t spend time coding. In the one way that matters, he’s a muggle. Ineligible to write an AI. Eliezer has not positioned himself to be the hero, the one who writes the AI or implements its utility function. The hero, if there is to be one, has not yet appeared on stage. No, Eliezer has positioned himself to be the mysterious old wizard—to lay out a path, and let someone else follow it. You want there to be oversight over Eliezer, and Eliezer wants to be the oversight over someone else to be determined.
But maybe we shouldn’t trust Eliezer to be the mysterious old wizard, either. If the hero/AI programmer comes to him with a seed AI, then he knows it exists, and finding out that a seed AI exists before it launches is the hardest part of any plan to steal it and rewrite its utility function to conquer the universe. That would be pretty evil, but would “transparency and oversight” make things turn out better, or worse? As far as I can tell, transparency would mean announcing the existence of a pre-launch AI to the world. This wouldn’t stop Eliezer from make a play to conquer the universe, but it would present that option to everybody else, including at least some people and organizations who are definitely evil.
So that’s a bad plan. A better plan would be to write a seed AI yourself, keep it secret from Eliezer, and when it’s time to launch, ask for my input instead.
(For the record: I’ve programmed in C++, Python, Java, wrote some BASIC programs on a ZX80 when I was 5 or 6, and once very briefly when MacOS System 6 required it I wrote several lines of a program in 68K assembly. I admit I haven’t done much coding recently, due to other comparative advantages beating that one out.)
Sounds about right. It wasn’t good code, I was young and working alone. Though it’s more like the code was strategically stupid than locally poorly written.
If the hero/AI programmer comes to him with a seed AI, then he knows it exists, and finding out that a seed AI exists before it launches is the hardest part of any plan to steal it and rewrite its utility function to conquer the universe.
I’m not aware of any reason to believe that recursively self-improving artificial general intelligence is going to be something you can ‘run away with’. It looks like some people here think so, that there will be some kind of, with hindsight, simple algorithm for intelligence that people can just run and get superhuman intelligence. Indeed, transparency could be very dangerous in that case. But that doesn’t mean it is an all or nothing decision. There are many other reasons for transparency, including reassurance and the ability to discern a trickster or impotent individual from someone who deserves more money. But as I said, I don’t see that anyway. It’ll more likely be a blue sheet of different achievements that are each not dangerous on their own. I further think it will be not just a software solution but also a conceptual and computational revolution. In those cases an open approach will allow public oversight. And even if someone is going to run with it, you want them to use your solution rather than one that will most certainly be unfriendly.
Evil is not necessarily stupid (well, it is, if we are talking about humans, but let’s abstract from that). Still, it would take a stupid Dr Evil to decide that pretending to be Eliezer Yudkowsky is the best available course of action.
You don’t think that being Eliezer Yudkowsky is an effective way to accomplish the task at hand? What should Dr Evil do, then?
FWIW, my usual comparison is not with Dr Evil, but with Gollum. The Singularity Instutute have explicitly stated said they are trying to form “The Fellowship of the AI”. Obviously we want to avoid Gollum’s final scene.
Gollum actually started out good—it was the exposure to the ring that caused problems later on.
what is the worst-case scenario regarding Yudkowsky and the SIAI and how can we discern it from what he is signaling? If the answer isn’t clear, one should ask for transparency and oversight.
It will become increasingly important to develop AI algorithms that are not just powerful and scalable, but also transparent to inspection—to name one of many socially important properties.
However, apparently, this doesn’t seem to mean open source software—e.g. here:
the Singularity Institute does not currently plan to develop via an open-source method
Uh, what? Transparency gets listed as a “socially important” virtue in the PR documents—but the plans apparently involve keeping the source code secret.
He means “transparent” as in “you can read its plans in the log files/with a debugger”, not as in “lots of people have access”. Transparency in the former sense is a good thing, since it lets the programmer verify that it’s sane and performing as expected. Transparency in the latter sense is a bad thing, because if lots of people had access then there would be no one with the power to say the AI wasn’t safe to run or give extra hardware, since anyone could take a copy and run it themselves.
Full transparency—with lots of people having access—is desirable from society’s point of view. Then, there are more eyes looking for flaws in the code—which makes is safer. Also, then, society can watch to ensure development is going along the right lines. This is likely to make the developers behave bettter, and having access to the code gives society the power to collectively protect itself aginst wrongdoers.
The most likely undesirable situation involves copyrighted/patented/secret server side machine intelligence sucking resources to benefit a minority at the expense of the rest of society. This is a closed-source scenario—and that isn’t an accident. Being able to exploit others for your own benefit is one of the most common reasons for secrecy.
EMACS is a powerful tool—but we do not keep it secret because the mafia might use it to their own advantage. It is better all round that everyone has access, rather than just an elite. Both society and EMACS itself are better because of that strategy.
The NSA is one of the more well-known examples of it being tried with some success. There we have a large organisation (many eyeballs inside mean diminishing returns from extra eyeballs) - and one with government backing. Despite this, the NSA often faces allegations of secretive, unethical behaviour.
I said that public access to an AI under development would be bad, because if it wasn’t safe to run—that is, if running it might cause it too foom and destroy the world—then no one would be able to make that judgment and keep others from running it. You responded with an analogy to EMACS, which no one believes or has ever believed to be dangerous, and which has no potential to do disastrous things that their operators did not intend. So that analogy is really a non sequitur.
“Dangerous” in this context does not mean “powerful”, it means “volatile”, as in “reacts explosively with Pentiums”.
Both types of software are powerful tools. Powerful tools are dangerous in the wrong hands, because they amplify the power of their users. That is the gist of the analogy.
I expect EMACS has been used for all kinds of evil purposes, from writing viruses, trojans, and worms to tax evasion and fraud.
That seems rather dubious as a general motto, but in this case, I am inclined to agree. In the case of intelligent machines, the positives of openness substantially outweigh their negatives, IMO.
Budding machine intelligence builders badly need to signal that they are not going to screw everyone over. How else are other people to know that they are not planning to screw everyone over?
Such signals should be expensive and difficult to fake. In this case, about the only credible signal is maximum transpareny. I am not going to screw you over, and look, here is the proof: what’s mine is yours.
If you don’t understand something I’ve written, please ask for clarification. Don’t guess what I said and respond to that instead; that’s obnoxious. Your comparison of my argument to
“Otherwise the terrorists will win!”
Leads me to believe that you didn’t understand what I said at all. How is destroying the world by accident like terrorism?
Er, characterising someone who disagrees with you on a technical point as “obnoxious” is not terribly great manners in itself! I never compared destroying the world by accident with terrorism—you appear to be projecting. However, I am not especially interested in the conversation being dragged into the gutter in this way.
If you did have a good argument favouring closed source software and reduced transparency I think there has been a reasonable chance to present it. However, if you can’t even be civil, perhaps you should consider waiting until you can.
I gave an argument that open-sourcing AI would increase the risk of the world being destroyed by accident. You said
I note that Anders Sandberg recently included: “Otherwise the terrorists will win!” …in his list of of signs that you might be looking at a weak moral argument.
I presented the mismatch between this statement and my argument as evidence that you had misunderstood what I was saying. In your reply,
I never compared destroying the world by accident with terrorism—you appear to be projecting.
You are misunderstanding me again. I think I’ve already said all that needs to be said, but I can’t clear up confusion if you keep attacking straw men rather than asking questions.
You are confusing socially important with societally important. Microsoft, for example, seeks to have its source code transparent to inspection, because Microsoft, as a corporate culture, produces software socially—that is, utilizing an evil conspiracy involving many communicating agents.
I deny confusing anything. I understand that transparency can be a matter of degree and perspective. What I am pointing out is lip-service to transparency. Full transparency would be different.
Microsoft’s software is not very transparent—and partly as a result it is some of the most badly-designed, insecure and virus-ridden software the planet has ever seen. We can see the mistake, can see its consequences—and know how to avoid it—but we have to, like actually do that—and that involves some alerting of others to the problems often associated with closed-source proposals.
In that case, I would expect a stupid Eliezer Yudkowsky. But one shouldn’t actually reason this way, the question is, what do you anticipate, given observations actually made; not how plausible are the observations actually made, given an uncaused hypothesis.
You can’t compute P(H|E) without computing P(E|H).
But one shouldn’t confuse the two.
What’s an uncaused hypothesis? And didn’t you just accidentally forbid people to think properly?
Why is evil stupid and what evidence is there that Yudkowsky is smart enough not to be evil?
If you got someone working on friendly AI you better ask if the person is friendly in the first place. You also shouldn’t make conclusions based on the output of the subject of your conclusions. If Yudkowsky states what is right and states that he will do what is right that provides no evidence about the rightness and honesty of those statements. Besides, the most advanced statements about Yudkowsky’s intentions are CEV and the meta-ethics sequence. Both are either criticized or not understood.
The question should be, what is the worst-case scenario regarding Yudkowsky and the SIAI and how can we discern it from what he is signaling? If the answer isn’t clear, one should ask for transparency and oversight.
You seem to be under the impression that Eliezer is going to create an artificial general intelligence, and oversight is necessary to ensure that he doesn’t create one which places his goals over humanity’s interests. It is important, you say, that he is not allowed unchecked power. This is all fine, except for one very important fact that you’ve missed.
Eliezer Yudkowsky can’t program. He’s never published a nontrivial piece of software, and doesn’t spend time coding. In the one way that matters, he’s a muggle. Ineligible to write an AI. Eliezer has not positioned himself to be the hero, the one who writes the AI or implements its utility function. The hero, if there is to be one, has not yet appeared on stage. No, Eliezer has positioned himself to be the mysterious old wizard—to lay out a path, and let someone else follow it. You want there to be oversight over Eliezer, and Eliezer wants to be the oversight over someone else to be determined.
But maybe we shouldn’t trust Eliezer to be the mysterious old wizard, either. If the hero/AI programmer comes to him with a seed AI, then he knows it exists, and finding out that a seed AI exists before it launches is the hardest part of any plan to steal it and rewrite its utility function to conquer the universe. That would be pretty evil, but would “transparency and oversight” make things turn out better, or worse? As far as I can tell, transparency would mean announcing the existence of a pre-launch AI to the world. This wouldn’t stop Eliezer from make a play to conquer the universe, but it would present that option to everybody else, including at least some people and organizations who are definitely evil.
So that’s a bad plan. A better plan would be to write a seed AI yourself, keep it secret from Eliezer, and when it’s time to launch, ask for my input instead.
(For the record: I’ve programmed in C++, Python, Java, wrote some BASIC programs on a ZX80 when I was 5 or 6, and once very briefly when MacOS System 6 required it I wrote several lines of a program in 68K assembly. I admit I haven’t done much coding recently, due to other comparative advantages beating that one out.)
I can’t find it by search, but haven’t you stated that you’ve written hundreds of KLOC?
Yep, he have.
Sounds about right. It wasn’t good code, I was young and working alone. Though it’s more like the code was strategically stupid than locally poorly written.
I disagree based on the following evidence:
You further write:
I’m not aware of any reason to believe that recursively self-improving artificial general intelligence is going to be something you can ‘run away with’. It looks like some people here think so, that there will be some kind of, with hindsight, simple algorithm for intelligence that people can just run and get superhuman intelligence. Indeed, transparency could be very dangerous in that case. But that doesn’t mean it is an all or nothing decision. There are many other reasons for transparency, including reassurance and the ability to discern a trickster or impotent individual from someone who deserves more money. But as I said, I don’t see that anyway. It’ll more likely be a blue sheet of different achievements that are each not dangerous on their own. I further think it will be not just a software solution but also a conceptual and computational revolution. In those cases an open approach will allow public oversight. And even if someone is going to run with it, you want them to use your solution rather than one that will most certainly be unfriendly.
Evil is not necessarily stupid (well, it is, if we are talking about humans, but let’s abstract from that). Still, it would take a stupid Dr Evil to decide that pretending to be Eliezer Yudkowsky is the best available course of action.
You don’t think that being Eliezer Yudkowsky is an effective way to accomplish the task at hand? What should Dr Evil do, then?
FWIW, my usual comparison is not with Dr Evil, but with Gollum. The Singularity Instutute have explicitly stated said they are trying to form “The Fellowship of the AI”. Obviously we want to avoid Gollum’s final scene.
Gollum actually started out good—it was the exposure to the ring that caused problems later on.
I seem to remember Smeagol being an unpleasant chap even before Deagol found the ring. But admittedly, we weren’t given much.
Transparency is listed as being desirable here:
However, apparently, this doesn’t seem to mean open source software—e.g. here:
You equivocate two unrelated senses of “transparency”.
Uh, what? Transparency gets listed as a “socially important” virtue in the PR documents—but the plans apparently involve keeping the source code secret.
He means “transparent” as in “you can read its plans in the log files/with a debugger”, not as in “lots of people have access”. Transparency in the former sense is a good thing, since it lets the programmer verify that it’s sane and performing as expected. Transparency in the latter sense is a bad thing, because if lots of people had access then there would be no one with the power to say the AI wasn’t safe to run or give extra hardware, since anyone could take a copy and run it themselves.
Full transparency—with lots of people having access—is desirable from society’s point of view. Then, there are more eyes looking for flaws in the code—which makes is safer. Also, then, society can watch to ensure development is going along the right lines. This is likely to make the developers behave bettter, and having access to the code gives society the power to collectively protect itself aginst wrongdoers.
The most likely undesirable situation involves copyrighted/patented/secret server side machine intelligence sucking resources to benefit a minority at the expense of the rest of society. This is a closed-source scenario—and that isn’t an accident. Being able to exploit others for your own benefit is one of the most common reasons for secrecy.
EMACS is a powerful tool—but we do not keep it secret because the mafia might use it to their own advantage. It is better all round that everyone has access, rather than just an elite. Both society and EMACS itself are better because of that strategy.
The idea that you can get security through obscurity is a common one—but it does not have a particularly-good history or reputation in IT.
The NSA is one of the more well-known examples of it being tried with some success. There we have a large organisation (many eyeballs inside mean diminishing returns from extra eyeballs) - and one with government backing. Despite this, the NSA often faces allegations of secretive, unethical behaviour.
You completely ignored my argument.
From my perspective, it seems inaccurate to claim that I ignored your argument—since I deat with it pretty explicitly in my paragraph about EMACS.
I certainly put a lot more effort into addressing your points than you just put into addressing mine.
I said that public access to an AI under development would be bad, because if it wasn’t safe to run—that is, if running it might cause it too foom and destroy the world—then no one would be able to make that judgment and keep others from running it. You responded with an analogy to EMACS, which no one believes or has ever believed to be dangerous, and which has no potential to do disastrous things that their operators did not intend. So that analogy is really a non sequitur.
“Dangerous” in this context does not mean “powerful”, it means “volatile”, as in “reacts explosively with Pentiums”.
Both types of software are powerful tools. Powerful tools are dangerous in the wrong hands, because they amplify the power of their users. That is the gist of the analogy.
I expect EMACS has been used for all kinds of evil purposes, from writing viruses, trojans, and worms to tax evasion and fraud.
I note that Anders Sandberg recently included:
“Otherwise the terrorists will win!”
...in his list of of signs that you might be looking at a weak moral argument.
That seems rather dubious as a general motto, but in this case, I am inclined to agree. In the case of intelligent machines, the positives of openness substantially outweigh their negatives, IMO.
Budding machine intelligence builders badly need to signal that they are not going to screw everyone over. How else are other people to know that they are not planning to screw everyone over?
Such signals should be expensive and difficult to fake. In this case, about the only credible signal is maximum transpareny. I am not going to screw you over, and look, here is the proof: what’s mine is yours.
If you don’t understand something I’ve written, please ask for clarification. Don’t guess what I said and respond to that instead; that’s obnoxious. Your comparison of my argument to
Leads me to believe that you didn’t understand what I said at all. How is destroying the world by accident like terrorism?
Er, characterising someone who disagrees with you on a technical point as “obnoxious” is not terribly great manners in itself! I never compared destroying the world by accident with terrorism—you appear to be projecting. However, I am not especially interested in the conversation being dragged into the gutter in this way.
If you did have a good argument favouring closed source software and reduced transparency I think there has been a reasonable chance to present it. However, if you can’t even be civil, perhaps you should consider waiting until you can.
I gave an argument that open-sourcing AI would increase the risk of the world being destroyed by accident. You said
I presented the mismatch between this statement and my argument as evidence that you had misunderstood what I was saying. In your reply,
You are misunderstanding me again. I think I’ve already said all that needs to be said, but I can’t clear up confusion if you keep attacking straw men rather than asking questions.
You are confusing socially important with societally important. Microsoft, for example, seeks to have its source code transparent to inspection, because Microsoft, as a corporate culture, produces software socially—that is, utilizing an evil conspiracy involving many communicating agents.
I deny confusing anything. I understand that transparency can be a matter of degree and perspective. What I am pointing out is lip-service to transparency. Full transparency would be different.
Microsoft’s software is not very transparent—and partly as a result it is some of the most badly-designed, insecure and virus-ridden software the planet has ever seen. We can see the mistake, can see its consequences—and know how to avoid it—but we have to, like actually do that—and that involves some alerting of others to the problems often associated with closed-source proposals.