And what do you do when an insider says, “If you don’t change the CEV programming to include X, then I am going public!” How do you handle that? How many people is it that you expect to remain quiet for two years?
For some reasonably-successful corporate secrecy, perhaps look to Apple. They use NDAs, need-to-know principles, and other techniques—and they are usually fairly successful at keeping their secrets. Some of the apparent leaks are probably PR exercises.
Or, show me Google’s source code—or the source code of any reasonable-size hedge fund. Secrecy seems fairly managable, in practice.
Google leaks like a sieve, actually, but that should be because of the sheer number of employees.
It’s true that there have been no source-code leaks (to my knowledge), but that could just as likely be because of the immense expected consequences of getting caught at leaking any, and you would probably get caught.
I think that a programmer who cared enough about CEV to be a secret-keeper would also care enough about getting CEV right to kill in order to prevent it from being done wrong. The public need not be involved at all.
Is a good teammate one who has the social skills to make everybody happy when they are doing something they don’t want to, or someone who thinks that the team’s task is so important that they will do anything to get it done?
Are major breakthroughs which require a lot of work more likely to be done by people who don’t care, or by people that do?
Do you really think the public would be interested in the opinions of a programmer who claims that some guys in a basement are building a superintelligent machine?
Most would regard him as a crack-pot, just like most people think that Eliezer and his ideas are crazy.
Perhaps in 20-30 years this will change, and the problem of FAI will be recognized as tremendously important by political leaders and the general public, but I’m skeptical.
ETA:
I meant of course, that most people, if they knew Eliezer would think he is crazy.
I suppose the only people who will get to the point of being “insiders” will be a subset of the people who are trustworthy and sane and smart and non-evil enough not to try something like that.
Are we using the same definition of “insider”? I was talking about people who are inside the FAI project and have privileged knowledge of its status and possibly access to the detailed theory and its source code, etc. I don’t get the relevance of your links.
My links mentioned three persons. Obviously at this point, Robin and Roko are not going to become insiders in an FAI construction project. If you could assure me that the third linked person will not be an insider either, it would relieve a lot of my worries.
The relevance of my links was to point out that when intelligent people with strong opinions get involved together in important projects with the future of mankind at stake, keeping everyone happy and focused on the goal may be difficult. Especially since the goal has not yet been spelled out and no one seems to want to work on clarifying the goal since it is apparently so damned disruptive to even talk about it.
Documents dated 2004 and labeled “already obsolete when written” for Gods sake!
Robin, we’re still talking about a local foom. Keeping security for two years may be difficult but is hardly unheard-of.
And what do you do when an insider says, “If you don’t change the CEV programming to include X, then I am going public!” How do you handle that? How many people is it that you expect to remain quiet for two years?
For some reasonably-successful corporate secrecy, perhaps look to Apple. They use NDAs, need-to-know principles, and other techniques—and they are usually fairly successful at keeping their secrets. Some of the apparent leaks are probably PR exercises.
Or, show me Google’s source code—or the source code of any reasonable-size hedge fund. Secrecy seems fairly managable, in practice.
Google leaks like a sieve, actually, but that should be because of the sheer number of employees.
It’s true that there have been no source-code leaks (to my knowledge), but that could just as likely be because of the immense expected consequences of getting caught at leaking any, and you would probably get caught.
I think that a programmer who cared enough about CEV to be a secret-keeper would also care enough about getting CEV right to kill in order to prevent it from being done wrong. The public need not be involved at all.
Agreed, in principle, but I’m not sure that such people would make very good teammates.
(Implying that AGI is more likely to be developed by people who don’t care that much.)
Is a good teammate one who has the social skills to make everybody happy when they are doing something they don’t want to, or someone who thinks that the team’s task is so important that they will do anything to get it done?
Are major breakthroughs which require a lot of work more likely to be done by people who don’t care, or by people that do?
That’s not my point, which is simply this:
A good teammate is probably not one who’s willing to kill you if you make the wrong move, and who—being human—may misinterpret your actions.
If there is no move you could make which would result in your teammate trying to kill you, then you have a different problem.
Do you really think the public would be interested in the opinions of a programmer who claims that some guys in a basement are building a superintelligent machine? Most would regard him as a crack-pot, just like most people think that Eliezer and his ideas are crazy. Perhaps in 20-30 years this will change, and the problem of FAI will be recognized as tremendously important by political leaders and the general public, but I’m skeptical.
ETA: I meant of course, that most people, if they knew Eliezer would think he is crazy.
He hasn’t reached that level yet. Most people just don’t know or care wtf Eliezer is! ;)
I suppose the only people who will get to the point of being “insiders” will be a subset of the people who are trustworthy and sane and smart and non-evil enough not to try something like that.
Ah, so no insider has ever walked off in a huff? No insider has ever said he would refuse to participate further if something he felt strongly about wasn’t done? Look at A3 here.
The SIAI must use some pretty remarkable personality tests to choose their personnel.
Are we using the same definition of “insider”? I was talking about people who are inside the FAI project and have privileged knowledge of its status and possibly access to the detailed theory and its source code, etc. I don’t get the relevance of your links.
My links mentioned three persons. Obviously at this point, Robin and Roko are not going to become insiders in an FAI construction project. If you could assure me that the third linked person will not be an insider either, it would relieve a lot of my worries.
The relevance of my links was to point out that when intelligent people with strong opinions get involved together in important projects with the future of mankind at stake, keeping everyone happy and focused on the goal may be difficult. Especially since the goal has not yet been spelled out and no one seems to want to work on clarifying the goal since it is apparently so damned disruptive to even talk about it.
Documents dated 2004 and labeled “already obsolete when written” for Gods sake!