Goetz,
For a superhuman AI to stop you and your friends from launching a competing AI, it suffices for it to take away your access to unsupervised computing resources. It does not have to kill you.
Richard_Hollerith2
Karma: 177
Andrix, if it is just a recoiling from that, then how do you explain Stalin, Mao, etc?
Yes, Nancy, as soon as an AI endorsed by Eliezer or me transcends to superintelligence, it will probably make a point of preventing any other AI from transcending, and there is indeed a chance that that will entail killing a few (probably very irresponsible) humans. It is very unlikely to entail the killing of millions, and I can go into that more if you want.
The points are that (1) self-preservation and staying in power is easy if you are the only superintelligence in the solar system and that (2) unlike a governing coalition of humans who believe the end justifies the means, a well-designed well-implemented superintelligence will not kill or oppress millions for a nominally prosocial end which is in reality a flimsy excuse for staying in power.
But, Nancy, the self-preservation can be an instrumental goal. That is, we can make it so that the only reason the AI wants to keep on living is that if it does not then it cannot help the humans.
It is refreshing to read something by Eliezer on morality I completely agree with.
And nice succinct summary by Zubon.
I imagine it might be difficult to find a room for people to mill around in.
Well, the restaurant in which we met was crowded, so I suggest a less crowded restaurant.
I’m not going to the Summit or the meetup, but will meet one-on-one with anyone for discussion of rationality or AI ethics especially if the meeting can be in San Francisco or anywhere within walking distance of a Bay Area Rapid Transit station.
Oh by the way, one thing I did not like about the last Bay Area meetup was that three extroverted people dominated the conversation at the big, 12-person round table at which I sat. That meetup was at a crowded restaurant. If there had been more room to move around, it would have been easier for me to contrive to hear from some of the less-extroverted attendees.
They’re still pretty awful, IMHO.
He says he isn’t ready to write code. If you don’t try to code up a general artificial intelligence you don’t succeed, but you don’t fail either.
Would people stop saying that! It is highly irresponsible in the context of general AI! (Well, at least the self-improving form of general AI, a.k.a., seed AI. I’m not qualified to say whether a general AI not deliberately designed for self-improvement might self-improve anyways.)
Noodling around with general-AI designs is the most probable of the prospective causes of the extinction of Earth-originating intelligence and life. Global warming is positively benign in comparison.
Eliezer of course will not be influenced by taunts of, “Show us the code,” but less responsible people might be.
I too thought Nesov’s comment was written by Eliezer.
It is true that my proposal does not benefit any person, human or otherwise, except as a means to further ends.
A human or a sentient has no intrinsic value in my way of thinking about morality—though of course humans have great instrumental value as long as they remain the only intelligent agents in the known universe.
Now note that one galaxy converted into a superintelligent cloud of matter and energy suffices to keep each and every human alive for billions of years, end disease and suffering, etc, with plenty of matter and energy left over for frivolous toys like a planet transformed into a child’s toy.
My proposal is mainly an answer to the question of what end to put all those other galaxies that are not being used to provide a nice place for the humans and their descendants to live.
turning the galaxy into computers that could run a question if there were any question to run.
That characterization of my system is unfair. The goal is more like turning the easy-to-reach matter and energy into computers and von-Neumann probes that will turn less-easy-to-reach matter and energy into computers and von-Neumann probes in an unending cycle, except that eventually the computers and probes will probably have to adopt other means of continuing, e.g., when it becomes clear that there is no way for computers and probes to continue to exist in this space-time continuum because the continuum itself will end in, e.g., a Big Rip.
If not, why aren’t you in the camp of those who wish to improve human intelligence?
I’ll take this one because I’m almost certain Eliezer would answer the same way.
Working on AI is a more effective way of increasing the intelligence of the space and matter around us than increasing human intelligence is. The probability of making substantial progress is higher.
In other words, there is no way to program a search for objective morality or for any other search target without the programmer specifying or defining what constitutes a successful conclusion of the search.
If you understand this, then I am wholly at a loss to understand why you think an AI should have “universal” goals or a goal system zero or whatever it is you’re calling it.
The flip answer is that the AI must have some goal system (and the designer of the AI must choose it). The community contains vocal egoists, like Peter Voss, Hopefully Anonymous, maybe Denis Bider. They want the AI to help them achieve their egoistic ends. Are you less at a loss to understand them than me?
Shane: “Roko, why not”
Let make Shane’s reply more formal, so that Roko has something concrete to attack.
I did not have time to learn how to indent things on this blog, so I use braces to indicate indentation and semicolon to indicate the start of a new line.
Let state be a vector of length n such that for every integer time, (state[time] == A) or (state[time] == B).
U(state) == (sum as i goes from 0 to n in steps of 2) {2 if (state[i] == B) and (state[i+1] == A); 1 if (state[i] == B) or (state[i+1] == A); 0 otherwise}
Roko, will you please exhibit a mind that you believe is not a utility maximizer? I am having trouble imagining one. For example, I consider a mind that maximizes the probability of some condition X coming to pass. Well, that is a utility maximizer in which possible futures satifying condition X have utility 1 whereas the other possible futures have utility 0. I consider a mind produced by natural selection, e.g., a mammalian mind or a human mind. Well, I see no reason to believe that that mind is not a utility maximizer with a complicated utility function that no one can describe completely, which to me is a different statement than saying the function does not exist.
If Eliezer had not abandoned the metaethics he adopted in 1997 or so by the course described in this blog entry, he might have abandoned it later in the design of the seed AI when it became clear to him that the designer of the seed must choose the criterion the AI will use to recognize objective morality when it finds it. In other words, there is no way to program a search for objective morality or for any other search target without the programmer specifying or defining what constitutes a successful conclusion of the search.
The reason a human seems to be able to search for things without being able to define clearly at the start of the search what he or she is searching for is that humans have preferences and criteria that no one can articulate fully. Well, the reader might be thinking, why not design the AI so that it, too, has criteria that no one can articulate? My answer has 2 parts: one part explains that CEV is not such an unarticulatable design; the other part asserts that any truly unarticulatable design would be irresponsible.
Although it is true that no one currently in existence can articulate the volition of the humans, it is possible for some of us to specify or define with enough precision and formality what the volition of the humans is and how the AI should extrapolate it. In turn, a superintelligent AI in possession of such a definition can articulate the volition of the humans.
The point is that although it is a technically and scientifically challenging problem, it is not outside the realm of current human capability to define what is meant by the phrase “coherent extrapolated volition” in sufficient precision, reliability and formality to bet the outcome of the intelligence explosion on it.
Like I said, humans have systems of value and systems of goals that no one can articulate. The only thing that keeps it from being completely unethical to rely on humans for any important purpose is that we have no alternative means of achieving the important purpose. In contrast, it is possible to design a seed AI whose goal system is “articulatable”, which means that some human or some team or community of humans can understand it utterly, the way that some humans can currently understand relativity theory utterly. An agent with an articulatable goal system is vastly preferrable to the alternative because it is vastly desirable for the designer of the agent to do his or her best in choosing the optimization target of the agent, and choosing an unarticulatable goal system is simply throwing away that ability to choose—leaving the choice up to “chance”.
To switch briefly to a personal note, when I found Eliezer’s writings in 2001 his home page still linked to his explanation of the metaethics he adopted in 1997 or so, which happened to coincide with my metaethics at the time (which coincidence made me say to myself, “What a wonderful young man!” ). I can for example recall using the argument Eliezer give below in a discussion of ethics with my roommate in 1994:
In the event that life is meaningless, nothing is the “right” thing to do; therefore it wouldn’t be particularly right to respect people’s preferences in this event.
Anyway, I have presented the argument against the metaethics to which Eliezer and I used to subscribe that I find the most persuasive.
I retract my endorsement of Simon’s last comment. Simon writes that S == (F or not W). False: S ==> (F or not W), but the converse does not hold (because even if F or not W, we could all be killed by, e.g., a giant comet). Moreover, Simon writes that F ==> S. False (for the same reason). Finally, Simon writes, “Note that none of these probabilities are conditional on survival,” and concludes from that that there are no selection effects. But the fact that a true equation does not contain any explicit reference to S does not mean that any of the propositions mentioned in the equation are independent or conditionally independent of S. In other words, we have established neither P(W|F) == P(W|F,S) nor P(F|W) == P(F|W,S) nor P(W) == P(W|S) nor P(F) == P(F|S), which makes me wonder how we can conclude the absence of an observational selection effect.
Simon’s last comment is well said, and I agree with everything in it. Good job, Simon and Benja.
Although the trickiest question was answered by Simon and Benja, Eliezer asked a couple of other questions, and Yvain gave a correct and very clear answer to the final question.
Or so it seems to me.
I agree with Caledonian this time: pdf23′s comment shows a lack of basic knowledge of money.
Ah, now I see. Thanks for speaking up, steven.
Post-nanotech, the nanotech practices the virtues of rationality or it fails to achieve its goals.