And that, to this end, we would like to know what is or isn’t a person—or at least have a predicate that returns 1 for all people and could return 0 or 1 for anything that isn’t a person, so that, if the predicate returns 0, we know we have a definite nonperson on our hands.
So: define such a function—as is done by the world’s legal systems. Of course, in a post-human era, it probably won’t “carve nature at the joints” much better than the “how many hairs make a beard” function manages to.
And I will not, if at all possible, give any other human being the least cause to think that someone else might spark a better Singularity. I can make no promises upon the future, but I will at least not close off desirable avenues through my own actions.
A possible problem here is that your high entry requirement specifications may well, with a substantial probability, allow others with lower standards to create a superintelligence before you do.
So: since you seem to think that would be pretty bad, and since you say you are a consequentialist—and believe in the greater good—you should probably act to stop them—e.g. by stepping up your own efforts to get there first by bringing the target nearer to you.
CEV runs once on a collection of existing humans then overwrites itself [...]
Ah. My objection doesn’t apply, then. It’s better than I had thought.
You’ve already said the friendly AI problem is terribly hard, and there’s a large chance we’ll fail to solve it in time. Why then do you keep adding these extra minor conditions on what it means to be “friendly”, making your design task all that harder?
While we are on the topic, the problem I see in this area is not that friendliness has too many extra conditions appended on it. It’s that the concept is so vague and amorphous that only Yudkowsky seems to know what it means.
When I last asked what it meant, I was pointed to the CEV document—which seems like a rambling word salad to me—I have great difficulty in taking it seriously. The most glaring problem with the document—from my point of view—is that it assumes that everyone knows what a “human” is. That might be obvious today, but in the future, things could well get a lot more blurry—especially if it is decreed that only “humans” have a say in the proposed future. Do uploads count? What about cyborgs? - and so on.
If it is proposed that everything in the future revolves around “humans” (until the “humans” say otherwise) then—apart from the whole issue of whether that is a good idea in the first place—we (or at least the proposed AI) would first need to know what a “human” is.
FWIW, Phil’s point there seems to be perfectly reasonable—and not in need of correction: if a moral system tells you to do what you were going to do anyway, it isn’t going to be doing much work.
Moral systems usually tell you not to do things that you would otherwise be inclined to do—on the grounds that they are bad. Common examples include taking things you want—and having sex.
If you would balk at killing a million people with a nuclear weapon, you should balk at this.
The main problem with death is that valuable things get lost.
Once people are digital, this problem tends to go away—since you can relatively easily scan their brains—and preserve anything of genuine value.
In summary, I don’t see why this issue would be much of a problem.
Re: Barry Schwartz’s The Paradox of Choice [...] talks about how offering people more choices can make them less happy. A simple intuition says this shouldn’t ought to happen to rational agents: If your current choice is X, and you’re offered an alternative Y that’s worse than X, and you know it, you can always just go on doing X. So a rational agent shouldn’t do worse by having more options. The more available actions you have, the more powerful you become—that’s how it should ought to work.
This makes no sense to me. A blind choice between lady and tiger is preferable to a blind choice between a lady and two tigers. Problems arise when you don’t know that the other choices are worse. So having more choices can be really bad—in a way that has nothing to do with the extra cycles burned in evaluating them.
Re: Vassar advocates that rationalists should learn to lie, I advocate that rationalists should practice telling the truth more effectively, and we’re still having that argument.
Uh huh. What are the goals of these hypothetical rational agents?
Still, expect to see some outraged comments on this very blog post, from commenters who think that it’s selfish and immoral, and above all a failure of imagination, to talk about human-level minds still running around the day after the Singularity.
For me, the very concept of “the day after the Singularity” is so far out—and off the rails—that I would hardly describe it as a failure of the imagination.
The idea seems more likely to be the result of an overactive, over-stimulated imagination—or perhaps the wild imaginings of some science fiction author.
Hmm. I wonder if this indicates that we may expect to see an exposition on the topic of Eliezer’s preferred attempt at a solution to the wirehead problem. That would be fun.
Sure, but that is not really a “Turing machine of that size” - it’s a substantially bigger one—since it has to encode the input, and another program expressing decoding it and writing it out.
The Busy Beaver problem is uncomputable—there is no fixed Turing machine that computes it for all N—because if you knew all the Busy Beaver numbers, you would have an infallible way of telling whether a Turing machine halts; just run it up for as long as the longest-running Turing machine of that size.
Hmm. The Busy Beaver functions only deal with the case of a TM with a blank tape. Have an arbitrary starting configuration on the tape, and TMs can run for much longer.
The halting problem normally deals with arbitrary input configurations: “Alan Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist”.
If living systems can unite, they can also be divided. I don’t see what the problem with that idea could be.
I mean—most agents with utility functions shouldn’t have such a hard time describing their perfect universe.
As I understand it, most organisms act as though they want to accelerate universal heat death.
That’s been an important theory at least since Lotka’s Contribution to the Energetics of Evolution, 1922 - and has been explicated in detail in modern times by Roderick Dewar with the MEP.
To change that would require self-directed evolution, a lot of discipline—and probably not encountering any aliens—but what would the motivation be?
I think many value the quality of life of their friends and loved ones more than they value hypothetical far-future abstractions. This has to do with evolution’s impact on psychology—and doesn’t have much to do with how big the universe is.
Your point that if I don’t give the answer the audience literally has no choice but to make their own guesses, which can only be based on even less information
“The answer”: heh. No doubt the audience members are collectively pleased to hear about how poorly informed they supposedly are.
Persistent ones that accumulate. Mainly viruses. In a lifetime people gradually rack up a load of Persistent Viral Infections—and many of them strike when a person is ill and their immune system is weakened.
An well-trodden point from my perspective—but de Grey’s list of damage makes no mention of pathogens. Yet a number of types of pathogens are long-lived entities that accumulate and eventually do you in. So: a pathogen plan is needed—and dealing with pathogens is not an easy problem.
However, I see no reason to suppose that an AGI would apply its intelligence to every aspect of its design except its goal structure.
I don’t think that describes the probable outcome anybody’s superintelligence construction plan.
Our minds are made by (essentially) stone-age genes, but they import up-to-date memes—and are a product of influences from both sources.
So: our minds are actually pretty radically different from stone-age minds—because they have downloaded and are running a very different set of brain-software routines. This influence of memes explains why modern society is so different from the societies present in the stone age.