Phil_Goetz6

Karma: 56

Phil_Goetz6 13 Feb 2009 23:44 UTC
0 points
on: The Evolutionary-Cognitive Boundary
“I think it really is important to use different words that draw a hard boundary between the evolutionary computation and the cognitive computation.”

Does this boundary even exist? It’s a distinction we can make, for purposes of discussion; but not a hard boundary we can draw. You can find examples that fall clearly into one category (reflex) or another (addition), but you can also find examples that don’t. This is just the sort of thing I was talking about in my post on false false dichotomies. It’s a dichotomy that we can sometimes use for discussion, but not a true in-the-world binary distinction.

Eliezer responds yes: “Anna, you’re talking about a messiness of the human system, not a difficulty in drawing hard distinctions between human-style messiness and evolutionary-style messiness.”

I can’t figure out what that’s supposed to mean. I think it means Eliezer didn’t understand what she said. The “messiness” is that you can’t draw that hard distinction.

The entire discussion is cast in terms that imply Eliezer thinks evolutionary psychology deals with issues of conscious vs. subconscious motivations. AFAIK it sidesteps the issue whenever possible. Psychologists don’t want to ask whether behavior comes from conscious or subconscious motivations. They want to observe behavior, record it, and explain it. Not trying to slice it up into conscious vs. subconscious pieces is the good part of behaviorism.

Phil_Goetz6 18 Dec 2008 18:01 UTC
1 point
on: Prolegomena to a Theory of Fun
Michael Vassar: “Phil: Eliezer has repeatedly said that ems (formerly uploads) are people. Eliezer, can you please clarify this point in a simple direct comment aimed at Phil?”

Huh? No need. Why would you think I’m unaware of that?

I notice that several people replied to my question, Why not colonize America?; yet no one addressed it. I think they fail to see the strength of the analogy. Humans use many more resources than ems or AIs. If you take the resources from the humans and give them to the AI, you will at some point be able to support 100 times as many “equivalent”, equally happy people. Make an argument for not doing that. And don’t, as komponisto did, just say that it’s the right thing to do.

Everybody says that not taking the land from the Native Americans would have been the right thing to do; but nobody wants to give it back.

An argument against universe-tiling would also be welcome.

Phil_Goetz6 18 Dec 2008 0:53 UTC
1 point
on: Prolegomena to a Theory of Fun
You’re solving the wrong problem. Did you really just call a body of experimental knowledge a political inconvenience?
Oh, snap.

Still, expect to see some outraged comments on this very blog post, from commenters who think that it’s selfish and immoral … to talk about human-level minds still running around the day after the Singularity.
We’re offended by the inequity—why does that big hunk of meat get to use 2,000W plus 2000 square feet that it doesn’t know what to do with, while the poor, hardworking, higher-social-value em gets 5W and one square inch? And by the failure to maximize social utility.

Fun is a cognitive phenomenon. Whatever your theory of fun is, I predict that more fun will be better than less fun, and the moral thing to do seems to be to pack in as much fun as you can before the heat death of the universe. Following that line of thought could lead to universe-tiling.

Suppose you develop a theory of fun/good/morality. What are arguments for not tiling the universe in a way that maximizes it? Are there any such arguments that don’t rely on either diversity as an inherent good, or on the possibility that your theory is wrong?

Your post seems to say that fun and morality are the same. But we use the term “moral” only in cases when the moral thing to do isn’t fun. I think morality = fun only if it’s a collective fun. If that collective fun is also summed over hypothetical agents you could create, then we come back to moral outrage at humans.

The problem brings to mind the colonization of America. Would it have been the moral thing to do to turn around and leave the Indians alone, instead of taking their land and using it to build an advancing civilization that can support a population of about 100 times as many people, who think they are living more pleasurable and interesting lives, and hardly ever cut out their neighbors’ hearts on the tops of temples to the sun god? Intellectuals today unanimously say “yes”. But I don’t think they’ve allowed themselves to actually consider the question.

What is the moral argument for not colonizing America?

Phil_Goetz6 18 Dec 2008 0:15 UTC
16 points
0
on: Prolegomena to a Theory of Fun
I can only analogize the experience to a theist who’s suddenly told that they can know the mind of God, and it turns out to be only twenty lines of Python.
And the twenty lines are from the “spam” sketch. :)

Phil_Goetz6 17 Dec 2008 19:23 UTC
0 points
on: Visualizing Eutopia
Ben: “There’s absolutely no guarantee that humanity won’t go the way of the neanderthal in the grand scheme of things.”

Are you actually hoping that won’t happen? That we’ll still be human a million years from now?

Phil_Goetz6 16 Dec 2008 20:24 UTC
−2 points
on: Visualizing Eutopia
“Questions like “what do I want out of life?” or “what do I want the universe to look like?” are super important questions to ask, regardless of whether you have a magic genie. I personally have had the unfortunate experience of answering some parts of those question wrong and then using those wrong answers to run my life for a while.”

I think that asking what you want the universe to look like in the long run, has little or no bearing on how to live your life in the present. (Except insofar as you direct your life to planning the universe’s future.) The problems confronted are different.

Phil_Goetz6 16 Dec 2008 4:51 UTC
4 points
on: Not Taking Over the World
To ask what God should do to make people happy, I would begin by asking whether happiness or pleasure are coherent concepts in a future in which every person had a Godbot to fulfill their wishes. (This question has been addressed many times in science fiction, but with little imagination.) If the answer is no, then perhaps God should be “unkind”, and prevent desire-saturation dynamics from arising. (But see the last paragraph of this comment for another possibility.)

What things give us the most pleasure today? I would say, sex, creative activity, social activity, learning, and games.

Elaborating on sexual pleasure probably leads to wireheading. I don’t like wireheading, because it fails my most basic ethical principle, which is that resources should be used to increase local complexity. Valuing wireheading qualia also leads to the conclusion that one should tile the universe with wireheaders, which I find revolting, although I don’t know how to justify that feeling.

Social activity is difficult to analyze, especially if interpersonal boundaries, and the level of the cognitive hierarchy to relate to as a “person”, are unclear. I would begin by asking whether we would get any social pleasure from interacting with someone whose thoughts and decision processes were completely known to us.

Creative activity and learning may or may not have infinite possibilities. Can we continue constructing more and more complex concepts, to infinity? If so, then knowledge is probably also infinite, for as soon as we have constructed a new concept, we have something new to learn about. If not, then knowledge—not specific knowledge of what you had for lunch today, but general knowledge—may be limited. Creative activity may have infinite possibilities, even if knowledge is finite.

(The answer to whether intelligence has infinite potential has many other consequences; notably, Bayesian reasoners are likely only in a universe in which there are finite useful concepts, because otherwise it will be preferable to be a non-Bayesian reasoning over more complex concepts using faster algorithms.)

Games largely rely on uncertainty, improving mastery, and competition. Most of what we get out of “life”, besides relationships and direct hormonal pleasures like sex, food, and fighting, is a lot like what we get from playing a game. One fear is that life will become like playing chess when you already know the entire game tree.

If we are so unfortunate as to live in a universe in which knowledge is finite, then conflict may serve as a substitute for ignorance in providing us a challenge. A future of endless war may be preferable to a future in which someone has won. It may even be preferable to a future of endless peace. If you study the middle ages of Europe, you will probably at some point ask, “Why did these idiots spend so much time fighting, when they could have all become wealthier if they simply stopped fighting long enough for their economies to grow?” Well, those people didn’t know that economies could grow. They didn’t believe that there was any further progress to be made in any domain—art, science, government—until Jesus returned. They didn’t have any personal challenges; the nobility often weren’t even allowed to do work. If you read what the nobles wrote, some of them said clearly that they fought because they loved fighting. It was the greatest thrill they ever had. I don’t like this option for our future, but I can’t rule out the possibility that war might once again be preferable to peace, if there actually is no more progress to be made and nothing to be done.

The answers to these questions also have a bearing on whether it is possible for God, in the long run, to be selfish. It seems that God would be the first person to have his desires saturated, and enter into this difficult position where it is hard to imagine how to want anything. I can imagine a universe, rather like the Buddhist universe, in which various gods, like bubbles, successively float to the top, and then burst into nothingness, from not caring anymore. I can also imagine an equilibrium, in which there are many gods, because the greater power than one acquires, the less interest one has in preserving that power.

Phil_Goetz6 12 Dec 2008 1:25 UTC
0 points
on: What I Think, If Not Why
It would have been better of me to reference Eliezer’s Al Qaeda argument, and explain why I find it unconvincing.

Vladimir:
Phil, in suggesting to replace an unFriendly AI that converges on a bad utility by a collection of AIs that never converge, you are effectively trying to improve the situation by injecting randomness in the system.
You believe evolution works, right?

You can replace randomness only once you understand the search space. Eliezer wants to replace the evolution of values, without understanding what it is that that evolution is optimizing. He wants to replace evolution that works, with a theory that has so many weak links in its long chain of logic that there is very little chance it will do what he wants it to, even supposing that what he wants it to do is the right thing to do.

Vladimir:
Your perception of lawful extrapolation of values as “stasis” seems to stem from intuitions about free will.
That’s a funny thing to say in response to what I said, including: ‘One question is where “extrapolation” fits on a scale between “value stasis” and “what a free wild-type AI would think of on its own.”’ It’s not that I think “extrapolation” is supposed to be stasis; I think it may be incoherent to talk about an “extrapolation” that is less free than “wild-type AI”, and yet doesn’t keep values out of some really good areas in value-space. Any way you look at it, it’s primates telling superintelligences what’s good.

As I just said, clearly “extrapolation” is meant to impose restrictions on the development of values. Otherwise it would be pointless.

Vladimir:
it could act as a special “luck” that in the end results in the best possible outcome given the allowed level of interference.
Please remember that I am not assuming that FAI-CEV is an oracle that magically works perfectly to produce the best possible outcome. Yes, an AI could subtly change things so that we’re not aware that it is RESTRICTING how our values develop. That doesn’t make it good for the rest of all time to be controlled by the utility functions of primates (even at a meta level).

Here’s a question whose answer could diminish my worries: Can CEV lead to the decision to abandon CEV? If smarter-than-humans “would decide” (modulo the gigantic assumption CEV makes that it makes sense to talk about what “smarter than humans would decide”, as if greater intelligence made agreement more rather than less likely—and, no, they will not be perfect Bayesians) that CEV is wrong, does that mean an AI guided by CEV would then stop following CEV?

If this is so, isn’t it almost probability 1 that CEV will be abandoned at some point?

Phil_Goetz6 11 Dec 2008 23:48 UTC
1 point
on: What I Think, If Not Why
Eliezer: “Tim probably read my analysis using the self-optimizing compiler as an example, then forgot that I had analyzed it and thought that he was inventing a crushing objection on his own. This pattern would explain a lot of Phil Goetz too.”

No; the dynamic you’re thinking of is that I raise objections to things that you have already analyzed, because I think your analyis was unconvincing. Eg., the recent Attila the Hun / Al Qaeda example. The fact that you have written about something doesn’t mean you’ve dealt with it satsifactorily.

Phil_Goetz6 11 Dec 2008 23:42 UTC
0 points
on: What I Think, If Not Why
Eliezer: “and then gets smart enough to do guaranteed self-improvement, at which point its values freeze (forever).”

Why do the values freeze? Because there is no more competition? And if that’s the problem, why not try to plan a transition from pre-AI to an ecology of competing AIs that will not converge to a singleton? Or spell out the problem clearly enough that we can figure whether one can achieve a singleton that doesn’t have that property?

(Not that Eliezer hasn’t heard me say this before. I made a bit of a speech about AI ecology at the end of the first AGI conference a few years ago.)

Robin: “In a foom that took two years, if the AI was visible after one year, that might give the world a year to destroy it.”

Yes. The timespan of the foom is important largely because it changes what the AI is likely to do, because it changes the level of danger that the AI is in and the urgency of its actions.

Eliezer: “When I try myself to visualize what a beneficial superintelligence ought to do, it consists of setting up a world that works by better rules, and then fading into the background.”

There are many sociological parallels between Eliezer’s “movement”, and early 20th-century communism.

Eliezer: “I truly do not understand how anyone can pay any attention to anything I have said on this subject, and come away with the impression that I think programmers are supposed to directly impress their non-meta personal philosophies onto a Friendly AI.”

I wonder if you’re thinking that I meant that. You can see that I didn’t in my first comment on Visions of Heritage. But I do think you’re going one level too few meta. And I think that CEV would make it very hard to escape the non-meta philosophies of the programmers. It would be worse at escaping them than the current, natural system of cultural evolution is.

Numerous people have responded to some of my posts by saying that CEV doesn’t restrict the development of values (or equivalently, that CEV doesn’t make AIs less free). Obviously it does. That’s the point of CEV. If you’re not trying to restrict how values develop, you might as well go home and watch TV and let the future spin out of control. One question is where “extrapolation” fits on a scale between “value stasis” and “what a free wild-type AI would think of on its own.” Is it “meta-level value stasis”?

I think that evolution and competition have been pretty good at causing value development. (That’s me going one more level meta.) Having competition between different subpopulations with different values is a key part of this. Taking that away would be disastrous.

Not to mention the fact that value systems are local optima. If you’re doing search, it might make sense to average together some current good solutions and test the results out, in competition with the original solutions. It is definitely a bad idea to average together your current good solutions and replace them with the average.

Phil_Goetz6 9 Dec 2008 15:54 UTC
0 points
on: Is That Your True Rejection?
I don’t think it did help, though. I think I failed to comprehend it. I didn’t file it away and think about it; I completely missed the point. Later, my subconscious somehow changed gears so that I was able to go back and comprehend it. But communication failed.

Buddhists say that great truths can’t be communicated; they have to be experienced, only after which you can understand the communication. This was something like that. Discouraging.

Phil_Goetz6 9 Dec 2008 15:48 UTC
7 points
on: Disjunctions, Antipredictions, Etc.
But if you’re going to bother visualizing the future, it does seem to help to visualize more than one way it could go, instead of concentrating all your strength into one prediction. So I try not to ask myself “What will happen?” but rather “Is this possibility allowed to happen, or is it prohibited?”

I thought that you were changing your position; instead, you have used this opening to lead back into concentrating all your strength into one prediction.

I think this characterizes a good portion of the recent debate: Some people (me, for instance) keep saying “Outcomes other than FOOM are possible”, and you keep saying, “No, FOOM is possible.” Maybe you mean to address Robin specifically; and I don’t recall any acknowledgement from Robin that foom is >5% probability. But in the context of all the posts from other people, it looks as if you keep making arguments for “FOOM is possible” and implying that they prove “FOOM is inevitable”.

A second aspect is that some people (again, eg., me) keep saying, “The escalation leading up to the first genius-level AI might be on a human time-scale,” and you keep saying, “The escalation must eventually be much faster than human time-scale.” The context makes it look as if this is a disagreement, and as if you are presenting arguments that AIs will eventually self-improve themselves out of the human timescale and saying that they prove FOOM.

Phil_Goetz6 9 Dec 2008 15:23 UTC
2 points
on: Is That Your True Rejection?
You could say that embracing timeless decision theory is a global meta-commitment, that makes you act as if you made commitment in all the situations where you benefit from having made the commitment.
I think this is correct.

It’s perplexing: This seems like a logic problem, and I expect to make progress on logic problems using logic. I would expect reading an explanation to be more helpful than having my subconscious mull over a logic problem. But instead, the first time I read it, I couldn’t understand it properly because I was not framing the problem properly. Only after I suddenly understood the problem better, without consciously thinking about it, was I able to go back, re-read this, and understand it.

Phil_Goetz6 9 Dec 2008 15:08 UTC
1 point
on: Is That Your True Rejection?
I may be wrong about Newcomb’s paradox.

Phil_Goetz6 8 Dec 2008 17:45 UTC
−1 points
on: Artificial Mysterious Intelligence
Eliezer: I was making a parallel. I didn’t mean “how are these different”; I really meant, “This statement below about consciousness is wrong; yet it seems very similar to Eliezer’s post. What is different about Eliezer’s post that would make it not be wrong in the same way?”

That said, we don’t know what consciousness is, and we don’t know what intelligence is; and both occur in every instance of intelligence that we know of; and it would be surprising to find one without the other even in an AI; so I don’t think we can distinguish between them.

Phil_Goetz6 8 Dec 2008 16:18 UTC
0 points
on: Artificial Mysterious Intelligence
How is this different from saying,

“For a long time, many different parties and factions in AI, adherent to more than one ideology, have been trying to build AI without understanding consciousness. Unfortunate habits of thought will already begin to arise, as soon as you start thinking of ways to create Artificial Intelligence without having to penetrate the mystery of consciousness. Instead of all this mucking about with neurons and neuroanatomy and population encoding and spike trains, we should be facing up to the hard problem of understanding what consciousness is.”

Phil_Goetz6 5 Dec 2008 22:21 UTC
3 points
on: Sustained Strong Recursion
“The issue is that simulating a computers design require a lot of computational power. The advances made in going from 65nm to 45nm now moving to 32nm were enabled by computers that could better simulate the designs without todays computers it would be hard to design the fabrication systems or run the fabrication system for the future processors.”

I believe (strongly) that the bottleneck is figuring out how to make 45nm and 32nm circuits work reliably. If you learn how to do 32nm, you can probably get speedup just by re-using the same design you used at 45nm.

Phil_Goetz6 3 Dec 2008 23:09 UTC
2 points
on: Permitted Possibilities, & Locality
I designed, with a co-worker, a cognitive infrastructure for DARPA that is supposed to let AIs share code. I intended to have cognitive modules be web services (at present, they’re just software agents). Every representation used was to be evaluated using a subset of Prolog, so that expressions could be automatically converted between representations. (This was never implemented; nor was ontology mapping, which is really hard and would also be needed to translate content.) Unfortunately, my former employer didn’t let me publish anything on it. Also, it works only with symbolic AI.

It wouldn’t change the picture Eliezer is drawing much even if it worked perfectly, though.

on any sufficiently short timescale, progress should locally approximate an exponential because of competition between interest rates (even in the interior of a single mind).
Huh?

Phil_Goetz6 3 Dec 2008 17:45 UTC
3 points
on: Hard Takeoff
Eliezer: So really, the whole hard takeoff analysis of “flatline or FOOM” just ends up saying, “the AI will not hit the human timescale keyhole.” From our perspective, an AI will either be so slow as to be bottlenecked, or so fast as to be FOOM.
But the AI is tied up with the human timescale at the start. All of the work on improving the AI, possibly for many years, until it reaches very high intelligence, will be done by humans. And even after, it will still be tied up with the human economy for a time, relying on humans to build parts for it, etc. Remember that I’m only questioning the trajectory for the first year or decade.

(BTW, the term “trajectory” implies that only the state of the entity at the top of the heap matters. One of the human race’s backup plans should be to look for a niche in the rest of the heap. But I’ve already said my piece on that in earlier comments.)

Thomas: Even if it is wrong—I think it is correct—it is the most important thing to consider.
I think most of us agree it’s possible. I’m only arguing that other possibilities should also be considered. It would be unwise to adopt a strategy that has a 1% chance of making 90%-chance situation A survivable, if that strategy will make the otherwise-survivable 10%-chance situation B deadly.

Phil_Goetz6 2 Dec 2008 22:21 UTC
3 points
on: Hard Takeoff
“All these complications is why I don’t believe we can really do any sort of math that will predict quantitatively the trajectory of a hard takeoff. You can make up models, but real life is going to include all sorts of discrete jumps, bottlenecks, bonanzas, insights—and the “fold the curve in on itself” paradigm of recursion is going to amplify even small roughnesses in the trajectory.”

Wouldn’t that be a reason to say, “I don’t know what will happen”? And to disallow you from saying, “An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely”?

If you can’t make quantitative predictions, then you can’t say that the foom might take an hour or a day, but not six months.

A lower-bound (of the growth curve) analysis could be sufficient to argue the inevitability of foom.

I agree there’s a time coming when things will happen too fast for humans. But “hard takeoff”, to me, means foom without warning. If the foom doesn’t occur until the AI is smart enough to rewrite an AI textbook, that might give us years or decades of warning. If humans add and improve different cognitive skills to the AI one-by-one, that will start a more gently-sloping RSI.