Perplexed

Karma: 5,223

Perplexed Jun 12, 2011, 5:09 PM
1 point
in reply to: Peterdjones’s comment on: Welcome to Less Wrong!

… that has nothing to do with having a soul.

Well, it has nothing to do with what you think of as a ‘soul’.

Personally, I’m not that taken with the local tendency to demand that any problematic word be tabooed. But I think that it might have been worthwhile to make that demand of HopeFox when she first used the word ‘soul’.

Given my own background, I immediately attached a connotation of immortality upon seeing the word. And for that reason, I was puzzled at the conflation of moral worth with possession of a soul. Because my intuition tells me I should be more respectful of something that I might seriously damage than of someone that can survive anything I might do to it.

Perplexed Jun 12, 2011, 4:52 PM
33 points
on: How not to move the goalposts
Good choice of topic, but …

Unfortunately, most of the arguments I could choose for this discussion are either highly trivial or highly controversial. I’ll choose one that I hope won’t cause too much trouble.

I think it may be the worst possible choice. First, for suggesting that the question of compensation for engineers should be approached as a moral issue. Second, for failing to make the point that differences (between the sexes) in engineering aptitude in the general population says nothing about differences in engineering skill among people who have already been hired as engineers. Third, because gender differences between groups say little about differences between individuals. Fourth, because gender is a problematic subject in this forum, even when you do everything right.

I also second Nesov’s cringe at the implicit conflation of unpleasant and suitable-to-be-disagreed-with.

Perplexed Jun 12, 2011, 4:19 PM
0 points
in reply to: jimrandomh’s comment on: Welcome to Less Wrong!

… consider a hypothetical that’s as close to human as possible: uploads.

I would have suggested pets. Or the software objects of Chang’s story.

It is interesting that HopeFox’s intuitions rebel at assigning moral worth to something that is easily copied. I think she is on to something. The pets and Chang-software-objects which acquire moral worth do so by long acquaintance with the bestower of worth. In fact, my intuitions do the same with the humans whom I value.

Perplexed Jun 12, 2011, 3:58 PM
5 points
on: Help: Writing Marvin Minsky
Good project. Upvoted.

[Question from list] What probability do you assign to the possibility of a human level AI, respectively sub-human level AI, to self-modify its way up to massive superhuman intelligence within a matter of hours or days?

I think you are letting your own skepticism leak into your research here. The usual FOOM scenario starts from an AI that is already superhuman (say 2x or 10x human IQ) and allows weeks or months before it reaches the irreversible stage. In the Yudkowsky/Hanson debate, the figure of two years was most frequently used as the demarcation number distinguishing FOOM from something less dramatic.

Perplexed Jun 12, 2011, 3:32 PM
1 point
in reply to: [deleted]’s comment on: Unconditionally Convergent Expected Utility
Utility functions are not primitive. They are constructed using an algorithm specified by vN&M (or Savage, or A&A). Constructed from preferences over lotteries over outcomes. Preferences are primitive. Priors over states of nature are primitive. Utility functions are constructs. They are not arbitrary.

As has been mentioned, if you constrain preferences using one of the standard vN&M axioms, and if you assume that you can construct a lottery leading to any outcome, then you can prove that outcome utilities are bounded.

I think that the OP needs to be seen as a proposal for constraining the freedom to construct arbitrary lottery-probes. And, if the constraint is properly defined, we can have an algorithm that generates unbounded utilities, but not poorly behaved utilities—utilities which cannot be used to construct expectations that are not unconditionally convergent.

Perplexed Jun 11, 2011, 4:05 PM
0 points
in reply to: timtyler’s comment on: Safety Culture and the Marginal Effect of a Dollar

NancyL: Is there a body of knowledge about controlling self-modifying programs which could be used as a stepping stone to explaining what would be involved in FAI?

TimT: Self-modifying programs seems like a bit of a red herring. Most likely groups of synthetic agents will become capable of improving the design of machine minds before individual machines can do that. So, you would then have a self-improving ecosystem of synthetic intelligent agents. This probably helps with the wirehead problem …

I agree that a multi-agent systems perspective is the most fruitful way of looking at the problem. And I agree that coalitions are far less susceptible to the pathologies that can arise with mono-maniacal goal systems. A coalition of agents is rational in a different, softer way than is a single unified agent. For example, it might split its charitable contributions among charities. Does that weaker kind of rationality mean that coalitions should be denigrated? I think not.

To answer Nancy’s question, there is a huge and growing body of knowledge about controlling multi-agent systems. Unfortunately, so far as I know, little of it deals with the scenario in which the agents are busily constructing more agents.

Perplexed Jun 10, 2011, 10:31 PM
0 points
in reply to: DanielLC’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility
Yes, I understand conditional and unconditional convergence. What I don’t understand is how you get conditional convergence from

… if there was a 2^-n chance of getting 2^n utility and a 2^-n/n chance of getting −2^n/n utility (before normalizing)

I also do not understand how your “priors don’t allow that to happen”.

It almost seems that you are claiming to have unbounded utility but bounded expected utility. That is, no plausible sequence of events can make you confident that you will receive a big payoff, but you cannot completely rule it out.

Perplexed Jun 9, 2011, 1:42 PM
−4 points
in reply to: Eliezer Yudkowsky’s comment on: Safety Culture and the Marginal Effect of a Dollar
This comment seems to argue that “trying to assemble a team to solve basic FAI problems over however-many years and then afterward build FAI” is the real goal here and that “convincing someone to take something seriously” is barely worth thinking about. However, it certainly seems to me that convincing people to take the problem seriously is a productive (and perhaps an essential) first step toward assembling a team.

However, reading the subtext in that comment, it certainly appears that the real fear expressed here is that if safety consciousness should become endemic in the AGI community, there is a real risk that someone else might produce FAI before Eliezer.

Perplexed Jun 9, 2011, 1:12 PM
4 points
in reply to: XiXiDu’s comment on: Requirements for AI to go FOOM
I’m not disagreeing with your conclusions—only with your argument. In fact, several people have found flaws in your arguments. Technological impossibility ‘proofs’ are always flawed, IMHO. I’m a FOOM skeptic too. The arguments of FOOM believers are flawed. Point out those flaws. Don’t build flawed counter-arguments of your own.

You say someone who claims that there are design principles of intelligence that can take over the universe in a week is perfectly rational, while someone who says that such an belief is far-fetched makes up unwarranted assumptions.

Uh, no. I didn’t say that. What I said was more like “someone who claims to prove that such a belief is far-fetched is probably grounding their ‘proof’ on unwarranted assumptions.”

Perplexed Jun 8, 2011, 11:52 PM
1 point
on: Requirements for AI to go FOOM
Suppose AI_0 designs and builds an improved AI_1, which in turn designs and builds an even more powerful AI_2, which …, and so on. Does that count as recursive self-improvement?

If not, then I think you need to revise your definition of FOOMing.

If yes, then I think that P(P5) = 1.0.

This is an example of why technological skepticism arguments rarely are useful. The skeptic always makes unwarranted assumptions about how the technology must work. He then shows that it really can’t work like that. And then, like Vizzini in the Princess Bride, he announces that it is inconceivable that it might work any other way.

Perplexed Jun 8, 2011, 8:38 PM
1 point
in reply to: Normal_Anomaly’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility
Ah, yes. That works. Thanks.

Perplexed Jun 8, 2011, 8:36 PM
0 points
in reply to: Peterdjones’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility
Ok, but it seems to me that a virtue theorist must believe that information about virtue is a part of information about the state of the world. So does the consequentialist deny that all this virtue information is real information—information that can “pay rent” by generating correct anticipation of future experiences?

Odd, a few hours ago I thought I knew what a consequentialist was. But now I can’t seem to understand the concept regardless of whether I accept or reject Wei_Dai’s claim.

Perplexed Jun 8, 2011, 8:24 PM
2 points
in reply to: DanielLC’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility
Interesting. But it has been a while since I studied divergent series and the games that can be played with them. So more detail on your claim (“make the expected utility add to whatever you want by changing the order.”) would be appreciated.

It seems that you are adding one more axiom to the characterization of rationality (while at the same time removing the axiom that forces bounded utility.) Could you try to spell that new axiom out somewhat formally?

Perplexed Jun 8, 2011, 7:50 PM
1 point
in reply to: Wei Dai’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility
That leaves me curious as to what extraneous information a non-consequentialist sneaks into the utility function’s domain. The world’s state and the history of that state strike me as all there is.

Perplexed Jun 8, 2011, 5:40 PM
2 points
in reply to: lukeprog’s comment on: Pluralistic Moral Reductionism

I feel you’re being uncharitable.

I believe that you do feel that. But if you think that excluding people like Chappell from the discussion is a fruitful way to proceed, then I am curious why you believe that what you are discussing is properly termed “Meta-ethics”.

ETA: Ok, I’ll try for a bit more charity. Why does it matter whether a reduction of “ought” to physics can or cannot be accomplished? Why not simply present your ideas and then point out that they have the virtue of making ‘ought’ reducible?

It seems to me that you are trying too hard to be a good follower of Eliezer. Take my word for it; that isn’t necessary to be respected in this forum. It is possible (though admittedly sometimes expensive) to communicate without first achieving some kind of metaphysical solid ground.

Perplexed Jun 8, 2011, 2:36 PM
0 points
in reply to: lukeprog’s comment on: Pluralistic Moral Reductionism

… if somebody uses ‘ought’ terms to refer to things that can’t be reduced to physics …

Your emphasis on can’t. But do you really mean “can’t be reduced”? Or are you rather excluding anything that “hasn’t been reduced”? In the original posting you seem to be demanding that anyone using “ought language” be prepared to perform the reduction on the spot:

If they are making a claim about the world of is, then I ask them which part of the world of is they are discussing. I ask which ought-reductionism they have in mind.

I’m sorry, Luke. My commitment to reductionism is more of a vague hope for future physicalist explanation. I did not agree to use only that language which I can simultaneously show to be universally tabooable.

Perplexed Jun 8, 2011, 4:05 AM
1 point
in reply to: Morendil’s comment on: St. Petersburg Mugging Implies You Have Bounded Utility

Taken care of in the OP’s stipulations

Ah! I missed that. Thx. But I’m really not all that happy living in a world where TimFreeman was God, either. I suppose that means that I am not a real consequentialist.

Perplexed Jun 7, 2011, 9:52 PM
3 points
on: St. Petersburg Mugging Implies You Have Bounded Utility
I think that it is possible for me to have unbounded utility, yet still to assign a rather small utility to every outcome in any world in which TimFreeman is God (and I am not).

The same applies to Omega. If I do, in fact, live in a universe in which an omnipotent maniac performs psychological experiments, then much of my joy in living is lost.

There is an implicit assumption in all of these mugging scenarios that the existence of an all-powerful mugger who can intervene at any time has no effect on relative cardinal utilities of outcomes. That assumption seems unjustified.

Perplexed Jun 6, 2011, 8:43 PM
0 points
in reply to: ArisKatsaris’s comment on: More info about the book Eliezer is writing?
Proposal for first line: Stop Being Wrong!

Perplexed Jun 6, 2011, 2:12 AM
5 points
on: What are you working on? June 2011
I continue posting daily to the blog that I started two months ago. I’m a little disappointed at how little technical (or even technical-looking) content I have managed to publish so far, but I guess I can be a little bit proud of my sheer volume of published material.

A potential distraction from this self-imposed task just showed up. Sewing_machine’s recent discussion posting on the hypothetical inconsistency of mathematics contained links leading me to this hot topic in mathematical logic. Because I have been interested in Martin-Lof type theory and Lawvere’s program for some time, I think that this is something that I ought to try to understand. Even though homotopy theory has always been opaque to me. Anyone else want to try to work through the papers with me and then maybe try to make the core ideas accessible to the LW masses?