simon2

Karma: 83

simon2 6 Feb 2009 12:33 UTC
0 points
on: True Ending: Sacrificial Fire (7/8)
Psy-Kosh: I was using the example of pure baby eater values and conscious babies to illustrate the post Nick Tarleton linked to rather than apply it to this one.

Michael: if it’s “inevitable” that they will encounter aliens then it’s inevitable that each fragment will in turn encounter aliens, unless they do some ongoing pre-emptive fragmentation, no? But even then, if exponential growth is the norm among even some alien species (which one would expect) the universe should eventually become saturated with civilizations. In the long run, the only escape is opening every possible line from a chosen star and blowing up all the stars at the other ends of the lines.

Hmm. I guess that’s an argument in favour of cooperating with the superhappies. Though I wonder if they would still want to adopt babyeater values if the babyeaters were cut off, and if the ship would be capable of doing that against babyeater resistance.

simon2 6 Feb 2009 0:14 UTC
0 points
on: True Ending: Sacrificial Fire (7/8)
Nick, note that he treats the pebblesorters in parallel with the humans. The pebblesorters’ values lead them to seek primeness and Eliezer optimistically supposes that human values lead humans to seek an analogous rightness.

What Eliezer is trying to say in that post, I think, is that he would not consider it right to eat babies even conditional on humanity being changed by the babyeaters to have their values.

But the choice to seek rightness instead of rightness’ depends on humans having values that lead to rightness instead of rightness’.

simon2 5 Feb 2009 17:39 UTC
1 point
on: True Ending: Sacrificial Fire (7/8)
It’s evidence of my values which are evidence of typical human values. Also, I invite other people to really think if they are so different.

Eliezer tries to derive his morality from human values, rather than simply assuming that it is an objective morality, or asserting it as an arbitrary personal choice. It can therefore be undermined in principle by evidence of actual human values.

simon2 5 Feb 2009 12:24 UTC
6 points
on: True Ending: Sacrificial Fire (7/8)
Also, I think I would prefer blowing up the nova instead. The babyeater’s children’s suffering is unfortunate no doubt but hey, I spend money on ice cream instead of saving starving children in Africa. The superhappies’ degrading of their own, more important, civilization is another consideration.

(you may correctly protest about the ineffectiveness of aid—but would you really avoid ice cream to spend on aid, if it were effective and somehow they weren’t saved already?)

simon2 5 Feb 2009 12:04 UTC
0 points
on: True Ending: Sacrificial Fire (7/8)
If blowing up Huygens could be effective, why did it even occur to you to blow up Earth before you thought of this?

simon2 5 Feb 2009 1:26 UTC
0 points
on: Three Worlds Decide (5/8)
Sure it’s a story, but one with an implicit idea of human terminal values and such.

I’m actually inclined to agree with FarÃ© that they should count the desire to avoid a few relatively minor modifications over the eternal holocaust and suffering of baby-eater children.

I originally thought Eliezer was a utilitarian, but changed my mind due to his morality series.

(Though I still thought he was defending something that was fairly similar to utilitarianism. But he wasn’t taking additivity as a given but attempting to derive it from human terminal values themselves—so if human terminal values don’t say that we should apply equal additivity to baby-eater children, and I think they don’t, then Eliezer’s morality, I would have thought, would not apply additivity to them.)

This story however seems to show suspiciously utilitarian-like characteristics in his moral thinking. Or maybe he just has a different idea of human terminal values.

simon2 5 Feb 2009 0:54 UTC
0 points
on: Normal Ending: Last Tears (6/8)
...with babyeater values.

simon2 5 Feb 2009 0:53 UTC
0 points
on: Normal Ending: Last Tears (6/8)
Actually, I’m not sure if that’s what I thought about their intentions towards the babyeaters, but I at least didn’t originally expect them to still intend to modify themselves and humanity.

simon2 4 Feb 2009 23:59 UTC
3 points
on: Normal Ending: Last Tears (6/8)
John Maxwell:

No, they are simply implementing the original plan by force.

When I originally read part 5, I jumped to the same conclusion you did, based presumably on my prior expectations of what a reasonable being would do. But then I read nyu2′s comment which assumed the opposite and went back to look at what the text actually said, and it seemed to support that interpretation.

simon2 4 Feb 2009 1:42 UTC
0 points
on: Three Worlds Decide (5/8)
It seems we are at a disadvantage relative to Eliezer in thinking of alternative endings, since he has a background notion of what things are possible and what aren’t, and we have to guess from the story.

Things like:

How quickly can you go from star to star?
Does the greater advancement of the superhappies translate into higher travel speed, or is this constrained by physics?
Can information be sent from star to star without couriering it with a ship, and arrive in a reasonable time?
How long will the lines connected to the novaing star remain open?
Can information be left in the system in a way that it would likely be found by a human ship coming later?
Is it likely that there are multiple stars that connect the nova to one, two or all three alderson networks?

And also about behaviour:

Will the superhappies have the system they use to connect with the nova under guard?
How long will it be before the babyeaters send in another ship? the humans, if no information is received?
How soon will the superhappies send in their ships to begin modifying the babyeaters?

Here’s another option with different ways to implement it depending on the situation (possibly already mentioned by others, if so, sorry):

Cut off the superhappy connection, leaving or sending info for other humans to discover, so they deal with the babyeaters at their leisure.
Go back to give info to humans at Huygens, then cut off the superhappy connection.
Go back to get reinforcements, then quickly destroy the babyeater civilization (suicidally if necessary) and the novaing star (immediately after the fleet goes from it to the babyeater star(s), if necessary).

In all cases, I assume the superhappies will be able to guess what happened in retrospect. If not, send them an explicit message if possible.

simon2 3 Feb 2009 18:40 UTC
0 points
on: Three Worlds Decide (5/8)
… but relative to simply cooperating, it seems a clear win. Unless the superhappies have thought of it and planned a response.

Of course, the corollary for the real world would seem to be: those people who think that most people would not converge if “extrapolated” by Eliezer’s CEV ought to exterminate other people who they disagree with on moral questions before the AI is strong enough to stop them, if Eliezer has not programmed the AI to do something to punish that sort of thing.

Hmm. That doesn’t seem so intuitively nice. I wonder if it’s just a quantitative difference between the scenarios (eg quantity of moral divergence), or a qualitative one (eg. the babykillers are bad enough to justifiably be killed in the first place).

simon2 3 Feb 2009 18:14 UTC
0 points
on: Three Worlds Decide (5/8)
If the humans know how to find the babyeaters’ star,
and if the babyeater civilization can be destroyed by blowing up one star,

then I would like to suggest that they kill off the babyeaters.

Not for the sake of the babyeaters (I consider the proposed modifications to them better than annihilation from humanity’s perspective)
but to prevent the super-happies from making even watered down modifications adding baby-eater values -
not so much to humans, since this can also be (at least temporarily) prevented by destroying Huygens -
but to themselves, as they are going to be the dominant life form in the universe over time, being the fastest growing and advancing species.

Of course, relative to destroying Huygens the price to pay in terms of modifications to human values is high, so I would not make this decision lightly.

simon2 31 Jan 2009 22:57 UTC
0 points
on: War and/or Peace (2/8)
James Andrix: I don’t claim that the aliens would prefer modification over death, only that it is more consonant with my conception of human values to modify them than exterminate them, notwithstanding that the aliens may prefer the latter.

simon2 31 Jan 2009 14:21 UTC
3 points
on: War and/or Peace (2/8)
Akon claims this is a “true” prisoner’s dilemma situation, and then tries to add more values to one side of the scale. If he adds enough values to make cooperation higher value than defecting, then he was wrong to say it was a true prisoner’s dilemma. But the story has made it clear that the aliens appear to be not smart enough to accurately anticipate human behaviour (or vice versa for that matter), so this is not a situation where it is rational to cooperate in a true prisoner’s dilemma. If it really is a true prisoner’s dilemma, they should just defect.

Of course, there may be a more humane approach than extermination or requiring them to live under human law: forcible modification to remove the desire to eat babies, and reduce the amount of reproduction. It might be a little tricky to do this without completely messing up the aliens’ psychology.

Also, it seems a little unlikely that a third ship would arrive given that the arrival of even one alien ship was considered so surprising in the first installment.

simon2 5 Jan 2009 1:52 UTC
1 point
on: Changing Emotions
Strong enough to disrupt personal identity, if taken in one shot? That’s a difficult question to answer, especially since I don’t know what experiment to perform to test any hypotheses. On one hand, billions of neurons in my visual cortex undergo massive changes of activation every time my eyes squeeze shut when I sneeze—the raw number of flipped bits is not the key thing in personal identity. But we are already talking about serious changes of information, on the order of going to sleep, dreaming, forgetting your dreams, and waking up the next morning as though it were the next moment.

It sounds as if you believe in a soul (or equivalent) that is “different” for some set of possible changes and “the same” for other possible changes. I would suggest that that whether an entity at time n+1 is the same person as you at time n is not an objective fact of the universe. Humans have evolved so that we consider the mind that wakes up in the body of the mind that went to sleep to be the same person, but this intuitive sense is not an intuitive understanding of an objective reality; one could modify oneself to consider sleep to disrupt identity, and this would not be a “wrong” belief but just a different one.

I think most people are most comfortable retaining their evolution-given intuitions where they are strong, but where they are weak I think it is a mistake to try to overgeneralize them; instead one should try to shape them consciously. If you want to try being female for a while, why spoil your fun with hang ups about identity? Just decide that it’s still you.

simon2 16 Dec 2008 2:11 UTC
0 points
on: For The People Who Are Still Alive
Eliezer, whenever you start thinking about people who are completely causally unconnected with us as morally relevant, alarm bells should go off.

What’s worse though, is that if your opinion on this is driven by a desire to justify not agreeing with the “repugnant conclusion”, it may signify problems with your morality that could annihilate humanity if you give your morality to an AI. The repugnant conclusion requires valuing the bringing into existence of hypothetical people with total utility x by as much as reducing the utility of existing people by x, or annihilating people with utility x. Give that morality to a fast takeoff AI and they’ll quickly replace all humans with entities with greater capacity for utility. If the AI is programmed to believe the problem with the “repugnant conclusion” is what you claim, the AI will instead create randomized (for high uniqueness) minds with high capacity for utility, still annihilating humans.

simon2 25 Oct 2008 0:46 UTC
1 point
on: Expected Creative Surprises
I think what he means by “calibrated” is something like it not being possible for someone else to systematically improve the probabilities you give for the possible answers to a question just from knowing what values you’ve assigned (and your biases), without looking at what the question is.

I suppose the improvement would indeed be measured in terms of relative entropy of the “correct” guess with respect to the guess given.

simon2 24 Oct 2008 20:33 UTC
0 points
on: Which Parts Are “Me”?
Responding to Gaffa (I kind of intended to respond right after the comment, bot got sidetracked):

When approaching a scientific or mathematical problem, I often find myself trying hard to avoid having to calculate and reason, and instead try to reach for an “intuitive” understanding in the back of my mind, but that understanding, if I can even find it, is rarely sufficient when dealing with actual problems.

I would advise you to embrace calculation and reason, but just make sure you think about what you are doing and why. Use the tools, but try to get an intuitive understanding of why and how they work, both in general and each time you apply them. It is true that formualaic rules can serve as a crutch to avoid the need for understanding, but if you throw away calculation and reason, you are not likely to make much progress.

Finally, be realistic in your expectations: for a complicated problem, you should not expect to be able to get an intuitive understanding of the solution as a single step, but you can aim for a chain of individually intuitive steps and, if the chain is sufficiently short, an overall intuitive understanding of how the steps relate to one another.

simon2 21 Oct 2008 17:31 UTC
24 points
on: Prices or Bindings?
It might make an awesome movie, but if it were expected behaviour, it would defeat the point of the injunction. In fact if rationalists were expected to look for workarounds of any kind it would defeat the point of the injunction. So the injunction would have to be, not merely to be silent, but not to attempt to use the knowledge divulged to thwart the one making the confession in any way except by non-coercive persuasion.

Or alternatively, not to ever act in a way such that if the person making the confession had expected it they would have avoided making the confession.

simon2 19 Oct 2008 23:34 UTC
0 points
on: Ethical Inhibitions
To the extent that a commitment to ethics is externally verifiable, it would encourage other people to cooperate, just as a tendency to anger (a visible commitment to retribution) is a disincentive to doing harm.

Also, even if it is not verifiable, a person who at least announces their intention to hold to an ethical standard has raised the impact their failure to do so will have on their reputation, and thus the announcement itself should have some impact on the expectation that they will behave ethically.