Sebastian_Hagen2

Karma: 189

Sebastian_Hagen2 Jul 29, 2008, 3:20 PM
2 points
on: The Meaning of Right
Thank you for this post. “should” being a label for results of the human planning algorithm in backward-chaining mode the same way that “could” is a label for results of the forward-chaining mode explains a lot. It’s obvious in retrospect (and unfortunately, only in retrospect) to me that the human brain would do both kinds of search in parallel; in big search spaces, the computational advantages are too big not to do it.

I found two minor syntax errors in the post: “Could make sense to …”—did you mean “Could it make sense to …”? “(something that has a charge of should-ness”—that parenthesis is never closed.

Unknown wrote:

As I’ve stated before, we are all morally obliged to prevent Eliezer from programming an AI.
Speak for yourself. I don’t think EliezerYudkowsky::Right is quite the same function as SebastianHagen::Right, but I don’t see a real chance of getting an AI that optimizes only for SebastianHagen::Right accepted as sysop. I’d rather settle for an acceptable compromise in what values our successor-civilization will be built on than see our civilization being stomped into dust by an entirely alien RPOP, or destroyed by another kind of existential catastrophe.

Sebastian_Hagen2 Jul 16, 2008, 11:55 AM
0 points
on: Whither Moral Progress?
It’s harder to answer Subhan’s challenge—to show directionality, rather than a random walk, on the meta-level.
Even if one is ignorant of what humans mean when they talk about morality, or what aspects of the environment influence it, it should be possible to determine whether morality-development over time follows a random walk empirically: a random walk would, on average, cause more repeated reversals of a given value judgement than a directional process.
For performing this test, one would take a number of moral judgements that have changed in the past, and compare their development from a particular point in human history (the earlier, the better; unreversed recent changes may have been a result of the random walk only becoming sufficiently extreme in the recent past) to now, counting how often those judgements flipped during historical development. I’m not quite sure about the conditional probabilities, but a true random walk should result in more such flips than a directional (even a noisy directional) process.
Does anyone have suggestions for moral values that changed early in human development?

Sebastian_Hagen2 Jul 5, 2008, 1:14 PM
0 points
on: Is Morality Preference?
Regarding the first question,

Why do people seem to mean different things by “I want the pie” and “It is right that I should get the pie”?
I think the meaning of “it is (morally) right” may be easiest to explain through game theory. Humans in the EEA had plenty of chances for positive-sum interactions, but consistently helping other people runs the risk of being exploited by defection-prone agents. Accordingly, humans may have evolved a set of adaptions to exploit non-zero sumness between cooperating agents, but also avoid cooperating with defectors. Treating “X is (morally) right” as a warning of the form “If you don’t do X, I will classify that as defection” explains a lot. Assume a person A has just (honestly) warned a person B that “X is the right thing to do”:

If B continues not do X, A will likely be indignant; indignancy means A will be less likely to help B in the future (which makes sense according to game theory), and might also recommend the same to other members of the tribe. B might accept the claim about rightness; this will make it more likely for him to do the “right” thing. Since, in the EEA, being ostracized by the tribe would result in a significant hit to fitness, it’s likely for there to be an adaption predisposing people to evaluate claims about rightness in this manner. B’s short-term desires might override his sense of “moral rightness”, leading to him doing the (in his own conception) “wrong” thing. While B can choose to do the wrong thing, he cannot change which action is right by a simple individual decision, since the whole point of evaluating rightness at all is to evaluate it the same way as other people you interact with.

According to this view, moral duties function as rules which help members of a society to identify defectors (by defectors violating them).

Sebastian_Hagen2 Jul 3, 2008, 3:08 PM
7 points
on: The Bedrock of Fairness
This post reminds me a lot of DialogueOnFriendliness.

There’s at least one more trivial mistake in this post:

Is their nothing more to the universe than their conflict?
s/their/there/

Constant wrote:

Arguably the difficulty the three have in coming to a conclusion is related to the fact that none of the three has anything close to a legitimate claim on the pie.
If you modify the scenario by postulating that the pie is accompanied by a note reading “I hereby leave this pie as a gift to whomever finds it. Enjoy. -- Flying Pie-Baking Monster”, how does that make the problem any easier?

Sebastian_Hagen2 Jul 1, 2008, 6:45 PM
0 points
on: The Moral Void
Hal Finney:
Why doesn’t the AI do it verself? Even if it’s boxed (and why would it be, if I’m convinced it’s an FAI?), at the intelligence it’d need to make the stated prediction with any degree of confidence, I’d expect it to be able to take over my mind quickly. If what it claims is correct, it shouldn’t have any qualms about doing that (taking over one human’s body for a few minutes is a small price to pay for the utility involved).
If this happened in practice I’d be confused as heck, and the alleged FAI being honest about its intentions would be pretty far down on my list of hypotheses about what’s going on. I’d likely stare into space dumbfounded until I found some halfway-likely explanation, or the AI decided to take over my mind after all.

Sebastian_Hagen2 Jun 29, 2008, 9:25 PM
0 points
on: What Would You Do Without Morality?
Are there no vegetarians on OvBias?
I’m a vegetarian, though not because I particularly care about the suffering of meat animals.

Sebastian Hagen, people change. Of course you may refuse to accept it, but the current you will be dead in a second, and a different you born.
Of course people change; that’s why I talked about “future selves”—the interesting aspect isn’t that they exist in the future, it’s that they’re not exactly the same person as I am now. However, there’s still a lot of similarity between my present self and my one-second-in-the-future self, and they have effectively the same optimization target. Moreover, these changes are largly non-random and non-degenerative: a lot of them are a part of my mind improving its model of the universe and getting more effective at interacting with it.
I don’t think it is appropriate to term such small changes “death”. If an anvil drops on my head, crushing my brain to goo, I immediately lose more optimization power than I do in a decade of living without fatal accidents. The naive view of personal identity isn’t completely accurate, but the reason that it works pretty well in practice is that (in our current society) humans don’t change particularly quickly, except for when they suffer heavy injuries.
The anvil-dropped-on-head-scenario is what I envisioned in my last post: something annihilating or massively corrupting my mind, destroying the part that’s responsible for evaluating the desirability of hypothetical states of the universe.

Sebastian_Hagen2 Jun 29, 2008, 5:32 PM
1 point
on: What Would You Do Without Morality?
Suppose you learned, suddenly and definitively, that nothing is moral and nothing is right; that everything is permissible and nothing is forbidden.
I’m a physical system optimizing my environment in certain ways. I prefer some hypothetical futures to others; that’s a result of my physical structure. I don’t really know the algorithm I use for assigning utility, but that’s because my design is pretty messed up. Nevertheless, there is an algorithm, and it’s what I talk about when I use the words “right” and “wrong”.
Moral rightness is fundamentally a two-place function: it takes both an optimization process and a hypothetical future as arguments. In practice, people frequently use the curried form, with themselves as the implied first argument.

Suppose I proved that all utilities equaled zero.
That result is obviously false for my present self. If the proof pertains to that entity, it’s either incorrect or the formal system it is phrased in is inappropriate for modeling this aspect of reality.
It’s also false for all of my possible future selves. I refuse to recognize something which doesn’t have preferences over hypothetical futures as a future-self of me; whatever it is, it’s lost too many important functions for that.

Sebastian_Hagen2 Jun 18, 2008, 3:59 PM
0 points
on: Ghosts in the Machine
Here’s my vision of this, as a short scene from a movie. Off my blog: The Future of AI
To me, the most obvious reading of that conversation is that a significant part of what the AI says is a deliberate lie, and Anna is about to be dumped into a fun-and-educational adventure game at the end. Did you intend that interpretation?

Sebastian_Hagen2 Jun 8, 2008, 10:52 AM
0 points
on: Timeless Control
Eliezer:

If you think as though the whole goal is to save on computing power, and that the brain is actually fairly good at this (it has to be), then you won’t go far astray.
Ah, thanks! I hadn’t considered why you would think about isolated subystems in practice; knowing about the motivation helps a lot in filling in the implementation details.

Sebastian_Hagen2 Jun 7, 2008, 10:32 AM
4 points
on: Timeless Control
I’m trying to see exactly where your assertion that humans actually have choice comes in.
“choice” is a useful high-level abstraction of certain phenomena. It’s a lossy abstraction, and if you had infinite amounts of memory and computing power, you would have no need for it, at least when reasoning about other entities. It exists, in exactly the same way in which books (the concept of a book is also a high-level abstraction) exist.
If that sounded wrong or like nonsense to you, please taboo “choice” and explain what exactly your question is.

I also have a question of my own, regarding the rock-hill-system:

If you isolate a subsystem of reality, like a rock rolling down hill, then you can mathematically define the future-in-isolation of that subsystem; you can take the subsystem in isolation, and compute what would happen to it if you did not act on it. In this case, what would happen is that the rock would reach the bottom of the hill.
How does this isolation work? Do you assume that the forces acting on the system from outside stay constant (in some undefined fashion), without explicitly modeling the outside? If I assume no further interactions with the outside, I don’t expect to see the rock rolling down the hill, since there’s no planet below to gravitationally attract it. Or was the planet supposed to be part of this system?

Sebastian_Hagen2 Jun 4, 2008, 7:28 AM
0 points
on: Timeless Identity
What if cryonics were phrased as the ability to create an identical twin from your brain at some point in the future, rather than ‘you’ waking up. If all versions of people are the same, this distinction should be immaterial. But do you think it would have the same appeal to people?
I don’t know, and unless you’re trying to market it, I don’t think it matters. People make silly judgements on many subjects, blindly copying the majority in this society isn’t particularly good advice.

Each twin might feel strong regard for the other, but there’s no way they would actually be completely indifferent between pain for themselves and pain for their twin.
Any reaction of this kind is either irrational, based on divergence which has already taken place, or based on value systems very different from my own. In real life, you’d probably get a mix of the first two, and possibly also the last, from most people.

If another ‘me’ were created on mars and then got a bullet in the head, this would be sad, but no more so than any other death. It wouldn’t feel like a life-extending boon when he was created, nor a horrible blow to my immortality when he was destroyed.
For me, this would be a quantitative judgement: it depends on how much both instances have changed since the split. If the time lived before the split is significantly longer than that after, I would consider the other instance a near-backup, and judge the relevance of its destruction accordingly. Aside from the aspect of valuing the other person as a human like any other that also happens to share most of your values, it’s effectively like losing the only (and somewhat out-of-date) backup of a very important file: No terrible loss if you can keep the original intact until you can make a new backup, but an increased danger in the meantime.

If you truly believe that ‘the same atoms means its ‘you’ in every sense’, suppose I’m going to scan you and create an identical copy of you on mars. Would you immediately transfer half your life savings to a bank account only accessible from mars? What if I did this a hundred times?
Maybe, maybe not, depends on the exact strategy I’d mapped out beforehand for what each of the copies will do after the split. If I didn’t have enough foresight to do that beforehand, all of my instances would have to agree on the strategy (including allocation of initial resources) over IRC or wiki or something, which could get messy with a hundred of them—so please, if you ever do this, give me a week of advance warning. Splitting it up evenly might be ok in the case of two copies (assuming they both have comparable expected financial load and income in the near term), but would fail horribly for a hundred; there just wouldn’t be enough money left for any of them to matter at all (I’m a poor university student, currently; I don’t really have “life savings” in transferrable format).

Sebastian_Hagen2 Jun 3, 2008, 5:57 PM
0 points
on: Timeless Identity
Is the ‘you’ on mars the same as ‘you’ on Earth?
There’s one of you on earth, and one on mars. They start out (by assumption) the same, but will presumably increasingly diverge due to different input from the environment. What else is there to know? What does the word ‘same’ mean for you?

And what exactly does that mean if the ‘you’ on earth doesn’t get to experience the other one’s sensations first hand? Why should I care chat happens to him/me?
That’s between your world model and your values. If this happened to me, I’d care because the other instance of myself happens to have similar values to the instance making the judgement, and will therefore try to steer the future into states which we will both prefer.

Sebastian_Hagen2 May 23, 2008, 11:20 AM
2 points
on: My Childhood Role Model
But I don’t buy the idea of intelligence as a scalar value.
Do you have a better suggestion for specifying how effective a system is at manipulating its environment into specific future states? Unintelligent systems may work much better in specific environments than others, but any really intelligent system should be able to adapt to a wide range of environments. Which important aspect of intelligence do you think can’t be expressed in a scalar rating?

Sebastian_Hagen2 May 13, 2008, 8:05 PM
7 points
on: The Dilemma: Science or Bayes?
They only depend to within a constant factor. That’s not the problem; the REAL problem is that K-complexity is uncomputable, meaning that you cannot in any way prove that the program you’re proposing is, or is NOT, the shortest possible program to express the law.
I disagree; I think the underspecification is a more serious issue than the uncomputability. There are constant factors that outweigh, by a massive margin, all evidence ever collected by our species. Unless there’s a way for us to get our hands on an infinite amount of cputime, there are constant factors that outweigh, by a massive margin, all evidence we will ever have a chance to collect. For any two strings, you can assign a lower complexity to either one by choosing the description language appropriately. Some way to make a good enough (not necessarily optimal) judgement on the language to use is needed for the complexity metric to make any sense.

The uncomputability is unfortunate, but hardly fatal. You can just spend some finite effort trying to find the shortest program that produces the each string, using the best heuristics available for this job, and use that as an approximation and upper bound. If you wanted to turn this into a social process, you could reward people for discovering shorter programs than the shortest-currently-known for existing theories (proving that they were simpler than known up to that point), as well as for collecting new evidence to discriminate between them.

Sebastian_Hagen2 May 13, 2008, 11:41 AM
4 points
0
on: The Dilemma: Science or Bayes?
But when I say “macroscopic decoherence is simpler than collapse” it is actually strict simplicity; you could write the two hypotheses out as computer programs and count the lines of code.
Computer programs in which language? The kolmogorov complexity of a given string depends on the choice of description language (or programming language, or UTM) used. I’m not familiar with MML, but considering that it’s apparently strongly related to kolmogorov complexity, I’d expect its simplicity ratings to be similarly dependent on parameters for which there is no obvious optimal choice.

If one uses these metrics to judge the simplicity of hypotheses, any probability judgements based on them will ultimately depend strongly on this parameter choice. Given that, what’s the best way to choose these parameters? The only two obvious ways I see are to either 1) Make an intuitive judgement, which means the resulting complexity ratings might not turn out any more reliable than if you intuitively judged the simplicity of each individual hypothesis, or 2) Figure out which of the resulting choices can be implemented cheaper in this universe; i.e. try to build the smallest/least-energy-using computer for each reasonably-seeming language, and see which one turns out cheapest. Since resource use at runtime doesn’t matter for kolmogorov complexity, it would probably be appropriate to consider how well the designs would work if scaled up to include immense amounts of working memory, even if they’re never actually built at that scale.

Neither of those is particularly elegant. I think 2) might work out, but unfortunately is quite sensitive to parameter choice, itself.

Sebastian_Hagen2 May 12, 2008, 11:51 AM
11 points
on: The Failures of Eld Science
“A short time?” Jeffreyssai said incredulously. “How many minutes in thirty days? Hiriwa?”

“28800, sensei,” she answered. “If you assume sixteen-hour waking periods and daily sleep, then 19200 minutes.”I would have expected the answers to be 43200 (30d 24h/d 60/h) and 28800 (30d 16h/d 60/h), respectively. Do these people use another system for specifying time? It works out correctly if their hours have 40 minutes each.

Aside from that, this is an extremely insightful and quote-worthy post. I have^W^W My idiotic past-selves had a bad tendency to cognitively slow down in the absence of interesting and time-critical problems to solve. Accordingly, I find the hints about how to debug those tendencies very interesting. I find it rather quaint that those people still spend a significant part of their time sleeping, however.

Sebastian_Hagen2 May 10, 2008, 2:40 PM
4 points
0
on: If Many-Worlds Had Come First
I hope the following isn’t completely off-topic:

… if I’d been born into that time, instead of this one...
What exactly does a hypothetical scenario where “person X was born Y years earlier” even look like? I could see a somewhat plausible interpretation of that description in periods of extremely slow scientific and technological progress, but the twentieth century doesn’t qualify. In the 1920s: 1) The concept of a turing machine hadn’t been formulated yet. 2) There were no electronic computers. 3) ARPANET wasn’t even an idea yet, and wouldn’t be for decades. 4) Television was a novelty, years away from being used by a significant number of people. 5) WW1 was recent history.

Two persons with the same DNA and, except for results of global changes, very similar local environments during their childhood, would most likely turn into completely different adult humans if one of them was born in the 1920s and the other at some point in the last 30 years (roughly chosen to guarantee exposure to the idea of the internet as a teenager), and they both grew up in industrialized countries. The scientific and technological level one is born into is critical for mind development. What does it mean to consider a hypothetical world where a specific person was born into an environment very different in those respects? Why is this worth thinking about?

Sebastian_Hagen2 Apr 27, 2008, 10:26 AM
0 points
on: On Being Decoherent
Maybe later I’ll do a post about why you shouldn’t panic about the Big World. You shouldn’t be drawing many epistemic implications from it, let alone moral implications. As Greg Egan put it, “It all adds up to normality.” Indeed, I sometimes think of this as Egan’s Law.
While I’m not currently panicking about it, I’d be very interested in reading that explanation. It currently seems to me that there should be certain implications, e.g. in Quantum suicide experiments. If mangled worlds says that the entity perfoming such an experiment should not expect to survive many iterations, that doesn’t solve the space-like version of the issue: Some of the person’s alternate-selves on far away alternate-earths would be prevented from carrying out their plan by weird stuff (TM) coming in from space at just the right time.

Hopefully Anonymous asked:

10^(10^29) (is this different than 10^30?)
It’s different by a factor of roughly 10^(10^29). Strictly speaking the factor is 10^(10^29-30), but making that distinction isn’t much more meaningful than distinguishing between metres and lightyears at those distances.

Sebastian_Hagen2 Apr 25, 2008, 1:13 PM
1 point
on: Where Physics Meets Experience
Good writing, indeed! I also love what you’ve done with the Eborrian anzrf (spoiler rot13-encoded for the benefit of other readers since it hasn’t been mentioned in the previous comments).

The split/remerge attack on entities that base their anticipations of future input directly on how many of their future selves they expect to get specific input is extremely interesting to me. I originally thought that this should be a fairly straightforward problem to solve, but it has turned out a lot harder (or my understanding a lot more lacking) than I expected. I think the problem might be in the group of 500,003 brains double-counting anticipated input after the merge. They don’t stay exactly the same through the merge phase; in fact, for each of the 500,000 brains in green rooms, the re-integrated previously-in-green-rooms brain only depends to a very small part on them individually. In this particular case, the re-integrated brain will still be very similar to each of the pre-integration brains; but that is just a result of the pre-integration brains all being very similar to each other. Treating the re-integrated brain as a regular future-self for the purposes of anticipating future experience under these conditions seems highly iffy to me.

Sebastian_Hagen2 Apr 21, 2008, 5:53 PM
5 points
on: Three Dialogues on Identity
Similarly to “Zombies: The Movie”, this was very entertaining, but I don’t think I’ve learned anything new from it.

Z. M. Davis wrote:

Also, even if there are no moral facts, don’t you think the fact that no existing person would prefer a universe filled with paperclips …
Have you performed a comprehensive survey to establish this? Asserting “No existing person” in a civilization of 6.5e9 people amounts to assigning a probability of less than 1.54e-10 that a randomly chosen person would prefer a universe filled with paperclips. This is an extremely strong claim to make!

For example, note that the set of people alive includes a significant number of people who are certifiably insane, and in all probability others who, while reasonably sane, have gotten very fed up with various forms of torture inflicted on them over the last few days and might be willing to neglect collateral damage if they could make it stop.

If such a survey were performed, and the results were actually what you claim, I would assign a higher probability to the possibility of a nefarious anti-paperclip conspiracy having infiltrated the survey effort than to the possibility of the results being correct.

Unanimous agreement of our entire species is also a much stronger claim than you need to make for your argument.