This isn’t translatable as a function. ‘Meaningful’ and ‘meaningless’ aren’t values bivalent functions return so they shouldn’t be values in our logic.
So the sentence “The sentence ‘Everything written on the board in Room 33 is either false or meaningless.’ is meaningless” is not true?
Yes, humans performing outstandingly well in this sort of problem was my inspiration for this. I am not sure how far it is possible to generalize this sort of winning. Humans themselves are kinda complex machines, so, if we start with perfectly rational LW reader and paperclip maximizer with one-shot PD with randomized payoff matrix, what’s the least amount of handicaps we need to give them to reach this super-optimal solution? At first, I thought we could even remove the randomization alltogether, but it is making the whole problem more ambiguous I think.
Becoming a person doesn’t seem like something that you can do free of cost. There seems to be a lot of complexity hidden in that “Become a person” part.
Those properties that we think makes happy humans better than totally artificial smiling humans mimicing happy humans. You’d need to find it in order to grasp what it means to have a being that lacked moral value, and “both ideas” refers to the distinct ways of explaining what sort of paperclip maximizer we’re talking about.
But I’d think if I only said “It doesn’t have moral value in itself”, you’d still have to go back similar steps to find that property cluster that we assign value. I tried to transfer both ideas by using the word soul and claiming lack of moral value.
It requires us to know what sort of utility function the other player has, at the very least, and even then the result might be, at best, mutual defect or, against superrational players, mutual co-operation.
And? If you have multiple contradictory wishes what to do next, some of them are bound to be unfulfilled. CEV or negotiation are just ways to decide which ones.
Why do you think I lose?
Because there are a lot more of those with values totally different from yours, which made the CEV optimize a future that you didn’t like at all. If you’re negotiating will all those people, why would they give in to you any more than CEV would optimize for you?
So you’re bound to end up losing in this game, anyway, right? Negotiation in itself won’t bring you any additional power over the coherent extrapolated volition of humanity to change the future of the universe. If others think very much unlike you, you need to overpower them to bring your values back to the game or perish in the attempt.
The above is a caricature of ‘coherence’ as presented in the May 2004 document. If someone else can provide a better interpretation, that would be welcome.
It seemed accurate to me. Also, I didn’t find any problems from it that would seem frightening or so. Was it supposed to be problematic in some way?
Just an attempt to make it clear that we’re dealing with something like an intelligent calculator here with nothing in it that we’d find interesting or valuable in itself. Setting up this as the true PD.
So only by knowing about both dilemmas can you define value. Since, by the problem definition, you only know of one of them, defining relative value is impossible.
You’d only need to know what sort of utility function for that. Sure, you wouldn’t know that the other deal like that is happening right there, so it might not be possible to reach counterfactual agreement in this particular case, but as a general rule, it does seem possible to outperform agents that don’t go for this sort of “I take a hit for you if you’d take a hit for me in this sort of scenario where stakes are reversed”. Which leaves multiple questions open, I agree.
You haven’t read the sequences, have you? The idea of using evolution to produce safe-enough superintelligences was destroyed quite neatly there, say, here: http://lesswrong.com/lw/td/magical_categories/
Also, when we’re talking about artificial intelligences, the time period between the point “They’re intelligent enough to have some sort of ethical value” and the point “They’re intelligent enough to totally dominate us” is most likely really, really short, I’d say less than 10 years, some could say less than 10 days.
The assumption about superrationality is now much more explicitly stated.
I admit taking “Rational people playing true, one-shot PD with beings as rational as they, co-operate” for granted. I didn’t think this was going to be an issue, and so, since I’m building upon that as an axiom, things might look weird if you think that foundation is untrue. And for this reason, I’m unsure if this discussion should continue here. If you’re alone with that opinion, I think this discussion should take place elsewhere, but if there are many who disagree with me on that basic level, I guess that discussion should happen here.
How does the Multiverse know, I am just sleeping for 24 (or 24000) hours? How the Multiverse knows, I’ll not be rescued after the real suicide attempt after a quantum coin head popped up?
Because you won’t be back. Universe has the whole eternity to just wait for you to come back. If you don’t, the only remaining ones that keep on experiencing from where you left off are the branches where coin didn’t come heads.
Read chapter 2.
Oh dear.
I wish this sequel had long phase where things just flow peacefully. First chapter was fun, but I fear that things are gonna get messy after that sort ending.
Oh, right, now I get it.