I don’t think that human values are well described by a PDU. I remember Daniel talking about a hidden reward tape at one point, but I guess that didn’t make it into this paper.
Peter_de_Blanc
Karma: 1,558
AGI/FAI Theorist for Hire
This tracks how good a god you are, and seems to make the paradox disappear.
How? Are you assuming that P(N) goes to zero?
LCPW cuts two ways here, because there are two universal quantifiers in your claim. You need to look at every possible bounded utility function, not just every possible scenario. At least, if I understand you correctly, you’re claiming that no bounded utility function reflects your preferences accurately.
That doesn’t sound like an expected utility maximizer.
It seems to me that expanding further would reduce the risk of losing the utility it was previously counting on.
what if the universe turns out to be much larger than previously thought, and the AI says “I’m at 99.999% of achievable utility already, it’s not worth it to expand farther or live longer”?
It’s not worth what?
Depth perception can be gained through vision therapy, even if you’ve never had it before. This is something I’m looking into doing, since I also grew up without depth perception.
Our disagreement on this matter is a consequence of our disagreement on other issues that would be very difficult to resolve, and for which there are many apparently intelligent, honest and well informed people on both sides. Therefore, it seems likely that reaching agreement on this issue would take an awful lot of work and wouldn’t be much more likely to leave us both right than to leave us both wrong.
You say that as if resolving a disagreement means agreeing to both choose one side or the other. The most common result of cheaply resolving a disagreement is not “both right” or “both wrong”, but “both −3 decibels.”
Obviously I didn’t mean that being broke (or anything) is infinite disutility.
Then what asymptote were you referring to?
I thought human utility over money was roughly logarithmic, in which case loss of utility per cent lost would grow until (theoretically) hitting an asymptote.
So you’re saying that being broke is infinite disutility. How seriously have you thought about the realism of this model?
I praise you for your right action.
First, I imagine a billion bits. That’s maybe 15 minutes of high quality video, so it’s pretty easy to imagine a billion bits. Then I imagine that each of those bits represents some proposition about a year—for example, whether or not humanity still exists. If you want to model a second proposition about each year, just add another billion bits.
Me: Well, you’re human, so I don’t think you can really have concerns about what happens a billion years from now because you can’t imagine that period of time.
In what sense are you using the word imagine, and how hard have you tried to imagine a billion years?
Instead of a strict straight/bi/gay split, I prefer to think of it as a spectrum where 0 is completely straight, 5 is completely bisexual and 10 is completely gay.
Hah! You’re trying to squish two axes into one axis. Why not just have an “attraction to males” axis and an “attraction to females” axis? After all, it is possible for both to be zero or negative.
OK, I guess my biggest complaint is this:
“If this approximation is close enough to the true value, the rest of the argument goes through: given that the sum Δx+Δy+Δz is fixed, it’s best to put everything into the charity with the largest partial derivative at (X,Y,Z).”
What does “close enough” mean? I don’t see this established anywhere in your post.
I guess one sufficient condition would be that a single charity has the largest partial derivative everywhere in the space of reachable outcomes.
I voted this post down. You claim to have done math, and you tell a narrative of doing math, but for the most part your math is not shown. This makes it difficult for someone to form an opinion of your work without redoing the work from scratch.
[Edit: I was unnecessarily rude here, and I’ve removed the downvote.]
Weirdtopia: sex is private. Your own memories of sex are only accessible while having sex. People having sex in public will be noticed but forgotten. Your knowledge of who your sex partners are is only accessible when it is needed to arrange sex. You will generally have warm feelings towards your sex partners, but you will not know the reason for these feelings most of the time, nor will you be curious. When you have sex, you will take great joy in realizing/remembering that this person you love is your sex partner.
If you want to predict how someone will answer a question, your own best answer is a good guess. Even if you think the other person is less intelligent than you, they are more likely to say the correct answer than they are to say any particular wrong answer.
Similarly, if you want to predict how someone will think through a problem, and you lack detailed knowledge of how that person’s mind happens to be broken, then a good guess is that they will think the same sorts of thoughts that a non-broken mind would think.
It is not available. The thinking on this matter was that sharing a bibliography of (what we considered) AGI publications relevant to the question of AGI timelines could direct researcher attention towards areas more likely to result in AGI soon, which would be bad.