Aumann update works only if I believe you’re a perfect Bayesian rationalist. So, no thanks.
Too bad. Let’s just agree to disagree then, until the brain scanning technology is sufficiently advanced.
I’ve pointed out people who don’t wish for the examples you gave
So far, I didn’t see a convincing example of a person who truly wished for everyone to die, even in extrapolation.
Otherwise the false current beliefs will keep on being very relevant to them
To them, yes, but not to their CEV.
You could very easily build a much happier life for them just by allocating some resources (land, computronium, whatever) and going by their current values
Well… ok, lets assume a happy life is their single terminal value. Then by definition of their extrapolated values, you couldn’t build a happier life for them if you did anything else other than follow their extrapolated values!
In all of their behavior throughout their lives, and in their own words today, they honestly have this value
This is the conditional that I believe is false when I say “they are probably lying, trolling, joking”. I believe that when you use the brain scanner on those nihilists, and ask them whether they would prefer the world where everyone is dead to any other possible world, and they say yes, the brain scanner would show they are lying, trolling or joking.
Well, assuming EY’s view of intelligence, the “cautionary position” is likely to be a mathematical statement. And then why not prove it? Given several decades? That’s a lot of time.
Even if they do, it will be the best possible thing for them, according to their own (extrapolated) values.
we anticipate there will be no extrapolated wishes that literally everyone agrees on
Well, now you know there exist people who believe that there are some universally acceptable wishes. Let’s do the Aumann update :)
Lots of people religiously believe...
False beliefs ⇒ irrelevant after extrapolation.
Some others believe that life in this world is suffering, negative utility, and ought to be stopped for its own sake (stopping the cycle of rebirth)
False beliefs (rebirth, existence of nirvana state) ⇒ irrelevant after extrapolation.
My conditional was “cautionary position is the correct one”. I meant, provably correct.
How can we even start defining CEV without brain scanning technology able to do much more than answering the original question?
What is the chance some of them will try to seize first-mover advantage, and refuse to wait for another 30 years, and ignore Friendliness? I estimate high. The payoff is the biggest in human history: first-mover will potentially control a singleton that will rewrite to order the very laws of physics in its future light-cone, and prevent any other AGI from ever being built! This is beyond even “rule the world forever and reshape it in your image” territory. The greatest temptation ever. Do you seriously expect no-one would succumb to it?
Remember, we’re describing the situation where the cautionary position is provably correct. So your “greatest temptation ever” is (provably) a temptation to die a horrible death together with everyone else. Anyone smart enough to even start building AI would know and understand this.
I only proposed a hypothesis, which will become testable earlier than the time when CEV could be implemented.
CEV doesn’t and cannot predict that the extrapolated wishes of everybody will perfectly coincide. Rather, it says it will find the best possible compromise.
What I proposed in this thread is that CEV would forcibly implement only the (extrapolated) wish(es) of literally everyone. Regarding the rest, it is to minimize its influence, leaving all decisions to people.
Any true believer in a better afterlife qualifies
No, because they believe in afterlife. They do not wish for universal death. Extrapolating their wish with correct knowledge solves the problem.
how are these methods ethically or practically different from “forcibly changing their minds” or their values?
I (and CEV) do not propose changing their minds or their values. What happens is that their current values (as modeled within FAI) get corrected in the presence of truer knowledge and lots of intelligence, and these corrected values are used for guiding the FAI.
If someone’s mind & values are so closed as to be unextrapolateable—completely incompatible with truth—then I’m ok with ignoring these particular persons. But I don’t believe there are actually any such people.
But if AGI is not closer than several decades ahead, then, assuming the cautionary position is the correct one, it will become wide-spread and universally accepted. Any official well-funded teams will work with many safeguards and lots of constraints. Only stupid cranks will work without these, and they’ll work without funding too.
AFAIK, CEV is not well-defined or fully specified, except as a declaration of intent, a research direction. Thus, it does not make sense to say whether CEV as a model for FAI does or does not in fact do specific things. It only makes sense to say whether the intention of CEV’s developers for it to do or not do those things, and whether CEV’s specification so far contradicts or does not contradict those things.
AFAIU, CEV’s developers’ intent and CEV’s specification so far (with added “unanimousity” condition, if it is not present in the standard CEV specification) do not contradict my statement.
Think of “sentient life continuing on the planet” as a single concept, extrapolatable in various directions as becomes necessary. So, “planet” can be substituted by something else.
I assume FAI, which includes full (super-)human understanding of what is actually meant by “sentient life to continue”.
I’m very dubious of CEV as a model for Friendly AI. I think it’s a bad idea for several reasons. So, not that either.
I don’t see how this is possible. One can be dubious about whether it can be defined in the way it is stated, or whether it can be implemented. But assuming it can, why would it be controversial to fulfill the wish(es) of literally everyone, while affecting everything else the least?
when you extrapolate the volition of crazy people, their volition is not, in particular, more sane
Extrapolating volition includes correcting wrong knowledge and increasing intelligence. So, you do stop seeing lizard people if they don’t exist.
Serial killers are more interesting example. But they too don’t want everyone to die. Assuming serial killers get full knowledge of their condition and sufficient intelligence for understanding it, what would their volition actually be? I don’t know, but I’m sure it’s not universal death.
Yes, it is only a hypothesis. Until we actually built an AI with such CEV as utility, we cannot know whether it could function. But at least, running it is uncontroversial by definition.
And I think I’ll be more surprised if anyone was found who really and truly had a terminal value for universal death. With some strain, I can imagine someone preferring it conditionally, but certainly not absolutely. The members of doomsday cults, I expect, are either misinformed, insincere, or unhappy about something else (which FAI could fix!).
CEV that functioned only unanimously wouldn’t function at all
Why not? I believe that at least one unanimous extrapolated wish exists—for (sentient) life on the planet to continue. If FAI ensured that and left everything else for us to decide, I’d be happy.
I’d think someone’s playing a practical joke on me.