Although, if you believe it always tells the truth, then you should follow whatever counterintuitive claim it makes about your own preferences and values, no? So if God were to tell you that sacrificing your son is what CEV_(Kawoomba) would do, would you do it?
I have a certain probability I ascribe to the belief that god always tells the truth, let’s say this is very high.
I also have a certain probability with which I believe that CEV_(Kawoomba) contained such a command. This is negligible because (from the definition) it certainly doesn’t fit with “were more the [man] [I] wished [I] were”.
However, we can lay that argument (evening out between a high and a very low probability) aside, there’s a more important one:
The point is that my values are not CEV_(Kawoomba), which is a concept that may make sense for an AI to feed with, or even to personally aspire to, but is not self-evidently a concept we should unequivocally aspire to. In a conflict between my values and some “optimized” (in whatever way) values that I do not currently have but that may be based on my current values, guess which ones win out? (My current ones.)
That aside, there is no way that the very foundation of my values could be turned topsy turvy and still fit with CEV’s mandate of “being the person I want to be”.
The point is that my values are not CEV_(Kawoomba)
You don’t mean … Kawoomba isn’t your real name?!!
Seriously, though, humans are not perfect reasoners, nor do we have perfect information. If we find onsomething that does, and it thinks our values are best implemented in a different way than we do, then we are wrong. Trivially so.
Although, if you believe it always tells the truth, then you should follow whatever counterintuitive claim it makes about your own preferences and values, no? So if God were to tell you that sacrificing your son is what CEV_(Kawoomba) would do, would you do it?
I have a certain probability I ascribe to the belief that god always tells the truth, let’s say this is very high.
I also have a certain probability with which I believe that CEV_(Kawoomba) contained such a command. This is negligible because (from the definition) it certainly doesn’t fit with “were more the [man] [I] wished [I] were”.
However, we can lay that argument (evening out between a high and a very low probability) aside, there’s a more important one:
The point is that my values are not CEV_(Kawoomba), which is a concept that may make sense for an AI to feed with, or even to personally aspire to, but is not self-evidently a concept we should unequivocally aspire to. In a conflict between my values and some “optimized” (in whatever way) values that I do not currently have but that may be based on my current values, guess which ones win out? (My current ones.)
That aside, there is no way that the very foundation of my values could be turned topsy turvy and still fit with CEV’s mandate of “being the person I want to be”.
You don’t mean … Kawoomba isn’t your real name?!!
Seriously, though, humans are not perfect reasoners, nor do we have perfect information. If we find onsomething that does, and it thinks our values are best implemented in a different way than we do, then we are wrong. Trivially so.
Or are you nitpicking the specification of “CEV”?