I mean, for example, if it turns out that implementing CEV creates a future that everyone living in desires and is made happy and fulfilled and satisfied by and continues to do so indefinitely, and that everyone living now would if informed of the details of also desire and etc., but we are never able to confirm that any of that is right… or worse yet, later philosophical analysis somehow reveals that it isn’t right, despite being desirable and fulfilling and satisfying and so forth… well, OK, we can decide at that time whether we want to give up what is desirable and etc. in exchange for what is right, but in the meantime I might well be satisfied by that result. Maybe it’s OK to leave future generations some important tasks to implement.
Or, if it turns out that EY’s approach is all wrong because nobody agrees on anything important to anyone, so that extrapolating humanity’s coherent volition leaves out everything that’s important to everyone, so that implementing it doesn’t do anything important… in that case, coming up with an alternate plan that has results as above would satisfy me.
It might turn out that what does satisfy us is to be “free”, to do what we want, even if that means that we will mess up our own future. It might turn out that humans are only satisfied if they can work on existential problems, “no risk no fun”. Or we might simply want to learn about the nature of reality. The mere existence of an FAI might spoil all of it. Would you care to do science if there was some AI-God that already knew all the answers? Would you be satisfied if it didn’t tell you the answers or made you forget that it does exist so that you’d try to invent AGI without ever succeeding?
But there is another possible end. Even today many people are really bored and don’t particularly enjoy life. What if it turns out that there is no “right” out there or that it can be reached fairly easily without any way to maximize it further. In other words, what if fun is something that isn’t infinite but a goal that can be reached? What if it all turns out to be wireheading, the only difference between 10 minutes of wireheading or 10^1000 years being the number enumerating the elapsed time? Think about it, would you care about 10^1000 years of inaction? What would you do if that was the optimum? Maybe we’ll just decide to choose the void instead.
This is a different context for satisfaction, but to answer your questions:
yes, I often find satisfying working through problems that have already been solved, though I appreciate that not everyone does;
no, I would not want to be denied the solutions if I asked (assuming there isn’t some other reason why giving me the solution is harmful), or kept in ignorance of the existence of those solutions (ibid);
if it turns out that all of my desires as they currently exist are fully implemented, leaving me with no room for progress and no future prospects better than endless joy, fulfillment and satisfaction, I’d be satisfied and fulfilled and joyful.
Admittedly, I might eventually become unsatisfied with that and desire something else, at which point I would devote efforts to satisfying that new desire. It doesn’t seem terribly likely that my non-existence would be the best possible way of doing so, but I suppose it’s possible, and if it happened I would cease to exist.
OK, we can decide at that time whether we want to give up what is desirable and etc. in exchange for what is right, but in the meantime I might well be satisfied by that result
No, once ostensibly-Friendly AI has run CEV and knows what it wants, it won’t matter if we eventually realize that CEV was wrong after all. The OFAI will go on to do what CEV says it should do, and we won’t have a say in the matter.
Depends on what would satisfy us, I suppose.
I mean, for example, if it turns out that implementing CEV creates a future that everyone living in desires and is made happy and fulfilled and satisfied by and continues to do so indefinitely, and that everyone living now would if informed of the details of also desire and etc., but we are never able to confirm that any of that is right… or worse yet, later philosophical analysis somehow reveals that it isn’t right, despite being desirable and fulfilling and satisfying and so forth… well, OK, we can decide at that time whether we want to give up what is desirable and etc. in exchange for what is right, but in the meantime I might well be satisfied by that result. Maybe it’s OK to leave future generations some important tasks to implement.
Or, if it turns out that EY’s approach is all wrong because nobody agrees on anything important to anyone, so that extrapolating humanity’s coherent volition leaves out everything that’s important to everyone, so that implementing it doesn’t do anything important… in that case, coming up with an alternate plan that has results as above would satisfy me.
Etc.
It might turn out that what does satisfy us is to be “free”, to do what we want, even if that means that we will mess up our own future. It might turn out that humans are only satisfied if they can work on existential problems, “no risk no fun”. Or we might simply want to learn about the nature of reality. The mere existence of an FAI might spoil all of it. Would you care to do science if there was some AI-God that already knew all the answers? Would you be satisfied if it didn’t tell you the answers or made you forget that it does exist so that you’d try to invent AGI without ever succeeding?
But there is another possible end. Even today many people are really bored and don’t particularly enjoy life. What if it turns out that there is no “right” out there or that it can be reached fairly easily without any way to maximize it further. In other words, what if fun is something that isn’t infinite but a goal that can be reached? What if it all turns out to be wireheading, the only difference between 10 minutes of wireheading or 10^1000 years being the number enumerating the elapsed time? Think about it, would you care about 10^1000 years of inaction? What would you do if that was the optimum? Maybe we’ll just decide to choose the void instead.
This is a different context for satisfaction, but to answer your questions:
yes, I often find satisfying working through problems that have already been solved, though I appreciate that not everyone does;
no, I would not want to be denied the solutions if I asked (assuming there isn’t some other reason why giving me the solution is harmful), or kept in ignorance of the existence of those solutions (ibid);
if it turns out that all of my desires as they currently exist are fully implemented, leaving me with no room for progress and no future prospects better than endless joy, fulfillment and satisfaction, I’d be satisfied and fulfilled and joyful.
Admittedly, I might eventually become unsatisfied with that and desire something else, at which point I would devote efforts to satisfying that new desire. It doesn’t seem terribly likely that my non-existence would be the best possible way of doing so, but I suppose it’s possible, and if it happened I would cease to exist.
It might indeed.
No, once ostensibly-Friendly AI has run CEV and knows what it wants, it won’t matter if we eventually realize that CEV was wrong after all. The OFAI will go on to do what CEV says it should do, and we won’t have a say in the matter.
Agreed: avoiding irreversible steps is desirable.