It seems to be an attempt to imagine what a FAI optimizing the world to given person’s values will do, and these are destined to fail
Yes, I think that this is right. An FAI would try to create a world in which we are all better off. That doesn’t mean that the world would be as any one of us considers perfect. Perhaps each of us would still consider it to be very suboptimal. But all of us would still desire it over the present world.
In other words, let’s grant, for the sake of argument, that most of us are doomed to continue to suffer from losing zero-sum status games forever. Nonetheless, there are possible worlds within which we all, even the losers, would much rather play these games. So there is still a lot that an FAI could do for us.
You can’t think about what specifically FAI will do, period. It seems quite likely there will be no recognizable humans in a world rebuilt by FAI. Any assumption is suspect, even the ones following from the most reliable moral heuristics.
Is it correct to call it FAI, then? Do you see a world with “no recognizable humans” as a very likely thing for the human race (or its extrapolated volition) to collectively want?
I’m considering the case of FAI, that is humanity’s preference correctly rendered.
Do you see a world with “no recognizable humans” as a very likely thing for the human race (or its extrapolated volition) to collectively want?
Status quo has no power. So the question shouldn’t be whether “no recognizable humans” is the particular thing humanity wants, but rather whether “preserving recognizable humans” happens to be the particular thing that humanity wants. And I’m not sure there are strong enough reasons to expect “world with recognizable humans” to be the optimal thing to do with the matter. It might be, but I’m not convinced we know enough to locate this particular hypothesis. The default assumption that humans want humans seems to stem from the cached moral intuition promoted by availability in the current situation, but reconstructing the optimal situation from preference is a very indirect process, that won’t respect the historical accidents of natural development of humanity, only humanity’s values.
You can’t think about what specifically FAI will do, period.
“Specifically” is relative. By some standards, we have never thought specifically about anything at all. (I have never traced precisely the path of every atom involved in any action.)
Nonetheless, one can think more or less specifically, and to think at all is to think a thought that is specific to some extent. To think, as you wrote above, that an “FAI is a system expected to do something good” is to think something more specific than one might, if one were committed to thinking nothing specific, period. (This is assuming that your words have any meaning whatsoever.)
ETA: In other words, as Eliezer wrote in his Coming of Age sequence, you must be thinking something that is specific to some extent, for otherwise you couldn’t even pose the problem of FAI to yourself.
Sure. The specific thing you say is that the outcome is “good”, but what that means exactly is very hard to decipher, and in particular hard or impossible to decipher in a form of a story, with people, their experiences and social constructions. It is the story that can’t be specific.
[ETA: I wrote the following when your comment read simply “Sure, why?”. I can see the plausibility of your claim that narrative moral imaginings can contribute nothing to the development of FAI, though it’s not self-evidently obvious to me. ]
Perhaps I missed the point of your previous comment.
I presumed that you thought that I was being too specific. I read you as expressing this thought by saying that one should not think specifically, “period”. I was pointing out the impossibility or meaninglessness of that injunction, at least in its extreme form. I was implicitly encouraging you to indicate the non-extreme meaning that you had intended.
Yes, I think that this is right. An FAI would try to create a world in which we are all better off. That doesn’t mean that the world would be as any one of us considers perfect. Perhaps each of us would still consider it to be very suboptimal. But all of us would still desire it over the present world.
In other words, let’s grant, for the sake of argument, that most of us are doomed to continue to suffer from losing zero-sum status games forever. Nonetheless, there are possible worlds within which we all, even the losers, would much rather play these games. So there is still a lot that an FAI could do for us.
You can’t think about what specifically FAI will do, period. It seems quite likely there will be no recognizable humans in a world rebuilt by FAI. Any assumption is suspect, even the ones following from the most reliable moral heuristics.
Is it correct to call it FAI, then? Do you see a world with “no recognizable humans” as a very likely thing for the human race (or its extrapolated volition) to collectively want?
I’m considering the case of FAI, that is humanity’s preference correctly rendered.
Status quo has no power. So the question shouldn’t be whether “no recognizable humans” is the particular thing humanity wants, but rather whether “preserving recognizable humans” happens to be the particular thing that humanity wants. And I’m not sure there are strong enough reasons to expect “world with recognizable humans” to be the optimal thing to do with the matter. It might be, but I’m not convinced we know enough to locate this particular hypothesis. The default assumption that humans want humans seems to stem from the cached moral intuition promoted by availability in the current situation, but reconstructing the optimal situation from preference is a very indirect process, that won’t respect the historical accidents of natural development of humanity, only humanity’s values.
“Specifically” is relative. By some standards, we have never thought specifically about anything at all. (I have never traced precisely the path of every atom involved in any action.)
Nonetheless, one can think more or less specifically, and to think at all is to think a thought that is specific to some extent. To think, as you wrote above, that an “FAI is a system expected to do something good” is to think something more specific than one might, if one were committed to thinking nothing specific, period. (This is assuming that your words have any meaning whatsoever.)
ETA: In other words, as Eliezer wrote in his Coming of Age sequence, you must be thinking something that is specific to some extent, for otherwise you couldn’t even pose the problem of FAI to yourself.
Sure. The specific thing you say is that the outcome is “good”, but what that means exactly is very hard to decipher, and in particular hard or impossible to decipher in a form of a story, with people, their experiences and social constructions. It is the story that can’t be specific.
[ETA: I wrote the following when your comment read simply “Sure, why?”. I can see the plausibility of your claim that narrative moral imaginings can contribute nothing to the development of FAI, though it’s not self-evidently obvious to me. ]
Perhaps I missed the point of your previous comment.
I presumed that you thought that I was being too specific. I read you as expressing this thought by saying that one should not think specifically, “period”. I was pointing out the impossibility or meaninglessness of that injunction, at least in its extreme form. I was implicitly encouraging you to indicate the non-extreme meaning that you had intended.