If so, then that’s a pretty straightforward point that any FAI would respect
I’m merely suspicious of that, since no one still know what an FAI really looks like. You seem to tend to collate an FAI with agreeable, pleasant or fun super-singleton, but these are your heuristics. It’s obvious that a friendly AI will respect my wishes, but it’s not obvious that an AI possibly developed by the people in this forum will.
(Even though I tend to think it’s mistaken, or at least, awfully inefficient.)
And that’s exactly the reason why I made the post.
I’m merely suspicious of that, since no one still know what an FAI really looks like. You seem to tend to collate an FAI with agreeable, pleasant or fun super-singleton, but these are your heuristics. It’s obvious that a friendly AI will respect my wishes, but it’s not obvious that an AI possibly developed by the people in this forum will.
A FAI will respect your preferences (as far as it can). That’s the whole point of Friendliness. Of course, an UFAI might just ignore you. But then, asking it to spare you probably won’t help you, either. It would be an awful coincidence that we got an AI that will try to force people into orgasmium, but allow them to opt-out.
That was not my point. I was not appealing to some abstract “friendly AI in the sky” as a new kind of rationalist god.
My post was meant to address the people in the forum, a sort of warning that shouted: “Hey, lesswrongians! I saw you are in love with complexity and so on, but don’t forget that people might have radically different preferences. Indeed, these are mine. Don’t forget to include them in a friendly AI, if you happen to build one”.
Baseline: I think the fun theory that circulates the most in this forum, as per Yudkowsky sequence, is dangerously narrow. That could have potentially catastrophic consequences, and I wanted to prevent that.
The most commonly discussed FAI won’t have any (human) terminal values built in (except maybe for bootstrapping), but will examine humans to get a complete understanding of what humans actually value and then optimize that.
Therefore, whatever we humans now think about what might be fun or not is fairly irrelevant for the FAI simply because we don’t know what our values are. (That is also why I referenced the confusion between terminal and instrumental values in my first comment. I think that pretty much everyone talking about their “values” is only talking about instrumental values and that properly reduced terminal values are much, much simpler.)
I’m merely suspicious of that, since no one still know what an FAI really looks like. You seem to tend to collate an FAI with agreeable, pleasant or fun super-singleton, but these are your heuristics. It’s obvious that a friendly AI will respect my wishes, but it’s not obvious that an AI possibly developed by the people in this forum will.
And that’s exactly the reason why I made the post.
A FAI will respect your preferences (as far as it can). That’s the whole point of Friendliness. Of course, an UFAI might just ignore you. But then, asking it to spare you probably won’t help you, either. It would be an awful coincidence that we got an AI that will try to force people into orgasmium, but allow them to opt-out.
That was not my point. I was not appealing to some abstract “friendly AI in the sky” as a new kind of rationalist god.
My post was meant to address the people in the forum, a sort of warning that shouted: “Hey, lesswrongians! I saw you are in love with complexity and so on, but don’t forget that people might have radically different preferences. Indeed, these are mine. Don’t forget to include them in a friendly AI, if you happen to build one”.
Baseline: I think the fun theory that circulates the most in this forum, as per Yudkowsky sequence, is dangerously narrow. That could have potentially catastrophic consequences, and I wanted to prevent that.
The most commonly discussed FAI won’t have any (human) terminal values built in (except maybe for bootstrapping), but will examine humans to get a complete understanding of what humans actually value and then optimize that.
Therefore, whatever we humans now think about what might be fun or not is fairly irrelevant for the FAI simply because we don’t know what our values are. (That is also why I referenced the confusion between terminal and instrumental values in my first comment. I think that pretty much everyone talking about their “values” is only talking about instrumental values and that properly reduced terminal values are much, much simpler.)