How can you possibly create an AI that reasons morally the way you want it to unless you can describe how that moral reasoning works?
People want stuff. I suspect there is no simple description of what people want. The AI can infer what people want from their behavior (using the aforementioned automated empathy), take the average, and that’s the AI’s utility function.
If there is no simple description of what people want, a bunch of people debating the structure of this non-simple thing on a web site isn’t going to give clarity on the issue.
ETA:
Then we can tell you what the right thing to do is, and even help bring your feelings into alignment with that truth—as you go on to help save the world rather than being filled with pointless existential angst that the universe is made of math.
Hoping to change people’s feelings as part of an FAI implementation is steering toward failure. You’ll have to make the FAI based on the assumption that the vast majority of people won’t be persuaded by anything you say, unless you’ve had a lot more success persuading people than I have.
a bunch of people debating the structure of this non-simple thing on a web site
Downvoted for unnecessary status manoeuvring against the rest of LessWrong. Why should the location of discussion affect its value? Especially since the issue isn’t even one where people need to be motivated to act, but simply one that requires clear-headed thought.
Why should the location of discussion affect its value?
Because the anonymity of the internet causes discussions to derail in aggressive posturing as many social restraints are absent. Also because much communication is non verbal. Also because the internet presents a low barrier for entry into the conversation.
Mostly, a communication has value separate from where it is posted (although the message is not independent from the messenger, e.g. with the advent of the internet scholarly articles often influence their field while being read by relevant people in the editing stages by peers and go unread in their final draft form) but all else equal, knowing where a conversation is taking place helps one guess at its value. So you are mostly right.
Recently, I heard a novel anti-singularity argument. That ”...we have never witnessed a greater intelligence, therefore we have no evidence that one’s existence is possible.”. Not that intelligence isn’t very useful (a common but weak argument), but that one can’t extrapolate beyond the smartest human ever and believe it likely that a slightly greater level of intelligence is possible. Talk about low barriers to entry into the conversation! This community is fortunately good at policing itself.
Now if only I could find an example of unnecessary status manoevering ;-).
Hoping to change people’s feelings as part of an FAI implementation is steering toward failure.
I didn’t read this post as having direct implications for FAI convincing people of things. I think that for posts in which the FAI connection is tenuous, LW is best served by discussing rationality without it, so as to appeal to a wider audience.
I’m still intrigued by how the original post might be relevant for FAI in a way that I’m not seeing. Is there anything beyond, “here is how to shape the actions of an inquirer, P.S. an FAI could do it better than you can”? Because that postscript could go lots of places, and so pointing out it would fit here doesn’t tell me much.
I’m still intrigued by how the original post might be relevant for FAI in a way that I’m not seeing.
I didn’t quite understand what you said you were seeing, but I’ll try to describe the relevance.
The normal case is people talk about moral philosophy with a fairly relaxed emotional tone, from the point of view “it would be nice if people did such-and-such, they usually don’t, nobody’s listening to us, and therefore this conversation doesn’t matter much”. If you’re thinking of making an FAI, the emotional tone is different because the point of view is “we’re going to implement this, and we have to get it right because if it’s wrong the AI will go nuts and we’re all going to DIE!!!” But then you try to sound nice and calm anyway because accurately reflecting the underlying emotions doesn’t help, not to mention being low-status.
I think most talk about morality on this website is from the more tense point of view above. Otherwise, I wouldn’t bother with it, and I think many of the other people here wouldn’t either. A minority might think it’s an armchair philosophy sort of thing.
The problem with these discussions is that you have to know the design of the FAI is correct, so that design has to be as simple as possible. If we come up with some detailed understanding of human morality and program it into the FAI, that’s no good—we’ll never know it’s right. So IMO you need to delegate the work of forming a model of what people want to the FAI and focus on how to get the FAI to correctly build that model, which is simpler.
However, if lukeprog has some simple insight, it might be useful in this context. I’m expectantly waiting for his next post on this issue.
The part that got my attention was: “You’ll have to make the FAI based on the assumption that the vast majority of people won’t be persuaded by anything you say.”
Some people will be persuaded, and some won’t be, and the AI has to be able to tell them apart reliably regardless, so I don’t see assumptions about majorities coming into play, instead they seem like an unnecessary complication once you grant the AI a certain amount of insight into individuals that is assumed as the basis for the AI being relevant.
I.e., if it (we) has (have) to make assumptions for lack of understanding about individuals, the game is up anyway. So we still approach the issue from the standpoint of individuals (such as us) influencing other individuals, because an FAI doesn’t need separate group parameters, and because it doesn’t, it isn’t an obviously relevantly different scenario than anything else we can do and it can theoretically do better.
People want stuff. I suspect there is no simple description of what people want. The AI can infer what people want from their behavior (using the aforementioned automated empathy), take the average, and that’s the AI’s utility function.
If there is no simple description of what people want, a bunch of people debating the structure of this non-simple thing on a web site isn’t going to give clarity on the issue.
ETA:
Hoping to change people’s feelings as part of an FAI implementation is steering toward failure. You’ll have to make the FAI based on the assumption that the vast majority of people won’t be persuaded by anything you say, unless you’ve had a lot more success persuading people than I have.
Downvoted for unnecessary status manoeuvring against the rest of LessWrong. Why should the location of discussion affect its value? Especially since the issue isn’t even one where people need to be motivated to act, but simply one that requires clear-headed thought.
Because the anonymity of the internet causes discussions to derail in aggressive posturing as many social restraints are absent. Also because much communication is non verbal. Also because the internet presents a low barrier for entry into the conversation.
Mostly, a communication has value separate from where it is posted (although the message is not independent from the messenger, e.g. with the advent of the internet scholarly articles often influence their field while being read by relevant people in the editing stages by peers and go unread in their final draft form) but all else equal, knowing where a conversation is taking place helps one guess at its value. So you are mostly right.
Recently, I heard a novel anti-singularity argument. That ”...we have never witnessed a greater intelligence, therefore we have no evidence that one’s existence is possible.”. Not that intelligence isn’t very useful (a common but weak argument), but that one can’t extrapolate beyond the smartest human ever and believe it likely that a slightly greater level of intelligence is possible. Talk about low barriers to entry into the conversation! This community is fortunately good at policing itself.
Now if only I could find an example of unnecessary status manoevering ;-).
I didn’t read this post as having direct implications for FAI convincing people of things. I think that for posts in which the FAI connection is tenuous, LW is best served by discussing rationality without it, so as to appeal to a wider audience.
I’m still intrigued by how the original post might be relevant for FAI in a way that I’m not seeing. Is there anything beyond, “here is how to shape the actions of an inquirer, P.S. an FAI could do it better than you can”? Because that postscript could go lots of places, and so pointing out it would fit here doesn’t tell me much.
I didn’t quite understand what you said you were seeing, but I’ll try to describe the relevance.
The normal case is people talk about moral philosophy with a fairly relaxed emotional tone, from the point of view “it would be nice if people did such-and-such, they usually don’t, nobody’s listening to us, and therefore this conversation doesn’t matter much”. If you’re thinking of making an FAI, the emotional tone is different because the point of view is “we’re going to implement this, and we have to get it right because if it’s wrong the AI will go nuts and we’re all going to DIE!!!” But then you try to sound nice and calm anyway because accurately reflecting the underlying emotions doesn’t help, not to mention being low-status.
I think most talk about morality on this website is from the more tense point of view above. Otherwise, I wouldn’t bother with it, and I think many of the other people here wouldn’t either. A minority might think it’s an armchair philosophy sort of thing.
The problem with these discussions is that you have to know the design of the FAI is correct, so that design has to be as simple as possible. If we come up with some detailed understanding of human morality and program it into the FAI, that’s no good—we’ll never know it’s right. So IMO you need to delegate the work of forming a model of what people want to the FAI and focus on how to get the FAI to correctly build that model, which is simpler.
However, if lukeprog has some simple insight, it might be useful in this context. I’m expectantly waiting for his next post on this issue.
The part that got my attention was: “You’ll have to make the FAI based on the assumption that the vast majority of people won’t be persuaded by anything you say.”
Some people will be persuaded, and some won’t be, and the AI has to be able to tell them apart reliably regardless, so I don’t see assumptions about majorities coming into play, instead they seem like an unnecessary complication once you grant the AI a certain amount of insight into individuals that is assumed as the basis for the AI being relevant.
I.e., if it (we) has (have) to make assumptions for lack of understanding about individuals, the game is up anyway. So we still approach the issue from the standpoint of individuals (such as us) influencing other individuals, because an FAI doesn’t need separate group parameters, and because it doesn’t, it isn’t an obviously relevantly different scenario than anything else we can do and it can theoretically do better.