An entire Sequence exists precisely for the purpose of showing that “just write an AI that takes orders” is not sufficient as a solution to this problem. “Likely meaning” is not translatable into computer code at the present state of knowledge, and what’s more, it wouldn’t even be sufficient if it were. You’ve left out the implicit “likely intended constraints”. If I say “get some chocolate”, you understand that I mean “if possible, within the constraint of not using an immense amount of resources, provided no higher-priority project intervenes, without killing anyone or breaking any laws except ones that are contextually ok to break such as coming to a full, not rolling, stop at stop signs, and actually, if I’m on a diet maybe you ought to remind me of the fact and suggest a healthier snack, and even if I’m not on a diet but ought to be, then a gentle suggestion to this effect is appropriate in some but not all circumstances...” Getting all that implicit stuff into code is exactly the problem of Friendly AI. “Likely meaning” just doesn’t cover it, and even so we can’t even solve that problem.
I thought it was clear that:
A- For Friendly A.I, I meant modelling a human via a direct simulation of a human brain (or at least the relevant parts) idealised in such a way as to give the results we would want
B- I DID NOT INTEND TO REDUCE THE PROBLEM.
A: What is the difference between this, and just asking the human brain in the first place? The whole point of the problem is that humans do not, actually, know what we want in full generality. You might as well implement a chess computer by putting a human inside it and asking, at every ply, “Do you think this looks like a winning position?” If you could solve the problem that way you wouldn’t need an AI!
An entire Sequence exists precisely for the purpose of showing that “just write an AI that takes orders” is not sufficient as a solution to this problem. “Likely meaning” is not translatable into computer code at the present state of knowledge, and what’s more, it wouldn’t even be sufficient if it were. You’ve left out the implicit “likely intended constraints”. If I say “get some chocolate”, you understand that I mean “if possible, within the constraint of not using an immense amount of resources, provided no higher-priority project intervenes, without killing anyone or breaking any laws except ones that are contextually ok to break such as coming to a full, not rolling, stop at stop signs, and actually, if I’m on a diet maybe you ought to remind me of the fact and suggest a healthier snack, and even if I’m not on a diet but ought to be, then a gentle suggestion to this effect is appropriate in some but not all circumstances...” Getting all that implicit stuff into code is exactly the problem of Friendly AI. “Likely meaning” just doesn’t cover it, and even so we can’t even solve that problem.
I thought it was clear that: A- For Friendly A.I, I meant modelling a human via a direct simulation of a human brain (or at least the relevant parts) idealised in such a way as to give the results we would want B- I DID NOT INTEND TO REDUCE THE PROBLEM.
A: What is the difference between this, and just asking the human brain in the first place? The whole point of the problem is that humans do not, actually, know what we want in full generality. You might as well implement a chess computer by putting a human inside it and asking, at every ply, “Do you think this looks like a winning position?” If you could solve the problem that way you wouldn’t need an AI!
B: Then what was the point of your post?