Which is where I think politics offers a pretty strong hint to the possibility that the Friendliness Problem has no resolution:
We can’t agree on which political formations are more Friendly. That’s what “Politics is the Mindkiller” is all about; our inability to come to an agreement on political matters. It’s not merely a matter of the rules—which is to say, it’s not a matter of the output: We can’t even come to an agreement about which values should be used to form the rules.
I’m pretty sure this is a problem with human reasoning abilities, and not a problem with friendliness itself. Or in other words, I think this is only very weak evidence that friendliness is unresolvable.
Indeed. If we were perfect bayesians, who had unlimited introspective access, and we STILL couldn’t agree after an unconscionable amount of argument and discussion, then we’d have a bigger problem.
My point exactly. Only if we are sure agents are best representing themselves, can we be sure their values are not the same. If an agent is unsure of zir values, or extrapolates them incorrectly, then there will be disagreement that doesn’t imply different values.
With seven billion people, none of which are best representing themselves (they certainly aren’t perfect bayesians!) then we should expect massive disagreement. This is not an argument for fundamentally different values.
I disagree with the first statement, but agree with the second. That is, I disagree with a certainty that the problem is with our reasoning abilities, but agree that the evidence is very weak.
I’m pretty sure this is a problem with human reasoning abilities, and not a problem with friendliness itself. Or in other words, I think this is only very weak evidence that friendliness is unresolvable.
Indeed. If we were perfect bayesians, who had unlimited introspective access, and we STILL couldn’t agree after an unconscionable amount of argument and discussion, then we’d have a bigger problem.
Are perfect Bayesians with unlimited introspective access more inclined to agree on matters of first principles?
I’m not sure. I’ve never met one, much less two.
yes
They will agree on what values they have, and what the best action is relative to those values, but they still might have different values.
My point exactly. Only if we are sure agents are best representing themselves, can we be sure their values are not the same. If an agent is unsure of zir values, or extrapolates them incorrectly, then there will be disagreement that doesn’t imply different values.
With seven billion people, none of which are best representing themselves (they certainly aren’t perfect bayesians!) then we should expect massive disagreement. This is not an argument for fundamentally different values.
I disagree with the first statement, but agree with the second. That is, I disagree with a certainty that the problem is with our reasoning abilities, but agree that the evidence is very weak.
Um, I said I was “pretty sure”. Not absolutely certain.
Upvoted, and I’ll consider it fair if you downvote my reply. Sorry about that!
No worries!
I’m amused that you’ve retracted the post in question after posting this.