One possible attack: take the moral code and add “Notwithstanding all this, kill the humans.” to the end. This should be superior for all the remaining agents, since humans won’t be using up any resources after that’s accomplished (assuming we can’t put up enough of a fight).
A practical vulnerability to (perhaps unconsciously biased) self-interested gamers: untestable claims that although the moral code is at a local optimum, everyone needs to switch to a far away alternative, and then, after the new equilibrium, things will really be better. Sci-fi rejoinder: this explains why there are so many simulations :)
Yes, in a society with both human and non-human agents, if the humans contribute nothing at all to the non-humans, and only consume resources that might have been otherwise used, then the non-humans will judge the humans to be worthless. Worse than worthless. A drain to be eliminated.
But there is nothing special about my version of ethics in this regard. It is a problem that must be faced in any system of ethics. It is the FAI problem. Eliezer’s solution is apparently to tell the non-human agents as they are created “Humans are to be valued. And don’t you forget it when you self-modify.” I think that a better approach is to make sure that the humans actually contribute something tangible to the well-being of the non-humans.
Perhaps neither approach is totally safe in the long run.
Interesting.
One possible attack: take the moral code and add “Notwithstanding all this, kill the humans.” to the end. This should be superior for all the remaining agents, since humans won’t be using up any resources after that’s accomplished (assuming we can’t put up enough of a fight).
A practical vulnerability to (perhaps unconsciously biased) self-interested gamers: untestable claims that although the moral code is at a local optimum, everyone needs to switch to a far away alternative, and then, after the new equilibrium, things will really be better. Sci-fi rejoinder: this explains why there are so many simulations :)
Yes, in a society with both human and non-human agents, if the humans contribute nothing at all to the non-humans, and only consume resources that might have been otherwise used, then the non-humans will judge the humans to be worthless. Worse than worthless. A drain to be eliminated.
But there is nothing special about my version of ethics in this regard. It is a problem that must be faced in any system of ethics. It is the FAI problem. Eliezer’s solution is apparently to tell the non-human agents as they are created “Humans are to be valued. And don’t you forget it when you self-modify.” I think that a better approach is to make sure that the humans actually contribute something tangible to the well-being of the non-humans.
Perhaps neither approach is totally safe in the long run.