If we are to consider the agents embedded in our world (i.e. not external to it. i.e. not doing theology here), then it is the case that there is a multitude of engineering constraints. For instance the agent has to be a distributed system, with bandwidth and lag constraints (the bandwidth grows as surface area but number of operations grows as volume; and the speed of light limit, at very least, is very difficult to break). It is not unlike the mankind itself, which has developed morality as the protocol of inter-node interaction, with a kick start by cultural and/or biological selection.
Furthermore, different goals clearly have different complexity (and thus different prior probability).
The orthogonality can be destroyed by any of the zillions issues outside consideration, which would favour some goals over the others, and no such issue is likely to restore orthogonality exactly back. Thus the orthogonality thesis is an enormous conjunction of propositions of form ‘x=false’, which were made without awareness of making them, by some process that loosely approximates valid reasoning, but misses an incredibly important part of properly listing the assumptions that have to be made for validity.
Ultimately, the orthogonality thesis is as silly as the proposition that it is as cheap to make a bicycle as a full sized airplane. (You can’t show that airplane is more expensive than a bicycle using someone’s detached symbolic system, in a broken reasoning system. You can’t show that they are equally priced either. The thesis that they cost the same is true in some perverted sense, in which it is also irrelevant)
If we are to consider the agents embedded in our world (i.e. not external to it. i.e. not doing theology here), then it is the case that there is a multitude of engineering constraints. For instance the agent has to be a distributed system, with bandwidth and lag constraints (the bandwidth grows as surface area but number of operations grows as volume; and the speed of light limit, at very least, is very difficult to break). It is not unlike the mankind itself, which has developed morality as the protocol of inter-node interaction, with a kick start by cultural and/or biological selection.
Furthermore, different goals clearly have different complexity (and thus different prior probability).
The orthogonality can be destroyed by any of the zillions issues outside consideration, which would favour some goals over the others, and no such issue is likely to restore orthogonality exactly back. Thus the orthogonality thesis is an enormous conjunction of propositions of form ‘x=false’, which were made without awareness of making them, by some process that loosely approximates valid reasoning, but misses an incredibly important part of properly listing the assumptions that have to be made for validity.
Ultimately, the orthogonality thesis is as silly as the proposition that it is as cheap to make a bicycle as a full sized airplane. (You can’t show that airplane is more expensive than a bicycle using someone’s detached symbolic system, in a broken reasoning system. You can’t show that they are equally priced either. The thesis that they cost the same is true in some perverted sense, in which it is also irrelevant)