I agree. If our complex preferences can be represented as exact preferences then any drift from those exact preferences would be necessarily bad. However, it’s not clear to me that we actually would be drifting from our exact preference were we to follow the path of WBE.
It’s clear that the preferences we currently express most likely aren’t our exact preferences. The path of WBE could potentially lead to humans with fundamentally different exact preferences (bad), or it could simply lead to humans with the same exact preferences but with a different, closer expression of them in the surface preferences they actually present and are consciously aware of (good). Or the path could lead to someplace in between, obviously. Any drift is bad, I agree, but small enough drift could be acceptable if the trade off is good enough (such as preventing a negative singularity).
By the way, I move to label your definition “exact preference” and mine “complex preference”. Unless the context is clear, in which case we can just write “preference”. Thoughts?
I agree. If our complex preferences can be represented as exact preferences then any drift from those exact preferences would be necessarily bad. However, it’s not clear to me that we actually would be drifting from our exact preference were we to follow the path of WBE.
It’s clear that the preferences we currently express most likely aren’t our exact preferences. The path of WBE could potentially lead to humans with fundamentally different exact preferences (bad), or it could simply lead to humans with the same exact preferences but with a different, closer expression of them in the surface preferences they actually present and are consciously aware of (good). Or the path could lead to someplace in between, obviously. Any drift is bad, I agree, but small enough drift could be acceptable if the trade off is good enough (such as preventing a negative singularity).
By the way, I move to label your definition “exact preference” and mine “complex preference”. Unless the context is clear, in which case we can just write “preference”. Thoughts?