very smart people have issues with CEV, example: http://lesswrong.com/lw/2b7/hacking_the_cev_for_fun_and_profit/
and as far as I remember CEV was sort of abandoned a while ago by the community.
and yes, you value humans, others in the not so distant future might not given the possibility of body/brain modification. anyway, the gist of my argument is that CEV doesn’t seem to work if there is not going to be much coherence of all of humanity’s extrapolated volition’s-a point that’s already been made clear in previous threads by many people-what I’m trying to add to that is to point out the overwhelming possibility of there being ‘alien minds’ among us before a FAI could be built.
I also raised the question that If body modification is widely available, is it ok to prevent people from acquiring an ‘alien’ set of morals, one that would later on be a possible hindrance to CEV-like proposals? how can we tell if its alien or not in the first place?
but isn’t being presented with a to-do list or alternatively feeling hungry then finding food different than ‘forming goals’?
to be more precise, maybe the ‘survival instinct’ that leads them to seek food is not located in their emotional centers so some goals might survive regardless. but yes, the assumption is untested AFAIK.