The preference for “objective meaningfulness”—for states which do not compress very well—seems to me a fairly arbitrary (meaningless) preference. I don’t think it’s much different from paperclip maximization.
Who is to observe the “meaningful” states, if everyone is in a state where they are happy?
I am not even convinced that “happy and fulfilled” compresses easily. But if it did, what is the issue? Everyone will be so happy as to not mind the absence of complicated states.
I would go so far as to say that seeking complicated states is something we do right now because it is the most engaging substitute we have for being happy.
And not everyone does this. Most people prefer to empty their minds instead. It may even be that seeking complexity is a type of neurotic tendency.
Should the FAI be designed with a neurotic tendency?
The preference for “objective meaningfulness”—for states which do not compress very well—seems to me a fairly arbitrary (meaningless) preference. I don’t think it’s much different from paperclip maximization.
Who is to observe the “meaningful” states, if everyone is in a state where they are happy?
I am not even convinced that “happy and fulfilled” compresses easily. But if it did, what is the issue? Everyone will be so happy as to not mind the absence of complicated states.
I would go so far as to say that seeking complicated states is something we do right now because it is the most engaging substitute we have for being happy.
And not everyone does this. Most people prefer to empty their minds instead. It may even be that seeking complexity is a type of neurotic tendency.
Should the FAI be designed with a neurotic tendency?
I’m not so sure.