I do see selves, or personal identity, as closely related to goals or values. (Specifically, I think the concept of a self would have zero content if we removed everything based on preferences or values; roughly 100% of humans who’ve every thought about the nature of identity have said it’s more like a value statement than a physical fact.) However, I don’t think we can identify the two. Evolution is technically an optimization process, and yet has no discernible self. We have no reason to think it’s actually impossible for a ‘smarter’ optimization process to lack identity, and yet form instrumental goals such as preventing other AIs from hacking it in ways which would interfere with its ultimate goals. (The latter are sometimes called “terminal values.”)
I do see selves, or personal identity, as closely related to goals or values. (Specifically, I think the concept of a self would have zero content if we removed everything based on preferences or values; roughly 100% of humans who’ve every thought about the nature of identity have said it’s more like a value statement than a physical fact.) However, I don’t think we can identify the two. Evolution is technically an optimization process, and yet has no discernible self. We have no reason to think it’s actually impossible for a ‘smarter’ optimization process to lack identity, and yet form instrumental goals such as preventing other AIs from hacking it in ways which would interfere with its ultimate goals. (The latter are sometimes called “terminal values.”)