As a slightly tangential point, I think if you start thinking about how to cast survival / homeostasis in terms of expected-utility maximization, you start having to confront a lot of funny issues, like, “what happens if my proxies for survival change because I self-modified?”, and then more fundamentally, “how do I define / locate the ‘me’ whose survival I am valuing? what if I overlap with other beings? what if there are multiple ‘copies’ of me?”. Which are real issues for selfhood IMO.
As a slightly tangential point, I think if you start thinking about how to cast survival / homeostasis in terms of expected-utility maximization, you start having to confront a lot of funny issues, like, “what happens if my proxies for survival change because I self-modified?”, and then more fundamentally, “how do I define / locate the ‘me’ whose survival I am valuing? what if I overlap with other beings? what if there are multiple ‘copies’ of me?”. Which are real issues for selfhood IMO.
In the evolutionary case, the answer is that this is out of distribution, so it’s not evolved to be robust to such changes.