I wonder what a CEV-implementing AI would do with such cases.
Even if it does turn out that my current conception of personal identity isn’t the same as my old one, but is rather I similar concept I adopted after realizing my values were incoherent, the AI might still find that the CEVs of my past and present selves concur. This is because, if I truly did adopt a new concept of identity because of it’s similarity to my old one, this suggests I possess some sort of meta-value that values taking my incoherent values and replacing them with coherent ones that are as similar as possible to the original. If this is the case the AI would extrapolate that meta-value and give me a nice new coherent sense of personal identity, like the one I currently possess.
Of course, if I am right and my current conception of personal identity is based on my simply figuring out what I meant all along by “identity,” then the AI would just extrapolate that.
This is because, if I truly did adopt a new concept of identity because of it’s similarity to my old one, this suggests I possess some sort of meta-value that values taking my incoherent values and replacing them with coherent ones that are as similar as possible to the original. If this is the case the AI would extrapolate that meta-value and give me a nice new coherent sense of personal identity, like the one I currently possess.
Maybe, but I doubt whether “as similar as possible” is (or can be made) uniquely denoting in all specific cases. This might sink it.
Even if it does turn out that my current conception of personal identity isn’t the same as my old one, but is rather I similar concept I adopted after realizing my values were incoherent, the AI might still find that the CEVs of my past and present selves concur. This is because, if I truly did adopt a new concept of identity because of it’s similarity to my old one, this suggests I possess some sort of meta-value that values taking my incoherent values and replacing them with coherent ones that are as similar as possible to the original. If this is the case the AI would extrapolate that meta-value and give me a nice new coherent sense of personal identity, like the one I currently possess.
Of course, if I am right and my current conception of personal identity is based on my simply figuring out what I meant all along by “identity,” then the AI would just extrapolate that.
Maybe, but I doubt whether “as similar as possible” is (or can be made) uniquely denoting in all specific cases. This might sink it.