I guess the ability to find oneself in the environment depends on no strange things happening in the environments you care about (which are “probable”), so that you can simply pattern-match the things that qualify as you, starting with an extremely simple idea of what “you” is, in some sense inbuilt is our minds by evolution. But in general, if there are tons of almost-you running around, you need the exact specification to figure out which of them you actually control.
This is basically the same idea I want to use to automatically extract human preference, even though strictly speaking one already needs full preference to recognize its instance.
I guess the ability to find oneself in the environment depends on no strange things happening in the environments you care about (which are “probable”), so that you can simply pattern-match the things that qualify as you, starting with an extremely simple idea of what “you” is, in some sense inbuilt is our minds by evolution. But in general, if there are tons of almost-you running around, you need the exact specification to figure out which of them you actually control.
This is basically the same idea I want to use to automatically extract human preference, even though strictly speaking one already needs full preference to recognize its instance.