Yeah, at some point we’ll need a proper theory of consciousness regardless, since many humans will want to radically self-improve and it’s important to know which cognitive enhancements preserve consciousness.
Yeah. My point was, we can’t even be sure which behavior-preserving optimizations (of the kind done by optimizing compilers, say) will preserve consciousness. It’s worrying because these optimizations can happen innocuously, e.g. when your upload gets migrated to a newer CPU with fancier heuristics. And yeah, when self-modification comes into the picture, it gets even worse.
Yeah, at some point we’ll need a proper theory of consciousness regardless, since many humans will want to radically self-improve and it’s important to know which cognitive enhancements preserve consciousness.
Yeah. My point was, we can’t even be sure which behavior-preserving optimizations (of the kind done by optimizing compilers, say) will preserve consciousness. It’s worrying because these optimizations can happen innocuously, e.g. when your upload gets migrated to a newer CPU with fancier heuristics. And yeah, when self-modification comes into the picture, it gets even worse.