Okay, I think my bright dilettante answer to this is the following: The key is what allows you to prove that the FHE is conscious. It is not, itself, the FHE’s consciousness, which is probably still silently running (although that can no longer be proven). Proof of consciousness and consciousness are different things, although they clearly are related, and something may or may not have proved it’s consciousness in the past before losing its ability to do so in the future.
I used the following thought experiment while thinking about this:
The thought experiment that occurs to me is simply looking at someone’s brain while they do something stereotypically consciousness-indicating! An outside observer watching a brain might say, “oh, that just looks like a wet, wobbly lump of meat, I can’t even remotely tell how it’s supposed to operate just by looking at it, why would I think it’s generating consciousness?” The analogue to FHE here would be a lack of knowledge about neuroscience & such.
The thought experiment that occurs to me is simply looking at someone’s brain while they do something stereotypically consciousness-indicating! An outside observer watching a brain might say, “oh, that just looks like a wet, wobbly lump of meat, I can’t even remotely tell how it’s supposed to operate just by looking at it, why would I think it’s generating consciousness?” The analogue to FHE here would be a lack of knowledge about neuroscience & such.