Has anyone attempted to prove the statement “Consciousness of a Turing machine is undecideable”? The proof (if it’s true) might look a lot like the proof that the halting problem is undecideable.
Your conjecture seems to follow from Rice’s theorem, assuming the personhood of a running computation is a property of the partial function its algorithm computes. Also, I think you can prove your conjecture by taking a certain proof that the Halting Problem is undecidable and replacing ‘halts’ with ‘is conscious’. I can track this down if you’re still interested.
But this doesn’t mess up Eliezer’s plans at all: you can have “nonhalting predicates” that output “doesn’t halt” or “I don’t know”, analogous to the nonperson predicates proposed here.
Fair enough, as long as you’re not presupposing that our value systems—which are probably better than “minimize pain”—are unlikely to have strong anti-torture preferences.
As for the other two points: you might have already argued for them somewhere else, but if not, feel free to say more here. It’s at least obvious that anti-em-torture is harder to enforce, but are you thinking it’s also probably too hard to even know whether a computation creates a person being tortured? Or that our notion of torture is probably confused with respect to ems (and possibly with respect to us animals too)?