It is not clear to me that consciousness automatically confers moral standing. Consider the well-known conundrum of The Pig That Wants To Be Eaten. By hypothesis it is conscious, and wants to become a tasty meal. (“Don’t worry, I’ll shoot myself quite humanely.”) Is it wrong to use it for that? Wrong to have created it? Would you rather eat a pig that didn’t want to be eaten?
If we make a conscious machine whose fulfilment lies only in serving us, which does not care how it is treated, and which has no objection to being turned off and discarded when it is obsolete, what moral standing would it have?
I’m not familiar with the canonical details of house elves in Harry Potter (whether Rowling’s or Eliezer’s version), but if they were magically created to be a race of servants who desire no other life, was it bad to have created them? Is it bad to give them the life of servitude they desire?
And to come back to current reality, the last question might be asked of some actual BDSM relationships.
I agree with this. If we are able to design consciousness such that a system is fulfilled by serving humans, then it’s possible that would be morally alright. I don’t think there is a strong enough consensus that I’d feel comfortable locking it in, but to me it seems ok.
By default though, I think we won’t be designing consciousness intentionally, and it will just emerge, and I don’t think that’s too likely to lead to this sort of situation.
It is not clear to me that consciousness automatically confers moral standing. Consider the well-known conundrum of The Pig That Wants To Be Eaten. By hypothesis it is conscious, and wants to become a tasty meal. (“Don’t worry, I’ll shoot myself quite humanely.”) Is it wrong to use it for that? Wrong to have created it? Would you rather eat a pig that didn’t want to be eaten?
If we make a conscious machine whose fulfilment lies only in serving us, which does not care how it is treated, and which has no objection to being turned off and discarded when it is obsolete, what moral standing would it have?
I’m not familiar with the canonical details of house elves in Harry Potter (whether Rowling’s or Eliezer’s version), but if they were magically created to be a race of servants who desire no other life, was it bad to have created them? Is it bad to give them the life of servitude they desire?
And to come back to current reality, the last question might be asked of some actual BDSM relationships.
I agree with this. If we are able to design consciousness such that a system is fulfilled by serving humans, then it’s possible that would be morally alright. I don’t think there is a strong enough consensus that I’d feel comfortable locking it in, but to me it seems ok.
By default though, I think we won’t be designing consciousness intentionally, and it will just emerge, and I don’t think that’s too likely to lead to this sort of situation.