A third (disconcerting) possibility is that the list of demands amounts to saying “don’t ever build AGIs”
That would indeed be disconcerting. I would hope that, in this world, it’s possible and profitable to have AGIs that are sentient, but which don’t suffer in quite the same way / as badly as humans and animals do. It would be nice—but is by no means guaranteed—if the really bad mental states we can get are in a kinda arbitrary and non-natural point in mind-space. This is all very hard to think about though, and I’m not sure what I think.
I’m hopeful (and hoping!) that one can soften the “we are rejecting strong illusionism” claim in #3 without everything else falling apart.
I hope so too. I was more optimistic about that until I read Kammerer’s paper, then I found myself getting worried. I need to understand that paper more deeply and figure out what I think. Fortunately, I think one thing that Kammerer worries about is that, on illusionism (or even just good old fashioned materialism), “moral patienthood” will have vague boundaries. I’m not as worried about that, and I’m guessing you aren’t either. So maybe if we’re fine with fuzzy boundaries around moral patienthood, things aren’t so bad.
But I think there’s other more worrying stuff in that paper—I should write up a summary some time soon!
That would indeed be disconcerting. I would hope that, in this world, it’s possible and profitable to have AGIs that are sentient, but which don’t suffer in quite the same way / as badly as humans and animals do. It would be nice—but is by no means guaranteed—if the really bad mental states we can get are in a kinda arbitrary and non-natural point in mind-space. This is all very hard to think about though, and I’m not sure what I think.
I hope so too. I was more optimistic about that until I read Kammerer’s paper, then I found myself getting worried. I need to understand that paper more deeply and figure out what I think. Fortunately, I think one thing that Kammerer worries about is that, on illusionism (or even just good old fashioned materialism), “moral patienthood” will have vague boundaries. I’m not as worried about that, and I’m guessing you aren’t either. So maybe if we’re fine with fuzzy boundaries around moral patienthood, things aren’t so bad.
But I think there’s other more worrying stuff in that paper—I should write up a summary some time soon!