We don’t live in a universe that’s nice or just all the time, so perhaps there are nightmare scenarios in our future. Not all traps have an escape. However, I think this one does, for two reasons.
(1) all the reasons that RobinHanson mentioned;
(2) we seem really confused about how consciousness works, which suggests there are large ‘unknown unknowns’ in play. It seems very likely that if we extrapolate our confused models of consciousness into extreme scenarios such as this, we’ll get even more confused results.
Addressing 2...this argument is compelling, I read it to be equivalent to the statement that ‘humqn ethics do not apply to ems, or human behavior regarding ems’, so acting from the standpoint of ‘ems are not human, therefore human ethics do not apply, and em suffering is not human suffering, so effective altruism does not apply to ems’ is a way out of the trap.
Taking it to its’ conclusion, we can view Ems as vampires (consume resources, produce no human children, are not alive but also not dead), and like all such abominations must be destroyed to preserve the lives and futures of humans!
We don’t live in a universe that’s nice or just all the time, so perhaps there are nightmare scenarios in our future. Not all traps have an escape. However, I think this one does, for two reasons.
(1) all the reasons that RobinHanson mentioned;
(2) we seem really confused about how consciousness works, which suggests there are large ‘unknown unknowns’ in play. It seems very likely that if we extrapolate our confused models of consciousness into extreme scenarios such as this, we’ll get even more confused results.
Addressing 2...this argument is compelling, I read it to be equivalent to the statement that ‘humqn ethics do not apply to ems, or human behavior regarding ems’, so acting from the standpoint of ‘ems are not human, therefore human ethics do not apply, and em suffering is not human suffering, so effective altruism does not apply to ems’ is a way out of the trap.
Taking it to its’ conclusion, we can view Ems as vampires (consume resources, produce no human children, are not alive but also not dead), and like all such abominations must be destroyed to preserve the lives and futures of humans!
Notice that if you need to plunge something through a live circuit board, a wooden stake is much better than a metal weapon.