Bad feelings are vastly less important than saved lives.
Omega: Hey, supposedlyfun. You are going to die in a car crash tomorrow. Your child will grow up without you, your family will never get over it, and no aligned AGI will recreate your mind once technology allows it. But! I can prevent that death if you let me torture a random person for a year, inflicting on them the maximum possible amount of pain that their nervous system can experience, at every interval of Planck time during the year. But I will then give that person instant therapy that undoes all the damage. What say you?
supposedlyfun: No.
*
How do you square your argument with my preference here?
Omega: Hey, supposedlyfun. You are going to die in a car crash tomorrow. Your child will grow up without you, your family will never get over it, and no aligned AGI will recreate your mind once technology allows it. But! I can prevent that death if you let me torture a random person for a year, inflicting on them the maximum possible amount of pain that their nervous system can experience, at every interval of Planck time during the year. But I will then give that person instant therapy that undoes all the damage. What say you?
supposedlyfun: No.
*
How do you square your argument with my preference here?