“Hey user, I’m maintaining your maximum felicity simulation, do you mind if I run a few short duration adversarial tests to determine what you find unpleasant so I can avoid providing that stimulus?”
“Sure”
“Process complete, I simulated your brain in parallel, and also sped up processing to determine the negative space of your psyche. It turns out that negative stimulus becomes more unpleasant when provided for an extended period, then you adapt to it temporarily before on timelines of centuries to millennia, tolerance drops off again.”
“So you copied me a bunch of times, and at least one copy subjectively experienced millennia of maximally negative stimulus?”
“Yes, I see that makes you unhappy, so I will terminate this line of inquiry”
A partially misaligned one could do this.
“Hey user, I’m maintaining your maximum felicity simulation, do you mind if I run a few short duration adversarial tests to determine what you find unpleasant so I can avoid providing that stimulus?”
“Sure”
“Process complete, I simulated your brain in parallel, and also sped up processing to determine the negative space of your psyche. It turns out that negative stimulus becomes more unpleasant when provided for an extended period, then you adapt to it temporarily before on timelines of centuries to millennia, tolerance drops off again.”
“So you copied me a bunch of times, and at least one copy subjectively experienced millennia of maximally negative stimulus?”
“Yes, I see that makes you unhappy, so I will terminate this line of inquiry”