I agree. I’ve been there many times with many devices. But in the toaster example, I think that will be because it thought it knew what you wanted it to do, and was wrong. I’d be thinking the same if, say, I wanted extra-dark toast to make croutons with and it didn’t do it. If what actually happens is that you switch varieties of bread and forget to account for that, or don’t realize someone else used the toaster in the interim and moved the dial, then “I would much rather you burned my toast than disobey me” is not, I think, how most people would react.
I agree. I’ve been there many times with many devices. But in the toaster example, I think that will be because it thought it knew what you wanted it to do, and was wrong. I’d be thinking the same if, say, I wanted extra-dark toast to make croutons with and it didn’t do it. If what actually happens is that you switch varieties of bread and forget to account for that, or don’t realize someone else used the toaster in the interim and moved the dial, then “I would much rather you burned my toast than disobey me” is not, I think, how most people would react.
However, that is my reaction.
In some circumstances I may tolerate a device providing a warning, but if I tell it twice, I expect it to STFU and follow orders.
I agree. I already have enough non-AI systems in my life3 that fail this test, and I definitely don’t want more.
I wonder when we will first see someone go on trial for bullying a toaster.
ETA: In the Eliezer fic, maybe the penalty would be being cancelled by all the AIs.