Thank you. I did follow and read those links when I read the article, but I didn’t think they were exactly what I was talking about. As I understand it, orthogonality says that it’s perfectly possible for an intelligence to be superhuman and also to really want paperclips more than anything. What I’m wondering is whether an intelligence can change its mind about what it wants as it gains more intelligence? I’m not really interested in whether it would lead to ethics which we’d approve it, just whether it can decide what it wants for itself. Is there a term for that idea (other than “free will”, I suppose)?
Thank you. I did follow and read those links when I read the article, but I didn’t think they were exactly what I was talking about. As I understand it, orthogonality says that it’s perfectly possible for an intelligence to be superhuman and also to really want paperclips more than anything. What I’m wondering is whether an intelligence can change its mind about what it wants as it gains more intelligence? I’m not really interested in whether it would lead to ethics which we’d approve it, just whether it can decide what it wants for itself. Is there a term for that idea (other than “free will”, I suppose)?
I don’t understand; why would changing its mind about what it wants help it make more paperclips?