-3. I’m assuming you are already familiar with some basics, and already know what ‘orthogonality’ and ‘instrumental convergence’ are and why they’re true. People occasionally claim to me that I need to stop fighting old wars here, because, those people claim to me, those wars have already been won within the important-according-to-them parts of the current audience. I suppose it’s at least true that none of the current major EA funders seem to be visibly in denial about orthogonality or instrumental convergence as such; so, fine. If you don’t know what ‘orthogonality’ or ‘instrumental convergence’ are, or don’t see for yourself why they’re true, you need a different introduction than this one.
Thank you. I did follow and read those links when I read the article, but I didn’t think they were exactly what I was talking about. As I understand it, orthogonality says that it’s perfectly possible for an intelligence to be superhuman and also to really want paperclips more than anything. What I’m wondering is whether an intelligence can change its mind about what it wants as it gains more intelligence? I’m not really interested in whether it would lead to ethics which we’d approve it, just whether it can decide what it wants for itself. Is there a term for that idea (other than “free will”, I suppose)?
The links here are talking about this topic:
Thank you. I did follow and read those links when I read the article, but I didn’t think they were exactly what I was talking about. As I understand it, orthogonality says that it’s perfectly possible for an intelligence to be superhuman and also to really want paperclips more than anything. What I’m wondering is whether an intelligence can change its mind about what it wants as it gains more intelligence? I’m not really interested in whether it would lead to ethics which we’d approve it, just whether it can decide what it wants for itself. Is there a term for that idea (other than “free will”, I suppose)?
I don’t understand; why would changing its mind about what it wants help it make more paperclips?