By self-awareness I mean being able to have goals that are distinct from the goals of the outside world. I didn’t think of your interpretation – Illusion of transparency, I guess… I modified the original post accordingly, thanks for pointing it out.
I feel like you might want to re-read this.
I’m not thinking in terms of self-awareness being a “ghost in the machine”, I’m thinking of it in terms of arising from complexity. So while I might be making a mistake, it’s at least a more subtle one.
By self-awareness I mean being able to have goals that are distinct from the goals of the outside world. I didn’t think of your interpretation – Illusion of transparency, I guess… I modified the original post accordingly, thanks for pointing it out.
I’m not thinking in terms of self-awareness being a “ghost in the machine”, I’m thinking of it in terms of arising from complexity. So while I might be making a mistake, it’s at least a more subtle one.
Is there some reason a simple program can’t “have goals different from those of the outside world?”