By self-awareness I mean being able to have goals that are distinct from the goals of the outside world. I didn’t think of your interpretation – Illusion of transparency, I guess… I modified the original post accordingly, thanks for pointing it out.
I feel like you might want to re-read this.
I’m not thinking in terms of self-awareness being a “ghost in the machine”, I’m thinking of it in terms of arising from complexity. So while I might be making a mistake, it’s at least a more subtle one.
So by self-awareness, do you mean “has a model of itself?” That seems like a perfectly fine thing to have in your Tool AI.
EDIT: I feel like you might want to re-read this. Maybe I’m wrong.
By self-awareness I mean being able to have goals that are distinct from the goals of the outside world. I didn’t think of your interpretation – Illusion of transparency, I guess… I modified the original post accordingly, thanks for pointing it out.
I’m not thinking in terms of self-awareness being a “ghost in the machine”, I’m thinking of it in terms of arising from complexity. So while I might be making a mistake, it’s at least a more subtle one.
Is there some reason a simple program can’t “have goals different from those of the outside world?”