I found the Clark et al. (2019) “Bayesing Qualia” article very useful, and that did give me an intuition of the account that perhaps sentience arises out of self-awareness. But they themselves acknowledged in their conclusion that the paper didn’t quite demonstrate that principle, and I didn’t find myself convinced of it.
Perhaps what I’d like readers to take away is that sentience and self-awareness can be at the very least conceptually distinguished. Even if it isn’t clear empirically whether or not they are intrinsically linked, we ought to maintain a conceptual distinction in order to form testable hypotheses about whether they are in fact linked, and in order to reason about the nature of any link. Perhaps I should call that “Theoretical orthogonality”. This is important to be able to reason whether, for instance, giving our AIs a self-awareness or situational awareness will cause them to be sentient. I do not think that will be the case, although I do think that, if you gave them the sort of detailed self-monitoring feelings that humans have, that may yield sentience itself. But it’s not clear!
I listened to the whole episode with Bach as a result of your recommendation! Bach hardly even got a chance to express his ideas, and I’m not much closer to understanding his account of
meta-awareness (i.e., awareness of awareness) within the model of oneself which acts as a ‘first-person character’ in the movie/dream/”controlled hallucination” that the human brain constantly generates for oneself is the key thing that also compels the brain to attach qualia (experiences) to the model. In other words, the “character within the movie” thinks that it feels something because it has meta-awareness (i.e., the character is aware that it is aware (which reflects the actual meta-cognition in the brain, rather than in the brain, insofar the character is a faithful model of reality).
which seems like a crux here.
He sort of briefly described “consciousness as a dream state” at the very end, but although I did get the sense that maybe he thinks meta-awareness and sentience are connected, I didn’t really hear a great argument for that point of view.
He spent several minutes arguing that agency, or seeking a utility function, is something humans have, but that these things aren’t sufficient for consciousness (I don’t remember whether he said whether they were necessary, so I suppose we don’t know if he thinks they’re orthogonal).
You might have noticed I didn’t actually fully differentiate intelligence and agency. It seems to me to exert agency a mind needs a certain amount of intelligence, and so I think all agents are intelligent, though not all intelligences are agentic. Agents that are minimally intelligent (like simple RL agents in simple computer models) also are pretty minimally agentic. I’d be curious to hear about a counter-example.
Incidentally I also like Anil Seth’s work and I liked his recent book on consciousness, apart from the bit about AGI. I read it right along with Damasio’s latest book on consciousness and they paired pretty well. Seth is a bit more concrete and detail oriented and I appreciated that.
It would make it much easier to understand ideas in this area if writers used more conceptual clarity, particularly empirical consciousness researchers (philosophers can be a bit better, I think, and I say that as an empirical researcher myself). When I read that quote from Seth, it seems clear he was arguing AGI is unlikely to be an existential threat because it’s unlikely to be conscious. Does he naively conflate consciousness with agency, because he’s not an artificial agency researcher and hasn’t thought much about it? Or does he have a sophisticated point of view about how agency and consciousness really are linked, based on his ~~couple decades of consciousness research? Seems very unlikely, given how much we know about artificial agents, but the only way to be clear is to ask him.
Similarly MANY people including empirical researchers and maybe philosophers treat consciousness and self-awareness as somewhat synonymous, or at least interdependent. Is that because they’re being naive about the link, or because, as outlined in Clark, Friston, & WIlkinson’s Bayesing Qualia, they have sophisticated theories based on evidence that there really are tight links between the two? I think when writing this post I was pretty sure consciousness and self-awareness were “orthogonal”/independent, and now, following other discussion in the comments here and on Facebook, I’m less clear about that. But I’d like more people do what Friston did as he explained exactly why he thinks consciousness arises from self-awareness/meta-cognition.