Criticism of Markov blankets as «boundaries» from Abram Demski
I spoke to Abram Demski about the general idea of using Markov blankets to specify «boundaries» and «boundary» violations, and he shared his doubt with me. In the text below, I have attempted to summarize his position in a fictional conversation.
Epistemic status: I wrote a draft of this section, and then Demski made several comments which I then integrated.
Me: «Boundaries»! Markov blankets!
AD: Eh, I think Markov blankets probably aren’t the way to go to represent “boundaries”. Markov blankets can’t follow entities through space as well as you might imagine. Suppose you have a Bayesian network whose variables are the physical properties of small regions in space-time. A Markov blanket is a set of nodes (with specific conditional independence requirements, but that part isn’t important right now). Say your probabilistic model contains a grasshopper who might hop in two different directions, depending on stimuli. Since your model represents both possibilities, you can’t select a Markov blanket to contain all and only the grasshopper. You’re forced to include some empty air (if you include either of the hop-trajectories inside your Markov blanket) or to miss some grasshopper (if you include only space-time regions certain to contain the grasshopper, by only circling the grasshopper before it makes the uncertain jump).
Me: Ah, I see.
Me: Hm, are there any other frameworks that you think might work instead?
AD: I don’t see any currently. (Including Cartesian Frames and Finite Factored Sets. And Active Inference doesn’t work for this as it just makes the same Markov-blanket mistake, IMO.) That said, I don’t see any fundamental difficulty in why this mathematical structure can’t exist. Frankly, I would like to find a mathematical definition of “trying not to violate another agent’s boundary”.
(FWIW, three other alignment researchers have told me that they don’t get Demski’s criticism and disagree with it a lot.)
I removed the following section from the main post because Abram updated. See: Agent Boundaries Aren’t Markov Blankets. [no longer endorsed].
Criticism of Markov blankets as «boundaries» from Abram Demski
I spoke to Abram Demski about the general idea of using Markov blankets to specify «boundaries» and «boundary» violations, and he shared his doubt with me. In the text below, I have attempted to summarize his position in a fictional conversation.
Epistemic status: I wrote a draft of this section, and then Demski made several comments which I then integrated.
(FWIW, three other alignment researchers have told me that they don’t get Demski’s criticism and disagree with it a lot.)