So far it seems like you are broadly reinventing concepts which are natural and understood in predictive processing and active inference.
Here is rough attempt at translation / pointer to what you are describing: what you call frames is usually called predictive models or hierarchical generative models in PP literature
Unlike logical propositions, frames can’t be evaluated as discretely true or false. Sure: predictive models are evaluated based on prediction error, which is roughly a combination of ability to predict outputs of lower level layers, not deviating too much from predictions of higher order models, and being useful for modifying the world.
Unlike Bayesian hypotheses, frames aren’t mutually exclusive, and can overlap with each other. This (along with point Frames in context Sure: predictive models overlap, and it is somewhat arbitrary where you would draw boundaries of individual models. E.g. you can draw a very broad boundary around a model call microeconomics, and a very broad boundary around a model called Buddhist philosophy, but both models likely share some parts modelling something like human desires
Unlike in critical rationalism, we evaluate frames (partly) in terms of how true they are (based on their predictions) rather than just whether they’ve been falsified or not. Sure: actually science roughly is “cultural evolution rediscovered active inference”. Models are evaluated based on prediction error.
Unlike Garrabrant traders and Rational Inductive Agents, frames can output any combination of empirical content (e.g. predictions about the world) and normative content (e.g. evaluations of outcomes, or recommendations for how to act). Sure: actually, the “any combination” goes even further. In active inference, there is no strict type difference between predictions about stuff like “what photons hit photoreceptors in your eyes” and stuff like “what should be a position of your muscles”. Recommendations how to act are just predictions about your actions conditional of wishful oriented beliefs about future states. Evaluations of outcomes are just prediction errors between wishful models and observations.
Unlike model-based policies, policies composed of frames can’t be decomposed into modules with distinct functions, because each frame plays multiple roles. Mostly but this description seems a bit confused. “This has distinct function” is a label you slap on a computation using design stance, if the design stance description is much shorter than the alternatives (e.g. physical stance description). In case of hierarchical predictive models, you can imagine drawing various boundaries around various parts of the system (e.g., you can imagine alternatives of including or not including layers computing edge detection in a model tracking whether someone is happy, and in the other direction you can imagine including and not including layers with some abstract conceptions of hedonic utilitarianism vs. some transcendental purpose). Once you select a boundary, you can sometimes assign “distinct function” to it, sometimes more than one, sometimes “distinct goal”, etc. It’s just a question of how useful are physical/design/intentional stances.
Unlike in multi-agent RL, frames don’t interact independently with their environment, but instead contribute towards choosing the actions of a single agent. Sure: this is exactly what hierarchical predictive models do in PP. All the time different models are competing for predictions about what will happen, or what will do.
Assuming this more or less shows that what you are talking about is mostly hierarchical generative models from active inference, here are more things the same model predict
a. Hierarchical generative models are the way how people do perception. predictive error is minimized between a stream of prediction from upper layers (containing deep models like “the world has gravity” or “communism is good”) and stream of errors from the direction of senses. Given that, what is naively understood as “observations” is … more complex phenomenon, where e.g. leaf flying sideways is interpreted given strong priors like there is gravity pointing downward, and an atmosphere, and given that, the model predicting “wind is blowing” decreases the sensory prediction error. Similarly, someone being taken into custody by KGB is, under the upstream model of “soviet communism is good” prior, interpreted as the person likely being a traitor. In this case competing broad model “soviet communism is evil totalitarian dictatorship” could actually predict the same person being taken into custody, just interpreting it as the system prosecuting dissidents.
b. It is possible to look at parts of this modelling machinery wearing intentional stance hat. If you do this, the system looks like multi-agent mind, and you can - derive a bunch of IFC/ICF style of intuitions - see parts of it as econ interaction or market—the predictive models compete for making predictions, “pay” a complexity cost, are rewarded for making “correct” predictions (correct here meaning minimizing error between the model and the reality, which can include changing the reality, aka pursuing goals) What’s the main difference between naive/straightforward multi-agent mind models is the “parts” live within a generative model, and interact with it and though it, not through the world. They don’t have any direct access to reality, and compete at the same time for interpreting sensory inputs and predicting actions.
Useful comment, ty! I’m planning to talk about predictive processing in the next two posts, and I do think that predictive processing is a very useful frame here. I will probably edit this post and the previous post too to highlight the connection.
I don’t want to fully round off “frames = hierarchical generative models” though, for a couple of reasons.
I feel pretty uncertain about the “Recommendations how to act are just predictions about your actions conditional of wishful oriented beliefs about future states. Evaluations of outcomes are just prediction errors between wishful models and observations” thing. Maybe that’s true at a low level but it doesn’t seem true at the level which makes it useful for doing epistemology. E.g. my high-level frames know the difference between acting and predicting, and use sophisticated planning to do the former. Now, you could say that explicit planning is still “just making a prediction”, but I think it’s a noncentral example of prediction.
Furthermore it seems like the credit assignment for actions needs to work differently from the credit assignment for predictions, otherwise you get the dark room problem.
More generally I think hierarchical generative models underrate the “reasoning” component of thinking. E.g. suppose I’m in a dark room and I have a sudden insight that transforms my models. Perhaps you could describe this naturally in HGM terms, but I suspect it’s tricky.
So right now I’m thinking of HGMs as “closely related to frames, but with a bunch of connotations that make it undesirable to start talking about them instead of about frames themselves”. But I’m still fairly uncertain about these points.
I broadly agree with something like “we use a lot of explicit S2 algorithms built on top of the modelling machinery described”, so yes, what I mean more directly apply to the low level, than to humans explicitly thinking about what steps to take.
I think practically useful epistemology for humans needs to deal with both “how is it implemented” and “what’s the content”. To use ML metaphor: human cognition is build out of both “trained neural nets” and “chain-of-thought type inferences in language” running on top of such nets. All S2 reasoning is a prediction in somewhat similar way as all GPT3 reasoning is a prediction—the NN predictor learns how to make “correct predictions” of language, but because the domain itself is partially symbolic world model, this maps to predictions about the world.
In my view some parts of traditional epistemology are confused in trying to do epistemology for humans basically only at the level of the language reasoning, which is a bit like if you try to fix LLM cognition just by writing smart prompts, and ignore there is this huge underlying computation which does the heavy lifting.
I’m certainly in favour of attempts to do epistemology for humans which are compatible with what the underlying computation actually does.
I do agree you can go too far in the opposite direction, ignoring the symbolic reason … but seems rare when people think about humans?
2. My personal take on dark room problem is it is in case of humans mostly fixed by “fixed priors” on interoceptive inputs. I.e. your body has evolutionary older machinery to compute hunger. This gets fed into the predictive processing machinery as input, and the evolutionary sensible belief (“not hungry”) gets fixed. (I don’t think calling this “priors” was good choice of terminology...).
This setup at least in theory rewards both prediction and action, and avoids dark room problems for practical purposes: let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
Given that evolution has good reasons to “fix priors” on multiple evolutionary relevant inputs, I would not expect actual humans to seek dark rooms, but I would expect the PP system occasionally seeking a way how to block or modify the interoceptive signals
3. My impression about how you use ‘frames’ is … the central examples are more like somewhat complex model ensembles including some symbolic/language based components, rather than e.g. “there is gravity” frame or “model of apple” frame. My guess is this will likely be useful for practical use, but with attempts to formalize it, I think a better option is to start with the existing HGM maths.
let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
I still think you need to have multiple types of belief here, because this fixed prior can’t be used to make later deductions about the world. For example, suppose that I’m stranded in the desert with no food. It’s a new situation, I’ve never been there before. If my prior strongly believes I won’t be hungry 10 hour in the future, I can infer that I’m going to be rescued; and if my prior strongly believes I won’t be sleepy 10 hours from now, then I can infer I’ll be rescued without needing to do anything except take a nap. But of course I can’t (and won’t) infer that.
(Maybe you’ll say “well, you’ve learned from previous experience that the prior is only true if you can actually figure out a way of making it true”? But then you may as well just call it a “goal”, I don’t see the sense in which it’s a belief.)
This type of thing is why I’m wary about “starting with the existing HGM maths”. I agree that it’s rare for humans to ignore symbolic reasoning… but the HGM math might ignore symbolic reasoning! And it could happen in a way that’s pretty hard to spot. If this were my main research priority I’d do it anyway (although even then maybe I’d write this sequence first) but as it is my main goal here is to have a minimum viable epistemology which refutes bayesian rationalism, and helps rationalists reason better about AI.
I’d be interested in your favorite links to the HGM math though, sounds very useful to read up more on it.
So far it seems like you are broadly reinventing concepts which are natural and understood in predictive processing and active inference.
Here is rough attempt at translation / pointer to what you are describing: what you call frames is usually called predictive models or hierarchical generative models in PP literature
Unlike logical propositions, frames can’t be evaluated as discretely true or false.
Sure: predictive models are evaluated based on prediction error, which is roughly a combination of ability to predict outputs of lower level layers, not deviating too much from predictions of higher order models, and being useful for modifying the world.
Unlike Bayesian hypotheses, frames aren’t mutually exclusive, and can overlap with each other. This (along with point Frames in context
Sure: predictive models overlap, and it is somewhat arbitrary where you would draw boundaries of individual models. E.g. you can draw a very broad boundary around a model call microeconomics, and a very broad boundary around a model called Buddhist philosophy, but both models likely share some parts modelling something like human desires
Unlike in critical rationalism, we evaluate frames (partly) in terms of how true they are (based on their predictions) rather than just whether they’ve been falsified or not.
Sure: actually science roughly is “cultural evolution rediscovered active inference”. Models are evaluated based on prediction error.
Unlike Garrabrant traders and Rational Inductive Agents, frames can output any combination of empirical content (e.g. predictions about the world) and normative content (e.g. evaluations of outcomes, or recommendations for how to act).
Sure: actually, the “any combination” goes even further. In active inference, there is no strict type difference between predictions about stuff like “what photons hit photoreceptors in your eyes” and stuff like “what should be a position of your muscles”. Recommendations how to act are just predictions about your actions conditional of wishful oriented beliefs about future states. Evaluations of outcomes are just prediction errors between wishful models and observations.
Unlike model-based policies, policies composed of frames can’t be decomposed into modules with distinct functions, because each frame plays multiple roles.
Mostly but this description seems a bit confused. “This has distinct function” is a label you slap on a computation using design stance, if the design stance description is much shorter than the alternatives (e.g. physical stance description). In case of hierarchical predictive models, you can imagine drawing various boundaries around various parts of the system (e.g., you can imagine alternatives of including or not including layers computing edge detection in a model tracking whether someone is happy, and in the other direction you can imagine including and not including layers with some abstract conceptions of hedonic utilitarianism vs. some transcendental purpose). Once you select a boundary, you can sometimes assign “distinct function” to it, sometimes more than one, sometimes “distinct goal”, etc. It’s just a question of how useful are physical/design/intentional stances.
Unlike in multi-agent RL, frames don’t interact independently with their environment, but instead contribute towards choosing the actions of a single agent.
Sure: this is exactly what hierarchical predictive models do in PP. All the time different models are competing for predictions about what will happen, or what will do.
Assuming this more or less shows that what you are talking about is mostly hierarchical generative models from active inference, here are more things the same model predict
a. Hierarchical generative models are the way how people do perception. predictive error is minimized between a stream of prediction from upper layers (containing deep models like “the world has gravity” or “communism is good”) and stream of errors from the direction of senses. Given that, what is naively understood as “observations” is … more complex phenomenon, where e.g. leaf flying sideways is interpreted given strong priors like there is gravity pointing downward, and an atmosphere, and given that, the model predicting “wind is blowing” decreases the sensory prediction error. Similarly, someone being taken into custody by KGB is, under the upstream model of “soviet communism is good” prior, interpreted as the person likely being a traitor. In this case competing broad model “soviet communism is evil totalitarian dictatorship” could actually predict the same person being taken into custody, just interpreting it as the system prosecuting dissidents.
b. It is possible to look at parts of this modelling machinery wearing intentional stance hat. If you do this, the system looks like multi-agent mind, and you can
- derive a bunch of IFC/ICF style of intuitions
- see parts of it as econ interaction or market—the predictive models compete for making predictions, “pay” a complexity cost, are rewarded for making “correct” predictions (correct here meaning minimizing error between the model and the reality, which can include changing the reality, aka pursuing goals)
What’s the main difference between naive/straightforward multi-agent mind models is the “parts” live within a generative model, and interact with it and though it, not through the world. They don’t have any direct access to reality, and compete at the same time for interpreting sensory inputs and predicting actions.
Useful comment, ty! I’m planning to talk about predictive processing in the next two posts, and I do think that predictive processing is a very useful frame here. I will probably edit this post and the previous post too to highlight the connection.
I don’t want to fully round off “frames = hierarchical generative models” though, for a couple of reasons.
I feel pretty uncertain about the “Recommendations how to act are just predictions about your actions conditional of wishful oriented beliefs about future states. Evaluations of outcomes are just prediction errors between wishful models and observations” thing. Maybe that’s true at a low level but it doesn’t seem true at the level which makes it useful for doing epistemology. E.g. my high-level frames know the difference between acting and predicting, and use sophisticated planning to do the former. Now, you could say that explicit planning is still “just making a prediction”, but I think it’s a noncentral example of prediction.
Furthermore it seems like the credit assignment for actions needs to work differently from the credit assignment for predictions, otherwise you get the dark room problem.
More generally I think hierarchical generative models underrate the “reasoning” component of thinking. E.g. suppose I’m in a dark room and I have a sudden insight that transforms my models. Perhaps you could describe this naturally in HGM terms, but I suspect it’s tricky.
So right now I’m thinking of HGMs as “closely related to frames, but with a bunch of connotations that make it undesirable to start talking about them instead of about frames themselves”. But I’m still fairly uncertain about these points.
I broadly agree with something like “we use a lot of explicit S2 algorithms built on top of the modelling machinery described”, so yes, what I mean more directly apply to the low level, than to humans explicitly thinking about what steps to take.
I think practically useful epistemology for humans needs to deal with both “how is it implemented” and “what’s the content”. To use ML metaphor: human cognition is build out of both “trained neural nets” and “chain-of-thought type inferences in language” running on top of such nets. All S2 reasoning is a prediction in somewhat similar way as all GPT3 reasoning is a prediction—the NN predictor learns how to make “correct predictions” of language, but because the domain itself is partially symbolic world model, this maps to predictions about the world.
In my view some parts of traditional epistemology are confused in trying to do epistemology for humans basically only at the level of the language reasoning, which is a bit like if you try to fix LLM cognition just by writing smart prompts, and ignore there is this huge underlying computation which does the heavy lifting.
I’m certainly in favour of attempts to do epistemology for humans which are compatible with what the underlying computation actually does.
I do agree you can go too far in the opposite direction, ignoring the symbolic reason … but seems rare when people think about humans?
2. My personal take on dark room problem is it is in case of humans mostly fixed by “fixed priors” on interoceptive inputs. I.e. your body has evolutionary older machinery to compute hunger. This gets fed into the predictive processing machinery as input, and the evolutionary sensible belief (“not hungry”) gets fixed. (I don’t think calling this “priors” was good choice of terminology...).
This setup at least in theory rewards both prediction and action, and avoids dark room problems for practical purposes: let’s assume I have this really strong belief (“fixed prior”) I won’t be hungry 1 hour in future. Conditional on that, I can compute what are my other sensory inputs half an hour from now. Predictive model of me eating a tasty food in half an hour is more coherent with me being not hungry than predictive model of me reading a book—but this does not need to be hardwired, but can be learned.
Given that evolution has good reasons to “fix priors” on multiple evolutionary relevant inputs, I would not expect actual humans to seek dark rooms, but I would expect the PP system occasionally seeking a way how to block or modify the interoceptive signals
3. My impression about how you use ‘frames’ is … the central examples are more like somewhat complex model ensembles including some symbolic/language based components, rather than e.g. “there is gravity” frame or “model of apple” frame. My guess is this will likely be useful for practical use, but with attempts to formalize it, I think a better option is to start with the existing HGM maths.
I still think you need to have multiple types of belief here, because this fixed prior can’t be used to make later deductions about the world. For example, suppose that I’m stranded in the desert with no food. It’s a new situation, I’ve never been there before. If my prior strongly believes I won’t be hungry 10 hour in the future, I can infer that I’m going to be rescued; and if my prior strongly believes I won’t be sleepy 10 hours from now, then I can infer I’ll be rescued without needing to do anything except take a nap. But of course I can’t (and won’t) infer that.
(Maybe you’ll say “well, you’ve learned from previous experience that the prior is only true if you can actually figure out a way of making it true”? But then you may as well just call it a “goal”, I don’t see the sense in which it’s a belief.)
This type of thing is why I’m wary about “starting with the existing HGM maths”. I agree that it’s rare for humans to ignore symbolic reasoning… but the HGM math might ignore symbolic reasoning! And it could happen in a way that’s pretty hard to spot. If this were my main research priority I’d do it anyway (although even then maybe I’d write this sequence first) but as it is my main goal here is to have a minimum viable epistemology which refutes bayesian rationalism, and helps rationalists reason better about AI.
I’d be interested in your favorite links to the HGM math though, sounds very useful to read up more on it.