Imagine that I’m watching the video of the squirgle, and suddenly the left half of the TV blue-screens. Then I’d probably think “ah, something messed up the TV, so it’s no longer showing me the squirgle” as opposed to “ah, half the squirgle just turned into a big blue square”. I know that big square chunks turning a solid color is a typical way for TVs to break, which largely explains away the observation; I think it much more likely that the blue half-screen came from some failure of the TV rather than an unprecedented behavior of the squirgle.
My mental model of this is something like: My concept of a squirgle is a function f(x) which maps latent variables x to observations such that likelier observations correspond to latent variables with lower description length.
Suppose that we currently settle on a particular latent variable x0, but we receive new observations that are incompatible with f(x0), and these new observations can be most easily accounted for by modifying x0 to a new latent variable x1 that’s pretty close to x0, then we say that this change is still about squirgle
But if we receive new observations that can be more easily accounted for by perturbing a different latent variable y that corresponds to another concept g(y) (eg about TV), then that is a change about a different thing and not the squirgle
The main property that enables this kind of separation is modularity of the world model, because when most components are independent of most other components at any given time, only a change in a few latent variables (as opposed to most latent variables) is required to accomodate new beliefs, & that allows us to attribute changes in beliefs into changes about disentangled concepts
My mental model of this is something like: My concept of a squirgle is a function f(x) which maps latent variables x to observations such that likelier observations correspond to latent variables with lower description length.
Suppose that we currently settle on a particular latent variable x0, but we receive new observations that are incompatible with f(x0), and these new observations can be most easily accounted for by modifying x0 to a new latent variable x1 that’s pretty close to x0, then we say that this change is still about squirgle
But if we receive new observations that can be more easily accounted for by perturbing a different latent variable y that corresponds to another concept g(y) (eg about TV), then that is a change about a different thing and not the squirgle
The main property that enables this kind of separation is modularity of the world model, because when most components are independent of most other components at any given time, only a change in a few latent variables (as opposed to most latent variables) is required to accomodate new beliefs, & that allows us to attribute changes in beliefs into changes about disentangled concepts