My current model is real-world Bayesian agents end up with a fractal of latents to address the complexity of the world, and for communication/interpretability/alignment, you want to ignore the vast majority of these latents. Furthermore, most agents are gonna be like species of microbes that are so obscure and weak that we just want to entirely ignore them.
Natural latents don’t seem to zoom in to what really matters, and instead would risk getting distracted by stuff like the above.
Good summary. I got that from your comment on our previous post, but it was less clear.
The main natural-latents-flavored answer to this would be: different latents are natural over different chunks of the world, and in particular some latents are natural over much bigger (in the volume-of-spacetime sense) parts of the world. So, for instance, the latent summarizing the common features of cats is distributed over all the world’s cats, of which there are many in many places on the scale of Earth’s surface. On the other hand, the latent summarizing the specifics of one particular cat’s genome is distributed over all the cells of that particular cat, but that means it’s relevant to a much smaller chunk of spacetime than the common-features-of-cats latent. And since one-cat’s-genome latent is relevant to a much smaller chunk of spacetime, it’s much less likely to be relevant to any particular agent or decision, unless the agent has strong information that it’s going to be nearby that particular cat a lot.
So there’s a general background prior that latents distributed over more spacetime are more likely to be relevant, and that general background prior can also be overridden by more agent-specific information, like e.g. nearby-ness or repeated encounters or whatever.
Or perhaps a better/more-human phrasing than my mouse comment is, the attributes that are in common between cats across the world are not the attributes that matter the most for cats. Cats are relatively bounded, so perhaps mostly their aggregate ecological impact is what matters.
Cats seem relatively epiphenomenal unless you’re like, a mouse. So let’s say you are a mouse. You need to avoid cats and find cheese without getting distracted by dust. In particular, you need to avoid the cat every time, not just on your 5th time.
My current model is real-world Bayesian agents end up with a fractal of latents to address the complexity of the world, and for communication/interpretability/alignment, you want to ignore the vast majority of these latents. Furthermore, most agents are gonna be like species of microbes that are so obscure and weak that we just want to entirely ignore them.
Natural latents don’t seem to zoom in to what really matters, and instead would risk getting distracted by stuff like the above.
Good summary. I got that from your comment on our previous post, but it was less clear.
The main natural-latents-flavored answer to this would be: different latents are natural over different chunks of the world, and in particular some latents are natural over much bigger (in the volume-of-spacetime sense) parts of the world. So, for instance, the latent summarizing the common features of cats is distributed over all the world’s cats, of which there are many in many places on the scale of Earth’s surface. On the other hand, the latent summarizing the specifics of one particular cat’s genome is distributed over all the cells of that particular cat, but that means it’s relevant to a much smaller chunk of spacetime than the common-features-of-cats latent. And since one-cat’s-genome latent is relevant to a much smaller chunk of spacetime, it’s much less likely to be relevant to any particular agent or decision, unless the agent has strong information that it’s going to be nearby that particular cat a lot.
So there’s a general background prior that latents distributed over more spacetime are more likely to be relevant, and that general background prior can also be overridden by more agent-specific information, like e.g. nearby-ness or repeated encounters or whatever.
Or perhaps a better/more-human phrasing than my mouse comment is, the attributes that are in common between cats across the world are not the attributes that matter the most for cats. Cats are relatively bounded, so perhaps mostly their aggregate ecological impact is what matters.
Cats seem relatively epiphenomenal unless you’re like, a mouse. So let’s say you are a mouse. You need to avoid cats and find cheese without getting distracted by dust. In particular, you need to avoid the cat every time, not just on your 5th time.