We both have a similar intuition about the kinds of optimizers we’re interested in. You say they optimize things that are “far away”, I say they affect “big pieces of the environment”. One difference is that I think of big as relative to the size of the agent, but something can be “far away” even if the agent is itself quite large, and it seems that agent size doesn’t necessarily matter to your scheme because the information lost over a given distance doesn’t depend on whether there’s a big agent or a small one trying to exert influence over this distance.
I think agent size (in the sense I’m thinking about it) is mainly relevant from the point of view of “how likely is it for such an agent to come about?” (which suggest something like “large measure, given initial conditions + dynamics” instead of “small size”).
I think my scheme needs some distinction between “microstates” and “macrostates” in order to offer a reasonable definition of “big features”. Your setup seems to have this fairly naturally in terms of the telephone theorem, though the precise analogy (if there is one) isn’t striking me immediately.
We both have a similar intuition about the kinds of optimizers we’re interested in. You say they optimize things that are “far away”, I say they affect “big pieces of the environment”. One difference is that I think of big as relative to the size of the agent, but something can be “far away” even if the agent is itself quite large, and it seems that agent size doesn’t necessarily matter to your scheme because the information lost over a given distance doesn’t depend on whether there’s a big agent or a small one trying to exert influence over this distance.
I think agent size (in the sense I’m thinking about it) is mainly relevant from the point of view of “how likely is it for such an agent to come about?” (which suggest something like “large measure, given initial conditions + dynamics” instead of “small size”).
Here are some of my thoughts on the issue: https://www.lesswrong.com/posts/me34KqMLwJNYAZKbs/is-evolutionary-influence-the-mesa-objective-that-we-re
I think my scheme needs some distinction between “microstates” and “macrostates” in order to offer a reasonable definition of “big features”. Your setup seems to have this fairly naturally in terms of the telephone theorem, though the precise analogy (if there is one) isn’t striking me immediately.