I’m inclined to think so, mostly because terms shouldn’t be introduced unnecessarily. If we can already talk about systems that are capable/competent at certain tasks, then we should just do that directly.
I guess the mesa- prefix helps point towards the fact that we’re talking about policies, not policies + optimisers.
Probably my preferred terminology would be:
Instead of mesa-controller, “competent policy”.
And then we can say that competent policies sometimes implement search or learning (or both, or possibly neither).
And when we need to be clear, we can add the mesa- prefix to search or learning. (Although I’m not sure whether something like AlphaGo is a mesa-searcher—does the search need to be emergent?)
This helps make it clear that mesa-controller isn’t a disjoint category from mesa-searcher, and also that mesa-controller is the default, rather than a special case.
Having written all this I’m now a little confused about the usefulness of the mesa-optimisation terminology at all, and I’ll need to think about it more. In particular, it’s currently unclear to me what the realistic alternative to mesa-optimisation is, which makes me wonder if it’s actually carving off an important set of possibilities, or just reframing the whole space of possibilities. (If the policy receives a gradient update every minute, is it useful to call it a mesa-optimiser? Or every hour? Or...)
I’m inclined to think so, mostly because terms shouldn’t be introduced unnecessarily. If we can already talk about systems that are capable/competent at certain tasks, then we should just do that directly.
Thinking about this more, I think maybe what I really want it to mean is: competent policies which are non-myopic in some sense. A truly myopic Q&A system doesn’t feel much like a controller / inner optimizer (even if it is misaligned, it’s not steering the world in a bad direction, because it’s totally myopic).
I’m not sure what sense of “myopia” I want to use, though.
To me it sounds like you’re describing (some version of) agency, and so the most natural term to use would be mesa-agent.
I’m a bit confused about the relationship between “optimiser” and “agent”, but I tend to think of the latter as more compressed, and so insofar as we’re talking about policies it seems like “agent” is appropriate. Also, mesa-optimiser is taken already (under a definition which assumes that optimisation is equivalent to some kind of internal search).
I used to define “agent” as “both a searcher and a controller”, IE, something which uses an internal selection/search of some kind to accomplish an external control task. This might be too restrictive, though.
I used to define “agent” as “both a searcher and a controller”
Oh, I really like this definition. Even if it’s too restrictive, it seems like it gets at something important.
I’m not sure what you meant by “more compressed”.
Sorry, that was quite opaque. I guess what I mean is that evolution is an optimiser but isn’t an agent, and in part this has to do with how it’s a very distributed process with no clear boundary around it. Whereas when you have the same problem being solved in a single human brain, then that compression makes it easier to point to the human as being an agent separate from its environment.
The rest of this comment is me thinking out loud in a somewhat incoherent way; no pressure to read/respond.
It seems like calling something a “searcher” describes only a very simple interface: at the end of the search, there needs to be some representation of the output which it has found. But that output may be very complex.
Whereas calling something a “controller” describes a much more complex interface between it and its environment: you need to be able to point not just to outcomes, but also to observations and actions. But each of those actions is usually fairly simple for a pure controller; if it’s complex, then you need search to find which action to take at each step.
Now, it seems useful to sometimes call evolution a controller. For example, suppose you’re trying to wipe out a virus, but it keeps mutating. Then there’s a straightforward sense in which evolution is “steering” the world towards states where the virus still exists, in the short term. You could also say that it’s steering the world towards states where all organisms have high fitness in the long term, but organisms are so complex that it’s easier to treat them as selected outcomes, and abstract away from the many “actions” by evolution which led to this point.
In other words, evolution searches using a process of iterative control. Whereas humans control using a process of iterative search.
(As a side note, I’m now thinking that “search” isn’t quite the right word, because there are other ways to do selection than search. For example, if I construct a mathematical proof (or a poem) by writing it one line at a time, letting my intuition guide me, then it doesn’t really seem accurate to say that I’m searching over the space of proofs/poems. Similarly, a chain of reasoning may not branch much, but still end up finding a highly specific conclusion. Yet “selection” also doesn’t really seem like the right word either, because it’s at odds with normal usage, which involves choosing from a preexisting set of options—e.g. you wouldn’t say that a poet is “selecting” a poem. How about “design” as an alternative? Which allows us to be agnostic about how the design occurred—whether it be via a control process like evolution, or a process of search, or a process of reasoning.)
I’m inclined to think so, mostly because terms shouldn’t be introduced unnecessarily. If we can already talk about systems that are capable/competent at certain tasks, then we should just do that directly.
I guess the mesa- prefix helps point towards the fact that we’re talking about policies, not policies + optimisers.
Probably my preferred terminology would be:
Instead of mesa-controller, “competent policy”.
And then we can say that competent policies sometimes implement search or learning (or both, or possibly neither).
And when we need to be clear, we can add the mesa- prefix to search or learning. (Although I’m not sure whether something like AlphaGo is a mesa-searcher—does the search need to be emergent?)
This helps make it clear that mesa-controller isn’t a disjoint category from mesa-searcher, and also that mesa-controller is the default, rather than a special case.
Having written all this I’m now a little confused about the usefulness of the mesa-optimisation terminology at all, and I’ll need to think about it more. In particular, it’s currently unclear to me what the realistic alternative to mesa-optimisation is, which makes me wonder if it’s actually carving off an important set of possibilities, or just reframing the whole space of possibilities. (If the policy receives a gradient update every minute, is it useful to call it a mesa-optimiser? Or every hour? Or...)
Thinking about this more, I think maybe what I really want it to mean is: competent policies which are non-myopic in some sense. A truly myopic Q&A system doesn’t feel much like a controller / inner optimizer (even if it is misaligned, it’s not steering the world in a bad direction, because it’s totally myopic).
I’m not sure what sense of “myopia” I want to use, though.
To me it sounds like you’re describing (some version of) agency, and so the most natural term to use would be mesa-agent.
I’m a bit confused about the relationship between “optimiser” and “agent”, but I tend to think of the latter as more compressed, and so insofar as we’re talking about policies it seems like “agent” is appropriate. Also, mesa-optimiser is taken already (under a definition which assumes that optimisation is equivalent to some kind of internal search).
I’m not sure what you meant by “more compressed”.
I used to define “agent” as “both a searcher and a controller”, IE, something which uses an internal selection/search of some kind to accomplish an external control task. This might be too restrictive, though.
Oh, I really like this definition. Even if it’s too restrictive, it seems like it gets at something important.
Sorry, that was quite opaque. I guess what I mean is that evolution is an optimiser but isn’t an agent, and in part this has to do with how it’s a very distributed process with no clear boundary around it. Whereas when you have the same problem being solved in a single human brain, then that compression makes it easier to point to the human as being an agent separate from its environment.
The rest of this comment is me thinking out loud in a somewhat incoherent way; no pressure to read/respond.
It seems like calling something a “searcher” describes only a very simple interface: at the end of the search, there needs to be some representation of the output which it has found. But that output may be very complex.
Whereas calling something a “controller” describes a much more complex interface between it and its environment: you need to be able to point not just to outcomes, but also to observations and actions. But each of those actions is usually fairly simple for a pure controller; if it’s complex, then you need search to find which action to take at each step.
Now, it seems useful to sometimes call evolution a controller. For example, suppose you’re trying to wipe out a virus, but it keeps mutating. Then there’s a straightforward sense in which evolution is “steering” the world towards states where the virus still exists, in the short term. You could also say that it’s steering the world towards states where all organisms have high fitness in the long term, but organisms are so complex that it’s easier to treat them as selected outcomes, and abstract away from the many “actions” by evolution which led to this point.
In other words, evolution searches using a process of iterative control. Whereas humans control using a process of iterative search.
(As a side note, I’m now thinking that “search” isn’t quite the right word, because there are other ways to do selection than search. For example, if I construct a mathematical proof (or a poem) by writing it one line at a time, letting my intuition guide me, then it doesn’t really seem accurate to say that I’m searching over the space of proofs/poems. Similarly, a chain of reasoning may not branch much, but still end up finding a highly specific conclusion. Yet “selection” also doesn’t really seem like the right word either, because it’s at odds with normal usage, which involves choosing from a preexisting set of options—e.g. you wouldn’t say that a poet is “selecting” a poem. How about “design” as an alternative? Which allows us to be agnostic about how the design occurred—whether it be via a control process like evolution, or a process of search, or a process of reasoning.)