I agree that it’s possible to get myopic models in the population after arbitrarily long runtime due to mutations. It seems less likely the more bits that need to change—in any model in the current population—to get a model that in completely myopic.
Yeah, I agree there’s something to think about here. The reason I responded as I did was because it seems more tractable to think about asymptotic results, ie, what happens if you run an algorithm to convergence. But ultimately we also need to think about what an algorithm does with realistic resources.
From a safety perspective, if the prospect of some learning algorithm yielding a non-myopic model is concerning, the prospect of it creating non-myopic models along the way is plausibly also concerning.
I’m not assuming partial agency is uniformly good or bad from a safety perspective; in some cases one, in some cases the other. If I want an algorithm to reliably produce full agency, I can’t use algorithms which produce some full-agency effects in the short term but which have a tendency to wash them out in the long term, unless I have a very clear understanding of when I’m in the short term.
In this example, if we train an environment model on the data collected so far (ignoring the “you want to learn on-line” part),
Ignoring the on-line part ignores too much, I think.
Also, people might experiment with evolutionary algorithms as an alternative to RL, for environments that can be simulated,
Similarly, assuming environments that can be simulated seems to assume too much. A simulatable environment gives us the ability to perfectly optimize from outside of that environment, allowing full optimization and hence full agency. So it’s not solving the problem I was talking about.
I think this is an example of selection/control confusion. Evolution is a selection algorithm, and can only be applied indirectly to control problems.
Yeah, I agree there’s something to think about here. The reason I responded as I did was because it seems more tractable to think about asymptotic results, ie, what happens if you run an algorithm to convergence. But ultimately we also need to think about what an algorithm does with realistic resources.
I’m not assuming partial agency is uniformly good or bad from a safety perspective; in some cases one, in some cases the other. If I want an algorithm to reliably produce full agency, I can’t use algorithms which produce some full-agency effects in the short term but which have a tendency to wash them out in the long term, unless I have a very clear understanding of when I’m in the short term.
Ignoring the on-line part ignores too much, I think.
Similarly, assuming environments that can be simulated seems to assume too much. A simulatable environment gives us the ability to perfectly optimize from outside of that environment, allowing full optimization and hence full agency. So it’s not solving the problem I was talking about.
I think this is an example of selection/control confusion. Evolution is a selection algorithm, and can only be applied indirectly to control problems.