The first sense of “optimizer” is an optimization algorithm, that given some formally specified problem computes the solution to that problem, e.g. a SAT solver or linear program solver. The second sense is an algorithm that acts upon its environment to change it. Joar believes that people often conflate the two in AI safety.
Planned opinion:
I agree that this is an important distinction to keep in mind. It seems to me that the distinction is whether the optimizer has knowledge about the environment: in canonical examples of the first kind of optimizer, it does not. If we somehow encoded the dynamics of the world as a SAT formula and asked a super-powerful SAT solver to solve for the actions that accomplish some goal, it would look like the second kind of optimizer.
If the super-powerful SAT solver thing finds the plans but doesn’t execute them, would you still lump it with optimizer_2? (I know it’s just terminology and there’s no right answer, but I’m just curious about what categories you find natural.)
(BTW this is more-or-less a description of my current Grand Vision For AGI Safety, where the “dynamics of the world” are discovered by self-supervised learning, and the search process (and much else) is TBD.)
Hmm, idk, it feels more like an optimizer_1 in that situation. Now that you’ve posed this question, the super-powerful SAT solver that acts in the world feels like both an optimizer_1 and an optimizer_2.
It seems to me that the distinction is whether the optimizer has knowledge about the environment
Alternatively, you could say the distinction is whether the optimizer cares about the environment. I think there’s a sense (or senses?) in which these things can be made/considered equivalent. I don’t feel like I totally understand or am satisfied with either way of thinking about it, though.
Planned summary:
The first sense of “optimizer” is an optimization algorithm, that given some formally specified problem computes the solution to that problem, e.g. a SAT solver or linear program solver. The second sense is an algorithm that acts upon its environment to change it. Joar believes that people often conflate the two in AI safety.
Planned opinion:
I agree that this is an important distinction to keep in mind. It seems to me that the distinction is whether the optimizer has knowledge about the environment: in canonical examples of the first kind of optimizer, it does not. If we somehow encoded the dynamics of the world as a SAT formula and asked a super-powerful SAT solver to solve for the actions that accomplish some goal, it would look like the second kind of optimizer.
If the super-powerful SAT solver thing finds the plans but doesn’t execute them, would you still lump it with optimizer_2? (I know it’s just terminology and there’s no right answer, but I’m just curious about what categories you find natural.)
(BTW this is more-or-less a description of my current Grand Vision For AGI Safety, where the “dynamics of the world” are discovered by self-supervised learning, and the search process (and much else) is TBD.)
Hmm, idk, it feels more like an optimizer_1 in that situation. Now that you’ve posed this question, the super-powerful SAT solver that acts in the world feels like both an optimizer_1 and an optimizer_2.
Alternatively, you could say the distinction is whether the optimizer cares about the environment. I think there’s a sense (or senses?) in which these things can be made/considered equivalent. I don’t feel like I totally understand or am satisfied with either way of thinking about it, though.