Optimization is an intuitively recognizable phenomenon where a system’s state gets steered into a smaller region of the state space. (An AGI will be a powerful optimizer.) What exactly do we mean by that? Can we find a formal definition that satisfactorily addresses the key examples?
In your list of research problems, there seems to be a lot of focus on optimisation. I want to point out that it seems to me that the interest in this concept peaked around 2019-2020 (“Risks from Learned Optimization in Advanced Machine Learning Systems”, Hubinger et al. 2019) and since that time the concept was falling out of fashion, which means that researchers don’t find it very useful or consequential. E.g., any LLM is an optimiser on many different levels: in-context learning, fine-tuning, pre-training, and even LLM-environment interactions are examples of processes which we can call optimisation[1], but there is not much of useful conclusions that we can draw from this.
Can optimizing systems be crisply taxonimized? (Perhaps we could identify more or less dangerous types of optimizing systems.)
Optimisers (which, according to the footnote[1], is just any identifiable system) in general can be taxonomised (e.g., Friston et al. (2022) suggested one such taxonomisation: inert, active, conservative, and strange particles aka agents), but crisp differences lie only in the domain of very simple systems. Any intelligent system of any real interest will be of the “highest” crisp type according to any such crisp taxonomy (strange particle aka agent, in Friston et al.’s classification). Furthermore, any intelligent systems of real interest will be complex (as I also pointed out in this comment), and thus their classification couldn’t be crisp, it will necessarily be nebulous.
(AI) psychology is where these taxonomisations are considered, but this is not the domain of agent foundations. E.g., Perez et al. estimate all psychological qualities (self-awareness, deceptiveness, etc.) in degrees not because they are not smart enough to come up with crisp criteria for these qualities and behaviours, but because this is impossible on this level of complexity.
Moreover, any environmentally identifiable system, even a rock or a molecule, could be seen as an optimiser in the course of its existence. “Inert particles” in the language of Friston et al. (2022) are optimising (inferring) their state.
In your list of research problems, there seems to be a lot of focus on optimisation. I want to point out that it seems to me that the interest in this concept peaked around 2019-2020 (“Risks from Learned Optimization in Advanced Machine Learning Systems”, Hubinger et al. 2019) and since that time the concept was falling out of fashion, which means that researchers don’t find it very useful or consequential. E.g., any LLM is an optimiser on many different levels: in-context learning, fine-tuning, pre-training, and even LLM-environment interactions are examples of processes which we can call optimisation[1], but there is not much of useful conclusions that we can draw from this.
Optimisers (which, according to the footnote[1], is just any identifiable system) in general can be taxonomised (e.g., Friston et al. (2022) suggested one such taxonomisation: inert, active, conservative, and strange particles aka agents), but crisp differences lie only in the domain of very simple systems. Any intelligent system of any real interest will be of the “highest” crisp type according to any such crisp taxonomy (strange particle aka agent, in Friston et al.’s classification). Furthermore, any intelligent systems of real interest will be complex (as I also pointed out in this comment), and thus their classification couldn’t be crisp, it will necessarily be nebulous.
(AI) psychology is where these taxonomisations are considered, but this is not the domain of agent foundations. E.g., Perez et al. estimate all psychological qualities (self-awareness, deceptiveness, etc.) in degrees not because they are not smart enough to come up with crisp criteria for these qualities and behaviours, but because this is impossible on this level of complexity.
Moreover, any environmentally identifiable system, even a rock or a molecule, could be seen as an optimiser in the course of its existence. “Inert particles” in the language of Friston et al. (2022) are optimising (inferring) their state.