Hi! Long time lurker, first time commenter. You have written a great piece here. This is a topic that has fascinated me for a while and I appreciate what you’ve laid out.
I’m wondering if there’s a base assumption on the whole intelligence vs values/beliefs/goals question that needs to be questioned.
sufficiently complex goals may only be well-represented by sufficiently intelligent agents
This statement points to my question. There’s necessarily a positive correlation between internal complexity and intelligence right?. So, in order for intelligence to increase, internal complexity must also increase. My understanding is that complexity is a characteristic of dynamic and generative phenomena, and not of purely mechanical phenomena. So, what do we have to assume in order to posit a super-intelligent entity exists? It must have maintained its entity-ness over time in order to have increased its intelligence/complexity to its current level.
Has anyone explored what it takes for an agent to complexify? I would presume that for an agent to simultaneously continue existing and complexify it must stay maintain some type of fixpoint/set of autopoietic (self-maintenance, self-propagation) values/beliefs/goals throughout its dynamic evolution. If this were the case, wouldn’t it be true that there must exist a set of values/beliefs/goals that are intrinsic to the agent’s ability to complexify? Therefore there must be another set of values/beliefs/goals that are incompatible with self-complexification. If so, can we not put boundary conditions on what values/beliefs/goals are both necessary as well as incompatible with sufficiently intelligent, self-complexifying agents? After all, if we observe a complex agent, the probability of it arising full-cloth and path-independently is vanishingly small, so it is safe to say that the observed entity has evolved to reach the observed state.
I don’t think my observation is incompatible with your argument, but might place further limits on what relationships we can possibly see between entities of sufficient intelligence and their goals/values/beliefs than the limits you propose.
I think situations like a paperclip maximizer may still occur but they are degenerate cases where an evolutionarily fit entity spawns something that inherits much of its intrinsic complexity but loses its autopoietic fixpoint. Such systems do occur in nature, but to get that system, you must also assume a more-complex (and hopefully more intelligent/adapted) entity exists as well. This other entity would likely place adversarial pressure on the degenerate paperclip maximizer as it threatens its continued existence.
Some relationships/overlaps with your arguments are as follows:
totally agree with the belief/value duality
Naive belief/value factorizations lead to optimization daemons. The optimization daemons observation points to an agent’s inability to maintain autopoiesis over time, implying misalignment of its values/beliefs/goals with its desire to increase its intelligence
Intelligence changes the ontology values are expressed in. I presume that any ontology expressed by an autopoietic embedded agent must maintain concepts of self, otherwise the embedded entity cannot continue to complexify over time, therefore there must be some fix point in ontological evolution that preserves the evolutionary drive of the entity in order for it to continue to advance its intelligence
Not sure what you mean by complexity here, is this like code size / Kolmogorov complexity? You need some of that to have intelligence at all (the empty program is not intelligent). At some point most of your gains come from compute rather than code size. Though code size can speed things up (e.g. imagine sending a book back to 1000BC, that would speed people up a lot; consider that superintelligence sending us a book would be a bigger speedup)
by “complexify” here it seems you mean something like “develop extended functional organization”, e.g. in brain development throughout evolution. And yeah, that involves dynamics with the environment and internal maintenance (evolution gets feedback from the environment). It seems it has to have a drive to do this which can either be a terminal or instrumental goal, though deriving it from instrumentals seems harder than baking it is as terminal (so I would guess evolution gives animals a terminal goal of developing functional complexity of mental structures etc, or some other drive that isn’t exactly a terminal goal)
see also my post relating optimization daemons to immune systems, it seems evolved organisms develop these; when having more extended functional organization, they protect it with some immune system functional organization.
to be competitive agents, having a “self” seems basically helpful, but might not be the best solution; selfish genes are an alternative, and perhaps extended notions of self can maintain competitiveness.
Hi! Long time lurker, first time commenter. You have written a great piece here. This is a topic that has fascinated me for a while and I appreciate what you’ve laid out.
I’m wondering if there’s a base assumption on the whole intelligence vs values/beliefs/goals question that needs to be questioned.
This statement points to my question. There’s necessarily a positive correlation between internal complexity and intelligence right?. So, in order for intelligence to increase, internal complexity must also increase. My understanding is that complexity is a characteristic of dynamic and generative phenomena, and not of purely mechanical phenomena. So, what do we have to assume in order to posit a super-intelligent entity exists? It must have maintained its entity-ness over time in order to have increased its intelligence/complexity to its current level.
Has anyone explored what it takes for an agent to complexify? I would presume that for an agent to simultaneously continue existing and complexify it must stay maintain some type of fixpoint/set of autopoietic (self-maintenance, self-propagation) values/beliefs/goals throughout its dynamic evolution. If this were the case, wouldn’t it be true that there must exist a set of values/beliefs/goals that are intrinsic to the agent’s ability to complexify? Therefore there must be another set of values/beliefs/goals that are incompatible with self-complexification. If so, can we not put boundary conditions on what values/beliefs/goals are both necessary as well as incompatible with sufficiently intelligent, self-complexifying agents? After all, if we observe a complex agent, the probability of it arising full-cloth and path-independently is vanishingly small, so it is safe to say that the observed entity has evolved to reach the observed state.
I don’t think my observation is incompatible with your argument, but might place further limits on what relationships we can possibly see between entities of sufficient intelligence and their goals/values/beliefs than the limits you propose.
I think situations like a paperclip maximizer may still occur but they are degenerate cases where an evolutionarily fit entity spawns something that inherits much of its intrinsic complexity but loses its autopoietic fixpoint. Such systems do occur in nature, but to get that system, you must also assume a more-complex (and hopefully more intelligent/adapted) entity exists as well. This other entity would likely place adversarial pressure on the degenerate paperclip maximizer as it threatens its continued existence.
Some relationships/overlaps with your arguments are as follows:
totally agree with the belief/value duality
Naive belief/value factorizations lead to optimization daemons. The optimization daemons observation points to an agent’s inability to maintain autopoiesis over time, implying misalignment of its values/beliefs/goals with its desire to increase its intelligence
Intelligence changes the ontology values are expressed in. I presume that any ontology expressed by an autopoietic embedded agent must maintain concepts of self, otherwise the embedded entity cannot continue to complexify over time, therefore there must be some fix point in ontological evolution that preserves the evolutionary drive of the entity in order for it to continue to advance its intelligence
Anyways, thank you for the essay.
Not sure what you mean by complexity here, is this like code size / Kolmogorov complexity? You need some of that to have intelligence at all (the empty program is not intelligent). At some point most of your gains come from compute rather than code size. Though code size can speed things up (e.g. imagine sending a book back to 1000BC, that would speed people up a lot; consider that superintelligence sending us a book would be a bigger speedup)
by “complexify” here it seems you mean something like “develop extended functional organization”, e.g. in brain development throughout evolution. And yeah, that involves dynamics with the environment and internal maintenance (evolution gets feedback from the environment). It seems it has to have a drive to do this which can either be a terminal or instrumental goal, though deriving it from instrumentals seems harder than baking it is as terminal (so I would guess evolution gives animals a terminal goal of developing functional complexity of mental structures etc, or some other drive that isn’t exactly a terminal goal)
see also my post relating optimization daemons to immune systems, it seems evolved organisms develop these; when having more extended functional organization, they protect it with some immune system functional organization.
to be competitive agents, having a “self” seems basically helpful, but might not be the best solution; selfish genes are an alternative, and perhaps extended notions of self can maintain competitiveness.