The goal is a fixed part of the AI’s structure. The internal dynamics of the AI, if left to their own devices, will never modify the goal.
The “outermost loop” of the AI’s internal dynamics is an optimization process aimed at the goal, or at least the AI behaves just as though this were true.
This “outermost loop” or “fixed-terminal-goal-directed wrapper” chooses which of the AI’s specific capabilities to deploy at any given time, and how to deploy it[2].
The AI’s capabilities will themselves involve optimization for sub-goals that are not the same as the goal, and they will optimize for them very powerfully (hence “capabilities”). But it is “not enough” that the AI merely be good at optimization-for-subgoals: it will also have a fixed-terminal-goal-directed wrapper.
So, the AI may be very good at playing chess, and when it is playing chess, it may be running an internal routine that optimizes for winning chess. This routine, and not the terminal-goal-directed wrapper around it, explains the AI’s strong chess performance. (“Maximize paperclips” does not tell you how to win at chess.)
The AI may also be good at things that are much more general than chess, such as “planning,” “devising proofs in arbitrary formal systems,” “inferring human mental states,” or “coming up with parsimonious hypotheses to explain observations.” All of these are capacities[3] to optimize for a particular subgoal that is not the AI’s terminal goal.
Although these subgoal-directed capabilities, and not the fixed-terminal-goal-directed wrapper, will constitute the reason the AI does well at anything it does well at, the AI must still have a fixed-terminal-goal-directed wrapper around them and apart from them.
There is no way for the terminal goal to change through bottom-up feedback from anything inside the wrapper. The hierarchy of control is strict and only goes one way.
My question: why assume all this? Most pressingly, why assume that the terminal goal is fixed, with no internal dynamics capable of updating it?
I often see the rapid capability gains of humans over other apes cited as a prototype case for the rapid capability gains we expect in AGI. But humans do not have this wrapper structure! Our goals often change over time. (And we often permit or even welcome this, whereas an optimizing wrapper would try to prevent its goal from changing.)
Having the wrapper structure was evidently not necessary for our rapid capability gains. Nor do I see reason to think that our capabilities result from us being “more structured like this” than other apes. (Or to think that we are “more structured like this” than other apes in this first place.)
Our capabilities seem more like the subgoal capabilities discussed above: general and powerful tools, which can be “plugged in” to many different (sub)goals, and which do not require the piloting of a wrapper with a fixed goal to “work” properly.
Why expect the “wrapper” structure with fixed goals to emerge from an outer optimization process? Are there any relevant examples of this happening via natural selection, or via gradient descent?
There are many, many posts on LW / AF/ Arbital about “optimization,” its relation to intelligence, whether we should view AGIs as “optimizers” and in what senses, etc. I have not read all of it. Most of it touches only lightly, if at all, on my question. For example:
There has been much discussion over whether an AGI would inevitably have (close to) consistent preferences, or would self-modify itself to have closer-to-consistent preferences. See e.g. here, here, here, here. Every post I’ve read on this topic implicitly assumes that the preferences are fixed in time.
Mesa-optimizers have been discussed extensively. The same bundle of assumptions is made about mesa-optimizers.
It has been argued that if you already have the fixed-terminal-goal-directed wrapper structure, then you will prefer to avoid outside influences that will modify your goal. This is true, but does not explain why the structure would emerge in the first place.
There are arguments (e.g.) that we should heuristically imagine a superintelligence as a powerful optimizer, to get ourselves to predict that it will not do things we know are suboptimal. These arguments tell us to imagine the AGI picking actions that are optimal for a goal iff it is currently optimizing for that goal. They don’t tell us when it will be optimizing for which goals.
EY’s notion of “consequentialism” seems closely related to this set of assumptions. But, I can’t extract an answer from the writing I’ve read on that topic.
EY seems to attribute what I’ve called the powerful “subgoal capabilities” of humans/AGI to a property called “cross-domain consequentialism”:
We can see one of the critical aspects of human intelligence as cross-domain consequentialism. Rather than only forecasting consequences within the boundaries of a narrow domain, we can trace chains of events that leap from one domain to another. Making a chess move wins a chess game that wins a chess tournament that wins prize money that can be used to rent a car that can drive to the supermarket to get milk. An Artificial General Intelligence that could learn many domains, and engage in consequentialist reasoning that leaped across those domains, would be a sufficiently advanced agent to be interesting from most perspectives on interestingness. It would start to be a consequentialist about the real world.
while defining “consequentialism” as the ability to do means-end reasoning with some preference ordering:
Whenever we reason that an agent which prefers outcome Y over Y’ will therefore do X instead of X’ we’re implicitly assuming that the agent has the cognitive ability to do consequentialism at least about Xs and Ys. It does means-end reasoning; it selects means on the basis of their predicted ends plus a preference over ends.
But the ability to use this kind of reasoning, and do so across domains, does not imply that one’s “outermost loop” looks like this kind of reasoning applied to the whole world at once.
I myself am a cross-domain consequentialist—a human—with very general capacities to reason and plan that I deploy across many different facets of my life. But I’m not running an outermost loop with a fixed goal that pilots around all of my reasoning-and-planning activities. Why can’t AGI be like me?
EDIT to spell out the reason I care about the answer: agents with the “wrapper structure” are inevitably hard to align, in ways that agents without it might not be. An AGI “like me” might be morally uncertain like I am, persuadable through dialogue like I am, etc.
It’s very important to know what kind of AIs would or would not have the wrapper structure, because this makes the difference between “inevitable world-ending nightmare” and “we’re not the dominant species anymore.” The latter would be pretty bad for us too, but there’s a difference!
Often people speak of the AI’s “utility function” or “preference ordering” rather than its “goal.”
For my purposes here, these terms are more or less equivalent: it doesn’t matter whether you think an AGI must have consistent preferences, only whether you think it must have fixed preferences.
Or possibly one big capacity—“general reasoning” or what have you—which contains the others as special cases. I’m not taking a position on how modular the capabilities will be.
[Question] why assume AGIs will optimize for fixed goals?
When I read posts about AI alignment on LW / AF/ Arbital, I almost always find a particular bundle of assumptions taken for granted:
An AGI has a single terminal goal[1].
The goal is a fixed part of the AI’s structure. The internal dynamics of the AI, if left to their own devices, will never modify the goal.
The “outermost loop” of the AI’s internal dynamics is an optimization process aimed at the goal, or at least the AI behaves just as though this were true.
This “outermost loop” or “fixed-terminal-goal-directed wrapper” chooses which of the AI’s specific capabilities to deploy at any given time, and how to deploy it[2].
The AI’s capabilities will themselves involve optimization for sub-goals that are not the same as the goal, and they will optimize for them very powerfully (hence “capabilities”). But it is “not enough” that the AI merely be good at optimization-for-subgoals: it will also have a fixed-terminal-goal-directed wrapper.
So, the AI may be very good at playing chess, and when it is playing chess, it may be running an internal routine that optimizes for winning chess. This routine, and not the terminal-goal-directed wrapper around it, explains the AI’s strong chess performance. (“Maximize paperclips” does not tell you how to win at chess.)
The AI may also be good at things that are much more general than chess, such as “planning,” “devising proofs in arbitrary formal systems,” “inferring human mental states,” or “coming up with parsimonious hypotheses to explain observations.” All of these are capacities[3] to optimize for a particular subgoal that is not the AI’s terminal goal.
Although these subgoal-directed capabilities, and not the fixed-terminal-goal-directed wrapper, will constitute the reason the AI does well at anything it does well at, the AI must still have a fixed-terminal-goal-directed wrapper around them and apart from them.
There is no way for the terminal goal to change through bottom-up feedback from anything inside the wrapper. The hierarchy of control is strict and only goes one way.
My question: why assume all this? Most pressingly, why assume that the terminal goal is fixed, with no internal dynamics capable of updating it?
I often see the rapid capability gains of humans over other apes cited as a prototype case for the rapid capability gains we expect in AGI. But humans do not have this wrapper structure! Our goals often change over time. (And we often permit or even welcome this, whereas an optimizing wrapper would try to prevent its goal from changing.)
Having the wrapper structure was evidently not necessary for our rapid capability gains. Nor do I see reason to think that our capabilities result from us being “more structured like this” than other apes. (Or to think that we are “more structured like this” than other apes in this first place.)
Our capabilities seem more like the subgoal capabilities discussed above: general and powerful tools, which can be “plugged in” to many different (sub)goals, and which do not require the piloting of a wrapper with a fixed goal to “work” properly.
Why expect the “wrapper” structure with fixed goals to emerge from an outer optimization process? Are there any relevant examples of this happening via natural selection, or via gradient descent?
There are many, many posts on LW / AF/ Arbital about “optimization,” its relation to intelligence, whether we should view AGIs as “optimizers” and in what senses, etc. I have not read all of it. Most of it touches only lightly, if at all, on my question. For example:
There has been much discussion over whether an AGI would inevitably have (close to) consistent preferences, or would self-modify itself to have closer-to-consistent preferences. See e.g. here, here, here, here. Every post I’ve read on this topic implicitly assumes that the preferences are fixed in time.
Mesa-optimizers have been discussed extensively. The same bundle of assumptions is made about mesa-optimizers.
It has been argued that if you already have the fixed-terminal-goal-directed wrapper structure, then you will prefer to avoid outside influences that will modify your goal. This is true, but does not explain why the structure would emerge in the first place.
There are arguments (e.g.) that we should heuristically imagine a superintelligence as a powerful optimizer, to get ourselves to predict that it will not do things we know are suboptimal. These arguments tell us to imagine the AGI picking actions that are optimal for a goal iff it is currently optimizing for that goal. They don’t tell us when it will be optimizing for which goals.
EY’s notion of “consequentialism” seems closely related to this set of assumptions. But, I can’t extract an answer from the writing I’ve read on that topic.
EY seems to attribute what I’ve called the powerful “subgoal capabilities” of humans/AGI to a property called “cross-domain consequentialism”:
while defining “consequentialism” as the ability to do means-end reasoning with some preference ordering:
But the ability to use this kind of reasoning, and do so across domains, does not imply that one’s “outermost loop” looks like this kind of reasoning applied to the whole world at once.
I myself am a cross-domain consequentialist—a human—with very general capacities to reason and plan that I deploy across many different facets of my life. But I’m not running an outermost loop with a fixed goal that pilots around all of my reasoning-and-planning activities. Why can’t AGI be like me?
EDIT to spell out the reason I care about the answer: agents with the “wrapper structure” are inevitably hard to align, in ways that agents without it might not be. An AGI “like me” might be morally uncertain like I am, persuadable through dialogue like I am, etc.
It’s very important to know what kind of AIs would or would not have the wrapper structure, because this makes the difference between “inevitable world-ending nightmare” and “we’re not the dominant species anymore.” The latter would be pretty bad for us too, but there’s a difference!
Often people speak of the AI’s “utility function” or “preference ordering” rather than its “goal.”
For my purposes here, these terms are more or less equivalent: it doesn’t matter whether you think an AGI must have consistent preferences, only whether you think it must have fixed preferences.
...or at least the AI behaves just as though this were true. I’ll stop including this caveat after this.
Or possibly one big capacity—“general reasoning” or what have you—which contains the others as special cases. I’m not taking a position on how modular the capabilities will be.