I’ve heard people argue that “most” utility functions lead to agents with strong convergent instrumental goals. This obviously depends a lot on how you quantify over utility functions. Here’s one intuition in the other direction. I don’t expect this to be persuasive to most people who make the argument above (but I’d still be interested in hearing why not).
If a non-negligible percentage of an agent’s actions are random, then to describe it as a utility-maximiser would require an incredibly complex utility function (because any simple hypothesised utility function will eventually be falsified by a random action). And so this generates arbitrarily simple agents whose observed behaviour can only be described as maximising a utility function for arbitrarily complex utility functions (depending on how long you run them).
I expect people to respond something like: we need a theory of how to describe agents with bounded cognition anyway. And if you have such a theory, then we could describe the agent above as “maximising simple function U, subject to the boundedness constraint that X% of its actions are random”.
I’m not sure if you consider me to be making that argument, but here are my thoughts: I claim that most reward functions lead to agents with strong convergent instrumental goals. However, I share your intuition that (somehow) uniformly sampling utility functions over universe-histories might not lead to instrumental convergence.
To understand instrumental convergence and power-seeking, consider how many reward functions we might specify automatically imply a causal mechanism for increasing reward. The structure of the reward function implies that more is better, and that there are mechanisms for repeatedly earning points (for example, by showing itself a high-scoring input).
Since the reward function is “simple” (there’s usually not a way to grade exact universe histories), these mechanisms work in many different situations and points in time. It’s naturally incentivized to assure its own safety in order to best leverage these mechanisms for gaining reward. Therefore, we shouldn’t be surprised to see a lot of these simple goals leading to the same kind of power-seeking behavior.
What structure is implied by a reward function?
Additive/Markovian: while a utility function might be over an entire universe-history, reward is often additive over time steps. This is a strong constraint which I don’t always expect to be true, but i think that among the goals with this structure, a greater proportion of them have power-seeking incentives.
Observation-based: while a utility function might be over an entire universe-history, the atom of the reward function is the observation. Perhaps the observation is an input to update a world model, over which we have tried to define a reward function. I think that most ways of doing this lead to power-seeking incentives.
Agent-centric: reward functions are defined with respect to what the agent can observe. Therefore, in partially observable environments, there is naturally a greater emphasis on the agent’s vantage point in the environment.
My theorems apply to the finite, fully observable, Markovian situation.[1] We might not end up using reward functions for more impressive tasks – we might express preferences over incomplete trajectories, for example. The “specify a reward function over the agent’s world model” approach may or may not lead to good subhuman performance in complicated tasks like cleaning warehouses. Imagine specifying a reward function over pure observations for that task – the agent would probably just get stuck looking at a wall in a particularly high-scoring way.
However, for arbitrary utility functions over universe histories, the structure isn’t so simple. With utility functions over universe histories having far more degrees of freedom, arbitrary policies can be rationalized as VNM expected utility maximization. That said, with respect to a simplicity prior over computable utility functions, the power-seeking ones might have most of the measure.
I claim that most reward functions lead to agents with strong convergent instrumental goals
I think this depends a lot on how you model the agent developing. If you start off with a highly intelligent agent which has the ability to make long-term plans, but doesn’t yet have any goals, and then you train it on a random reward function—then yes, it probably will develop strong convergent instrumental goals.
On the other hand, if you start off with a randomly initialised neural network, and then train it on a random reward function, then probably it will get stuck in a local optimum pretty quickly, and never learn to even conceptualise these things called “goals”.
I claim that when people think about reward functions, they think too much about the former case, and not enough about the latter. Because while it’s true that we’re eventually going to get highly intelligent agents which can make long-term plans, it’s also important that we get to control what reward functions they’re trained on up to that point. And so plausibly we can develop intelligent agents that, in some respects, are still stuck in “local optima” in the way they think about convergent instrumental goals—i.e. they’re missing whatever cognitive functionality is required for being ambitious on a large scale.
Agreed – I should have clarified. I’ve been mostly discussing instrumental convergence with respect to optimal policies. The path through policy space is also important.
Makes sense. For what it’s worth, I’d also argue that thinking about optimal policies at all is misguided (e.g. what’s the optimal policy for humans—the literal best arrangement of neurons we could possibly have for our reproductive fitness? Probably we’d be born knowing arbitrarily large amounts of information. But this is just not relevant to predicting or modifying our actual behaviour at all).
(I now think that you were very right in saying “thinking about optimal policies at all is misguided”, and I was very wrong to disagree. I’ve thought several times about this exchange. Not listening to you about this point was a serious error and made my work way less impactful. I do think that the power-seeking theorems say interesting things, but about eg internal utility functions over an internal planning ontology—not about optimal policies for a reward function.)
We do in fact often train agents using algorithms which are proven to eventually converge to the optimal policy.[1] Even if we don’t expect the trained agents to reach the optimal policy in the real world, we should still understand what behavior is like at optimum. If you think your proposal is not aligned at optimum but is aligned for realistic training paths, you should have a strong story for why.
Formal theorizing about instrumental convergence with respect to optimal behavior is strictly easier than theorizing about ϵ-optimal behavior, which I think is what you want for a more realistic treatment of instrumental convergence for real agents. Even if you want to think about sub-optimal policies, if you don’t understand optimal policies… good luck! Therefore, we also have an instrumental (...) interest in studying the behavior at optimum.
At least, the tabular algorithms are proven, but no one uses those for real stuff. I’m not sure what the results are for function approximators, but I think you get my point.
1. I think it’s more accurate to say that, because approximately none of the non-trivial theoretical results hold for function approximation, approximately none of our non-trivial agents are proven to eventually converge to the optimal policy. (Also, given the choice between an algorithm without convergence proofs that works in practice, and an algorithm with convergence proofs that doesn’t work in practice, everyone will use the former). But we shouldn’t pay any attention to optimal policies anyway, because the optimal policy in an environment anything like the real world is absurdly, impossibly complex, and requires infinite compute.
2. I think theorizing about ϵ-optimal behavior is more useful than theorizing about optimal behaviour by roughly ϵ, for roughly the same reasons. But in general, clearly I can understand things about suboptimal policies without understanding optimal policies. I know almost nothing about the optimal policy in StarCraft, but I can still make useful claims about AlphaStar (for example: it’s not going to take over the world).
Again, let’s try cash this out. I give you a human—or, say, the emulation of a human, running in a simulation of the ancestral environment. Is this safe? How do you make it safer? What happens if you keep selecting for intelligence? I think that the theorising you talk about will be actively harmful for your ability to answer these questions.
I’m confused, because I don’t disagree with any specific point you make—just the conclusion. Here’s my attempt at a disagreement which feels analogous to me:
TurnTrout: here’s how spherical cows roll downhill!
ricraz: real cows aren’t spheres.
My response in this “debate” is: if you start with a spherical cow and then consider which real world differences are important enough to model, you’re better off than just saying “no one should think about spherical cows”.
I think that the theorising you talk about will be actively harmful for your ability to answer these questions.
I don’t understand why you think that. If you can have a good understanding of instrumental convergence and power-seeking for optimal agents, then you can consider whether any of those same reasons apply for suboptimal humans.
Considering power-seeking for optimal agents is a relaxed problem. Yes, ideally, we would instantly jump to the theory that formally describes power-seeking for suboptimal agents with realistic goals in all kinds of environments. But before you do that, a first step is understanding power-seeking in MDPs. Then, you can take formal insights from this first step and use them to update your pre-theoretic intuitions where appropriate.
Thanks for engaging despite the opacity of the disagreement. I’ll try to make my position here much more explicit (and apologies if that makes it sound brusque). The fact that your model is a simplified abstract model is not sufficient to make it useful. Some abstract models are useful. Some are misleading and will cause people who spend time studying them to understand the underlying phenomenon less well than they did before. From my perspective, I haven’t seen you give arguments that your models are in the former category not the latter. Presumably you think they are in fact useful abstractions—why? (A few examples of the latter: behaviourism, statistical learning theory, recapitulation theory, Gettier-style analysis of knowledge).
My argument for why they’re overall misleading: when I say that “the optimal policy in an environment anything like the real world is absurdly, impossibly complex, and requires infinite compute”, or that safety researchers shouldn’t think about AIXI, I’m not just saying that these are inaccurate models. I’m saying that they are modelling fundamentally different phenomena than the ones you’re trying to apply them to. AIXI is not “intelligence”, it is brute force search, which is a totally different thing that happens to look the same in the infinite limit. Optimal tabular policies are not skill at a task, they are a cheat sheet, but they happen to look similar in very simple cases.
Probably the best example of what I’m complaining about is Ned Block trying to use Blockhead to draw conclusions about intelligence. I think almost everyone around here would roll their eyes hard at that. But then people turn around and use abstractions that are just as unmoored from reality as Blockhead, often in a very analogous way. (This is less a specific criticism of you, TurnTrout, and more a general criticism of the field).
if you start with a spherical cow and then consider which real world differences are important enough to model, you’re better off than just saying “no one should think about spherical cows”.
Forgive me a little poetic license. The analogy in my mind is that you were trying to model the cow as a sphere, but you didn’t know how to do so without setting its weight as infinite, and what looked to you like your model predicting the cow would roll downhill was actually your model predicting that the cow would swallow up the nearby fabric of spacetime and the bottom of the hill would fall into its event horizon. At which point, yes, you would be better off just saying “nobody should think about spherical cows”.
Thanks for elaborating this interesting critique. I agree we generally need to be more critical of our abstractions.
I haven’t seen you give arguments that your models [of instrumental convergence] are [useful for realistic agents]
Falsifying claims and “breaking” proposals is a classic element of AI alignment discourse and debate. Since we’re talking about superintelligent agents, we can’t predict exactly what a proposal would do. However, if I make a claim (“a superintelligent paperclip maximizer would keep us around because of gains from trade”), you can falsify this by showing that my claimed policy is dominated by another class of policies (“we would likely be comically resource-inefficient in comparison; GFT arguments don’t model dynamics which allow killing other agents and appropriating their resources”).
Even we can come up with this dominant policy class, so the posited superintelligence wouldn’t miss it either. We don’t know what the superintelligent policy will be, but we know what it won’t be (see also Formalizing convergent instrumental goals). Even though I don’t know how Gary Kasparov will open the game, I confidently predict that he won’t let me checkmate him in two moves.
Non-optimal power and instrumental convergence
Instead of thinking about optimal policies, let’s consider the performance of a given algorithm A. A(M,R) takes a rewardless MDP M and a reward function R as input, and outputs a policy.
Definition. Let R be a continuous distribution over reward functions with CDF F. The average return achieved by algorithm A at state s and discount rate γ is
∫RVA(M,R)R(s,γ)dF(R).
Instrumental convergence with respect to A’s policies can be defined similarly (“what is the R-measure of a given trajectory under A?”). The theory I’ve laid out allows precise claims, which is a modest benefit to our understanding. Before, we just had intuitions about some vague concept called “instrumental convergence”.
Here’s bad reasoning, which implies that the cow tears a hole in spacetime:
Suppose the laws of physics bestow godhood upon an agent executing some convoluted series of actions; in particular, this allows avoiding heat death. Clearly, it is optimal for the vast majority of agents to instantly become god.
The problem is that it’s impractical to predict what a smarter agent will do, or what specific kinds of action will be instrumentally convergent for A, or that the real agent would be infinitely smart. Just because it’s smart doesn’t mean it’s omniscient, as you rightly point out.
Here’s better reasoning:
Suppose that the MDP modeling the real world represents shutdown as a single terminal state. Most optimal agents don’t allow themselves to be shut down. Furthermore, since we can see that most goals offer better reward at non-shutdown states, superintelligent A can as well.[1] While I don’t know exactly what Awill tend to do, I predict that policies generated by A will tend to resist shutdown.
It might seem like I’m assuming the consequent here. This is not so – the work is first done by the theorems on optimal behavior, which do imply that most goals achieve greater return by avoiding shutdown. The question is whether reasonably intelligent suboptimal agents realize this fact. Given a uniformly drawn reward function, we can usually come up with a better policy than dying, so the argument is that A can as well.
I’m afraid I’m mostly going to disengage here, since it seems more useful to spend the time writing up more general + constructive versions of my arguments, rather than critiquing a specific framework.
If I were to sketch out the reasons I expect to be skeptical about this framework if I looked into it in more detail, it’d be something like:
1. Instrumental convergence isn’t training-time behaviour, it’s test-time behaviour. It isn’t about increasing reward, it’s about achieving goals (that the agent learned by being trained to increase reward).
2. The space of goals that agents might learn is very different from the space of reward functions. As a hypothetical, maybe it’s the case that neural networks are just really good at producing deontological agents, and really bad at producing consequentialists. (E.g, if it’s just really really difficult for gradient descent to get a proper planning module working). Then agents trained on almost all reward functions will learn to do well on them without developing convergent instrumental goals. (I expect you to respond that being deontological won’t get you to optimality. But I would say that talking about “optimality” here ruins the abstraction, for reasons outlined in my previous comment).
I expect you to respond that being deontological won’t get you to optimality. But I would say that talking about “optimality” here ruins the abstraction, for reasons outlined in my previous comment
I was actually going to respond, “that’s a good point, but (IMO) a different concern than the one you initially raised”. I see you making two main critiques.
(paraphrased) ”A won’t produce optimal policies for the specified reward function [even assuming alignment generalization off of the training distribution], so your model isn’t useful” – I replied to this critique above.
“The space of goals that agents might learn is very different from the space of reward functions.” I agree this is an important part of the story. I think the reasonable takeaway is “current theorems on instrumental convergence help us understand what superintelligent Awon’t do, assuming no reward-result gap. Since we can’t assume alignment generalization, we should keep in mind how the inductive biases of gradient descent affect the eventual policy produced.”
I remain highly skeptical of the claim that applying this idealized theory of instrumental convergence worsens our ability to actually reason about it.
ETA: I read some information you privately messaged me, and i see why you might see the above two points as a single concern.
We do in fact often train agents using algorithms which are proven to eventually converge to the optimal policy.[1]
At least, the tabular algorithms are proven, but no one uses those for real stuff. I’m not sure what the results are for function approximators, but I think you get my point. ↩︎
Is the point that people try to use algorithms which they think will eventually converge to the optimal policy? (Assuming there is one.)
And so this generates arbitrarily simple agents whose observed behaviour can only be described as maximising a utility function for arbitrarily complex utility functions (depending on how long you run them).
I object to the claim that agents that act randomly can be made “arbitrarily simple”. Randomness is basically definitionally complicated!
Eh, this seems a bit nitpicky. It’s arbitrarily simple given a call to a randomness oracle, which in practice we can approximate pretty easily. And it’s “definitionally” easy to specify as well: “the function which, at each call, returns true with 50% likelihood and false otherwise.”
If you get an ‘external’ randomness oracle, then you could define the utility function pretty simply in terms of the outputs of the oracle.
If the agent has a pseudo-random number generator (PRNG) inside it, then I suppose I agree that you aren’t going to be able to give it a utility function that has the standard set of convergent instrumental goals, and PRNGs can be pretty short. (Well, some search algorithms are probably shorter, but I bet they have higher Kt complexity, which is probably a better measure for agents)
If a reasonable percentage of an agent’s actions are random, then to describe it as a utility-maximiser would require an incredibly complex utility function (because any simple hypothesised utility function will eventually be falsified by a random action).
I’d take a different tack here, actually; I think this depends on what the input to the utility function is. If we’re only allowed to look at ‘atomic reality’, or the raw actions the agent takes, then I think your analysis goes through, that we have a simple causal process generating the behavior but need a very complicated utility function to make a utility-maximizer that matches the behavior.
But if we’re allowed to decorate the atomic reality with notes like “this action was generated randomly”, then we can have a utility function that’s as simple as the generator, because it just counts up the presence of those notes. (It doesn’t seem to me like this decorator is meaningfully more complicated than the thing that gave us “agents taking actions” as a data source, so I don’t think I’m paying too much here.)
This can lead to a massive explosion in the number of possible utility functions (because there’s a tremendous number of possible decorators), but I think this matches the explosion that we got by considering agents that were the outputs of causal processes in the first place. That is, consider reasoning about python code that outputs actions in a simple game, where there are many more possible python programs than there are possible policies in the game.
So in general you can’t have utility functions that are as simple as the generator, right? E.g. the generator could be deontological. In which case your utility function would be complicated. Or it could be random, or it could choose actions by alphabetical order, or...
And so maybe you can have a little note for each of these. But now what it sounds like is: “I need my notes to be able to describe every possible cognitive algorithm that the agent could be running”. Which seems very very complicated.
I guess this is what you meant by the “tremendous number” of possible decorators. But if that’s what you need to do to keep talking about “utility functions”, then it just seems better to acknowledge that they’re broken as an abstraction.
E.g. in the case of python code, you wouldn’t do anything analogous to this. You would just try to reason about all the possible python programs directly. Similarly, I want to reason about all the cognitive algorithms directly.
I realized my grandparent comment is unclear here:
but need a very complicated utility function to make a utility-maximizer that matches the behavior.
This should have been “consequence-desirability-maximizer” or something, since the whole question is “does my utility function have to be defined in terms of consequences, or can it be defined in terms of arbitrary propositions?”. If I want to make the deontologist-approximating Innocent-Bot, I have a terrible time if I have to specify the consequences that correspond to the bot being innocent and the consequences that don’t, but if you let me say “Utility = 0 - badness of sins committed” then I’ve constructed a ‘simple’ deontologist. (At least, about as simple as the bot that says “take random actions that aren’t sins”, since both of them need to import the sins library.)
In general, I think it makes sense to not allow this sort of elaboration of what we mean by utility functions, since the behavior we want to point to is the backwards assignment of desirability to actions based on the desirability of their expected consequences, rather than the expectation of any arbitrary property.
---
Actually, I also realized something about your original comment which I don’t think I had the first time around; if by “some reasonable percentage of an agent’s actions are random” you mean something like “the agent does epsilon-exploration” or “the agent plays an optimal mixed strategy”, then I think it doesn’t at all require a complicated utility function to generate identical behavior. Like, in the rock-paper-scissors world, and with the simple function ‘utility = number of wins’, the expected utility maximizing move (against tough competition) is to throw randomly, and we won’t falsify the simple ‘utility = number of wins’ hypothesis by observing random actions.
Instead I read it as something like “some unreasonable percentage of an agent’s actions are random”, where the agent is performing some simple-to-calculate mixed strategy that is either suboptimal or only optimal by luck (when the optimal mixed strategy is the maxent strategy, for example), and matching the behavior with an expected utility maximizer is a challenge (because your target has to be not some fact about the environment, but some fact about the statistical properties of the actions taken by the agent).
---
I think this is where the original intuition becomes uncompelling. We care about utility-maximizers because they’re doing their backwards assignment, using their predictions of the future to guide their present actions to try to shift the future to be more like what they want it to be. We don’t necessarily care about imitators, or simple-to-write bots, or so on. And so if I read the original post as “the further a robot’s behavior is from optimal, the less likely it is to demonstrate convergent instrumental goals”, I say “yeah, sure, but I’m trying to build smart robots (or at least reasoning about what will happen if people try to).”
Instead I read it as something like “some unreasonable percentage of an agent’s actions are random”
This is in fact the intended reading, sorry for ambiguity. Will edit. But note that there are probably very few situations where exploring via actual randomness is best; there will almost always be some type of exploration which is more favourable. So I don’t think this helps.
We care about utility-maximizers because they’re doing their backwards assignment, using their predictions of the future to guide their present actions to try to shift the future to be more like what they want it to be.
To be pedantic: we care about “consequence-desirability-maximisers” (or in Rohin’s terminology, goal-directed agents) because they do backwards assignment. But I think the pedantry is important, because people substitute utility-maximisers for goal-directed agents, and then reason about those agents by thinking about utility functions, and that just seems incorrect.
And so if I read the original post as “the further a robot’s behavior is from optimal, the less likely it is to demonstrate convergent instrumental goals”
What do you mean by optimal here? The robot’s observed behaviour will be optimal for some utility function, no matter how long you run it.
To be pedantic: we care about “consequence-desirability-maximisers” (or in Rohin’s terminology, goal-directed agents) because they do backwards assignment.
Valid point.
But I think the pedantry is important, because people substitute utility-maximisers for goal-directed agents, and then reason about those agents by thinking about utility functions, and that just seems incorrect.
This also seems right. Like, my understanding of what’s going on here is we have:
‘central’ consequence-desirability-maximizers, where there’s a simple utility function that they’re trying to maximize according to the VNM axioms
‘general’ consequence-desirability-maximizers, where there’s a complicated utility function that they’re trying to maximize, which is selected because it imitates some other behavior
The first is a narrow class, and depending on how strict you are with ‘maximize’, quite possibly no physically real agents will fall into it. The second is a universal class, which instantiates the ‘trivial claim’ that everything is utility maximization.
Put another way, the first is what happens if you hold utility fixed / keep utility simple, and then examine what behavior follows; the second is what happens if you hold behavior fixed / keep behavior simple, and then examine what utility follows.
Distance from the first is what I mean by “the further a robot’s behavior is from optimal”; I want to say that I should have said something like “VNM-optimal” but actually I think it needs to be closer to “simple utility VNM-optimal.”
I think you’re basically right in calling out a bait-and-switch that sometimes happens, where anyone who wants to talk about the universality of expected utility maximization in the trivial ‘general’ sense can’t get it to do any work, because it should all add up to normality, and in normality there’s a meaningful distinction between people who sort of pursue fuzzy goals and ruthless utility maximizers.
I’ve heard people argue that “most” utility functions lead to agents with strong convergent instrumental goals. This obviously depends a lot on how you quantify over utility functions. Here’s one intuition in the other direction. I don’t expect this to be persuasive to most people who make the argument above (but I’d still be interested in hearing why not).
If a non-negligible percentage of an agent’s actions are random, then to describe it as a utility-maximiser would require an incredibly complex utility function (because any simple hypothesised utility function will eventually be falsified by a random action). And so this generates arbitrarily simple agents whose observed behaviour can only be described as maximising a utility function for arbitrarily complex utility functions (depending on how long you run them).
I expect people to respond something like: we need a theory of how to describe agents with bounded cognition anyway. And if you have such a theory, then we could describe the agent above as “maximising simple function U, subject to the boundedness constraint that X% of its actions are random”.
I’m not sure if you consider me to be making that argument, but here are my thoughts: I claim that most reward functions lead to agents with strong convergent instrumental goals. However, I share your intuition that (somehow) uniformly sampling utility functions over universe-histories might not lead to instrumental convergence.
To understand instrumental convergence and power-seeking, consider how many reward functions we might specify automatically imply a causal mechanism for increasing reward. The structure of the reward function implies that more is better, and that there are mechanisms for repeatedly earning points (for example, by showing itself a high-scoring input).
Since the reward function is “simple” (there’s usually not a way to grade exact universe histories), these mechanisms work in many different situations and points in time. It’s naturally incentivized to assure its own safety in order to best leverage these mechanisms for gaining reward. Therefore, we shouldn’t be surprised to see a lot of these simple goals leading to the same kind of power-seeking behavior.
What structure is implied by a reward function?
Additive/Markovian: while a utility function might be over an entire universe-history, reward is often additive over time steps. This is a strong constraint which I don’t always expect to be true, but i think that among the goals with this structure, a greater proportion of them have power-seeking incentives.
Observation-based: while a utility function might be over an entire universe-history, the atom of the reward function is the observation. Perhaps the observation is an input to update a world model, over which we have tried to define a reward function. I think that most ways of doing this lead to power-seeking incentives.
Agent-centric: reward functions are defined with respect to what the agent can observe. Therefore, in partially observable environments, there is naturally a greater emphasis on the agent’s vantage point in the environment.
My theorems apply to the finite, fully observable, Markovian situation.[1] We might not end up using reward functions for more impressive tasks – we might express preferences over incomplete trajectories, for example. The “specify a reward function over the agent’s world model” approach may or may not lead to good subhuman performance in complicated tasks like cleaning warehouses. Imagine specifying a reward function over pure observations for that task – the agent would probably just get stuck looking at a wall in a particularly high-scoring way.
However, for arbitrary utility functions over universe histories, the structure isn’t so simple. With utility functions over universe histories having far more degrees of freedom, arbitrary policies can be rationalized as VNM expected utility maximization. That said, with respect to a simplicity prior over computable utility functions, the power-seeking ones might have most of the measure.
A more appropriate claim might be: goal-directed behavior tends to lead to power-seeking, and that’s why goal-directed behavior tends to be bad.
However, it’s well-known that you can convert finite non-Markovian MDPs into finite Markovian MDPs.
I’ve just put up a post which serves as a broader response to the ideas underpinning this type of argument.
I think this depends a lot on how you model the agent developing. If you start off with a highly intelligent agent which has the ability to make long-term plans, but doesn’t yet have any goals, and then you train it on a random reward function—then yes, it probably will develop strong convergent instrumental goals.
On the other hand, if you start off with a randomly initialised neural network, and then train it on a random reward function, then probably it will get stuck in a local optimum pretty quickly, and never learn to even conceptualise these things called “goals”.
I claim that when people think about reward functions, they think too much about the former case, and not enough about the latter. Because while it’s true that we’re eventually going to get highly intelligent agents which can make long-term plans, it’s also important that we get to control what reward functions they’re trained on up to that point. And so plausibly we can develop intelligent agents that, in some respects, are still stuck in “local optima” in the way they think about convergent instrumental goals—i.e. they’re missing whatever cognitive functionality is required for being ambitious on a large scale.
Agreed – I should have clarified. I’ve been mostly discussing instrumental convergence with respect to optimal policies. The path through policy space is also important.
Makes sense. For what it’s worth, I’d also argue that thinking about optimal policies at all is misguided (e.g. what’s the optimal policy for humans—the literal best arrangement of neurons we could possibly have for our reproductive fitness? Probably we’d be born knowing arbitrarily large amounts of information. But this is just not relevant to predicting or modifying our actual behaviour at all).
(I now think that you were very right in saying “thinking about optimal policies at all is misguided”, and I was very wrong to disagree. I’ve thought several times about this exchange. Not listening to you about this point was a serious error and made my work way less impactful. I do think that the power-seeking theorems say interesting things, but about eg internal utility functions over an internal planning ontology—not about optimal policies for a reward function.)
I disagree.
We do in fact often train agents using algorithms which are proven to eventually converge to the optimal policy.[1] Even if we don’t expect the trained agents to reach the optimal policy in the real world, we should still understand what behavior is like at optimum. If you think your proposal is not aligned at optimum but is aligned for realistic training paths, you should have a strong story for why.
Formal theorizing about instrumental convergence with respect to optimal behavior is strictly easier than theorizing about ϵ-optimal behavior, which I think is what you want for a more realistic treatment of instrumental convergence for real agents. Even if you want to think about sub-optimal policies, if you don’t understand optimal policies… good luck! Therefore, we also have an instrumental (...) interest in studying the behavior at optimum.
At least, the tabular algorithms are proven, but no one uses those for real stuff. I’m not sure what the results are for function approximators, but I think you get my point.
1. I think it’s more accurate to say that, because approximately none of the non-trivial theoretical results hold for function approximation, approximately none of our non-trivial agents are proven to eventually converge to the optimal policy. (Also, given the choice between an algorithm without convergence proofs that works in practice, and an algorithm with convergence proofs that doesn’t work in practice, everyone will use the former). But we shouldn’t pay any attention to optimal policies anyway, because the optimal policy in an environment anything like the real world is absurdly, impossibly complex, and requires infinite compute.
2. I think theorizing about ϵ-optimal behavior is more useful than theorizing about optimal behaviour by roughly ϵ, for roughly the same reasons. But in general, clearly I can understand things about suboptimal policies without understanding optimal policies. I know almost nothing about the optimal policy in StarCraft, but I can still make useful claims about AlphaStar (for example: it’s not going to take over the world).
Again, let’s try cash this out. I give you a human—or, say, the emulation of a human, running in a simulation of the ancestral environment. Is this safe? How do you make it safer? What happens if you keep selecting for intelligence? I think that the theorising you talk about will be actively harmful for your ability to answer these questions.
I’m confused, because I don’t disagree with any specific point you make—just the conclusion. Here’s my attempt at a disagreement which feels analogous to me:
My response in this “debate” is: if you start with a spherical cow and then consider which real world differences are important enough to model, you’re better off than just saying “no one should think about spherical cows”.
I don’t understand why you think that. If you can have a good understanding of instrumental convergence and power-seeking for optimal agents, then you can consider whether any of those same reasons apply for suboptimal humans.
Considering power-seeking for optimal agents is a relaxed problem. Yes, ideally, we would instantly jump to the theory that formally describes power-seeking for suboptimal agents with realistic goals in all kinds of environments. But before you do that, a first step is understanding power-seeking in MDPs. Then, you can take formal insights from this first step and use them to update your pre-theoretic intuitions where appropriate.
Thanks for engaging despite the opacity of the disagreement. I’ll try to make my position here much more explicit (and apologies if that makes it sound brusque). The fact that your model is a simplified abstract model is not sufficient to make it useful. Some abstract models are useful. Some are misleading and will cause people who spend time studying them to understand the underlying phenomenon less well than they did before. From my perspective, I haven’t seen you give arguments that your models are in the former category not the latter. Presumably you think they are in fact useful abstractions—why? (A few examples of the latter: behaviourism, statistical learning theory, recapitulation theory, Gettier-style analysis of knowledge).
My argument for why they’re overall misleading: when I say that “the optimal policy in an environment anything like the real world is absurdly, impossibly complex, and requires infinite compute”, or that safety researchers shouldn’t think about AIXI, I’m not just saying that these are inaccurate models. I’m saying that they are modelling fundamentally different phenomena than the ones you’re trying to apply them to. AIXI is not “intelligence”, it is brute force search, which is a totally different thing that happens to look the same in the infinite limit. Optimal tabular policies are not skill at a task, they are a cheat sheet, but they happen to look similar in very simple cases.
Probably the best example of what I’m complaining about is Ned Block trying to use Blockhead to draw conclusions about intelligence. I think almost everyone around here would roll their eyes hard at that. But then people turn around and use abstractions that are just as unmoored from reality as Blockhead, often in a very analogous way. (This is less a specific criticism of you, TurnTrout, and more a general criticism of the field).
Forgive me a little poetic license. The analogy in my mind is that you were trying to model the cow as a sphere, but you didn’t know how to do so without setting its weight as infinite, and what looked to you like your model predicting the cow would roll downhill was actually your model predicting that the cow would swallow up the nearby fabric of spacetime and the bottom of the hill would fall into its event horizon. At which point, yes, you would be better off just saying “nobody should think about spherical cows”.
Thanks for elaborating this interesting critique. I agree we generally need to be more critical of our abstractions.
Falsifying claims and “breaking” proposals is a classic element of AI alignment discourse and debate. Since we’re talking about superintelligent agents, we can’t predict exactly what a proposal would do. However, if I make a claim (“a superintelligent paperclip maximizer would keep us around because of gains from trade”), you can falsify this by showing that my claimed policy is dominated by another class of policies (“we would likely be comically resource-inefficient in comparison; GFT arguments don’t model dynamics which allow killing other agents and appropriating their resources”).
Even we can come up with this dominant policy class, so the posited superintelligence wouldn’t miss it either. We don’t know what the superintelligent policy will be, but we know what it won’t be (see also Formalizing convergent instrumental goals). Even though I don’t know how Gary Kasparov will open the game, I confidently predict that he won’t let me checkmate him in two moves.
Non-optimal power and instrumental convergence
Instead of thinking about optimal policies, let’s consider the performance of a given algorithm A. A(M,R) takes a rewardless MDP M and a reward function R as input, and outputs a policy.
Definition. Let R be a continuous distribution over reward functions with CDF F. The average return achieved by algorithm A at state s and discount rate γ is ∫RVA(M,R)R(s,γ)dF(R).
Instrumental convergence with respect to A’s policies can be defined similarly (“what is the R-measure of a given trajectory under A?”). The theory I’ve laid out allows precise claims, which is a modest benefit to our understanding. Before, we just had intuitions about some vague concept called “instrumental convergence”.
Here’s bad reasoning, which implies that the cow tears a hole in spacetime:
The problem is that it’s impractical to predict what a smarter agent will do, or what specific kinds of action will be instrumentally convergent for A, or that the real agent would be infinitely smart. Just because it’s smart doesn’t mean it’s omniscient, as you rightly point out.
Here’s better reasoning:
It might seem like I’m assuming the consequent here. This is not so – the work is first done by the theorems on optimal behavior, which do imply that most goals achieve greater return by avoiding shutdown. The question is whether reasonably intelligent suboptimal agents realize this fact. Given a uniformly drawn reward function, we can usually come up with a better policy than dying, so the argument is that A can as well.
I’m afraid I’m mostly going to disengage here, since it seems more useful to spend the time writing up more general + constructive versions of my arguments, rather than critiquing a specific framework.
If I were to sketch out the reasons I expect to be skeptical about this framework if I looked into it in more detail, it’d be something like:
1. Instrumental convergence isn’t training-time behaviour, it’s test-time behaviour. It isn’t about increasing reward, it’s about achieving goals (that the agent learned by being trained to increase reward).
2. The space of goals that agents might learn is very different from the space of reward functions. As a hypothetical, maybe it’s the case that neural networks are just really good at producing deontological agents, and really bad at producing consequentialists. (E.g, if it’s just really really difficult for gradient descent to get a proper planning module working). Then agents trained on almost all reward functions will learn to do well on them without developing convergent instrumental goals. (I expect you to respond that being deontological won’t get you to optimality. But I would say that talking about “optimality” here ruins the abstraction, for reasons outlined in my previous comment).
I was actually going to respond, “that’s a good point, but (IMO) a different concern than the one you initially raised”. I see you making two main critiques.
(paraphrased) ”A won’t produce optimal policies for the specified reward function [even assuming alignment generalization off of the training distribution], so your model isn’t useful” – I replied to this critique above.
“The space of goals that agents might learn is very different from the space of reward functions.” I agree this is an important part of the story. I think the reasonable takeaway is “current theorems on instrumental convergence help us understand what superintelligent A won’t do, assuming no reward-result gap. Since we can’t assume alignment generalization, we should keep in mind how the inductive biases of gradient descent affect the eventual policy produced.”
I remain highly skeptical of the claim that applying this idealized theory of instrumental convergence worsens our ability to actually reason about it.
ETA: I read some information you privately messaged me, and i see why you might see the above two points as a single concern.
Is the point that people try to use algorithms which they think will eventually converge to the optimal policy? (Assuming there is one.)
Something like that, yeah.
I object to the claim that agents that act randomly can be made “arbitrarily simple”. Randomness is basically definitionally complicated!
Eh, this seems a bit nitpicky. It’s arbitrarily simple given a call to a randomness oracle, which in practice we can approximate pretty easily. And it’s “definitionally” easy to specify as well: “the function which, at each call, returns true with 50% likelihood and false otherwise.”
If you get an ‘external’ randomness oracle, then you could define the utility function pretty simply in terms of the outputs of the oracle.
If the agent has a pseudo-random number generator (PRNG) inside it, then I suppose I agree that you aren’t going to be able to give it a utility function that has the standard set of convergent instrumental goals, and PRNGs can be pretty short. (Well, some search algorithms are probably shorter, but I bet they have higher Kt complexity, which is probably a better measure for agents)
I’d take a different tack here, actually; I think this depends on what the input to the utility function is. If we’re only allowed to look at ‘atomic reality’, or the raw actions the agent takes, then I think your analysis goes through, that we have a simple causal process generating the behavior but need a very complicated utility function to make a utility-maximizer that matches the behavior.
But if we’re allowed to decorate the atomic reality with notes like “this action was generated randomly”, then we can have a utility function that’s as simple as the generator, because it just counts up the presence of those notes. (It doesn’t seem to me like this decorator is meaningfully more complicated than the thing that gave us “agents taking actions” as a data source, so I don’t think I’m paying too much here.)
This can lead to a massive explosion in the number of possible utility functions (because there’s a tremendous number of possible decorators), but I think this matches the explosion that we got by considering agents that were the outputs of causal processes in the first place. That is, consider reasoning about python code that outputs actions in a simple game, where there are many more possible python programs than there are possible policies in the game.
So in general you can’t have utility functions that are as simple as the generator, right? E.g. the generator could be deontological. In which case your utility function would be complicated. Or it could be random, or it could choose actions by alphabetical order, or...
And so maybe you can have a little note for each of these. But now what it sounds like is: “I need my notes to be able to describe every possible cognitive algorithm that the agent could be running”. Which seems very very complicated.
I guess this is what you meant by the “tremendous number” of possible decorators. But if that’s what you need to do to keep talking about “utility functions”, then it just seems better to acknowledge that they’re broken as an abstraction.
E.g. in the case of python code, you wouldn’t do anything analogous to this. You would just try to reason about all the possible python programs directly. Similarly, I want to reason about all the cognitive algorithms directly.
That’s right.
I realized my grandparent comment is unclear here:
This should have been “consequence-desirability-maximizer” or something, since the whole question is “does my utility function have to be defined in terms of consequences, or can it be defined in terms of arbitrary propositions?”. If I want to make the deontologist-approximating Innocent-Bot, I have a terrible time if I have to specify the consequences that correspond to the bot being innocent and the consequences that don’t, but if you let me say “Utility = 0 - badness of sins committed” then I’ve constructed a ‘simple’ deontologist. (At least, about as simple as the bot that says “take random actions that aren’t sins”, since both of them need to import the sins library.)
In general, I think it makes sense to not allow this sort of elaboration of what we mean by utility functions, since the behavior we want to point to is the backwards assignment of desirability to actions based on the desirability of their expected consequences, rather than the expectation of any arbitrary property.
---
Actually, I also realized something about your original comment which I don’t think I had the first time around; if by “some reasonable percentage of an agent’s actions are random” you mean something like “the agent does epsilon-exploration” or “the agent plays an optimal mixed strategy”, then I think it doesn’t at all require a complicated utility function to generate identical behavior. Like, in the rock-paper-scissors world, and with the simple function ‘utility = number of wins’, the expected utility maximizing move (against tough competition) is to throw randomly, and we won’t falsify the simple ‘utility = number of wins’ hypothesis by observing random actions.
Instead I read it as something like “some unreasonable percentage of an agent’s actions are random”, where the agent is performing some simple-to-calculate mixed strategy that is either suboptimal or only optimal by luck (when the optimal mixed strategy is the maxent strategy, for example), and matching the behavior with an expected utility maximizer is a challenge (because your target has to be not some fact about the environment, but some fact about the statistical properties of the actions taken by the agent).
---
I think this is where the original intuition becomes uncompelling. We care about utility-maximizers because they’re doing their backwards assignment, using their predictions of the future to guide their present actions to try to shift the future to be more like what they want it to be. We don’t necessarily care about imitators, or simple-to-write bots, or so on. And so if I read the original post as “the further a robot’s behavior is from optimal, the less likely it is to demonstrate convergent instrumental goals”, I say “yeah, sure, but I’m trying to build smart robots (or at least reasoning about what will happen if people try to).”
This is in fact the intended reading, sorry for ambiguity. Will edit. But note that there are probably very few situations where exploring via actual randomness is best; there will almost always be some type of exploration which is more favourable. So I don’t think this helps.
To be pedantic: we care about “consequence-desirability-maximisers” (or in Rohin’s terminology, goal-directed agents) because they do backwards assignment. But I think the pedantry is important, because people substitute utility-maximisers for goal-directed agents, and then reason about those agents by thinking about utility functions, and that just seems incorrect.
What do you mean by optimal here? The robot’s observed behaviour will be optimal for some utility function, no matter how long you run it.
Valid point.
This also seems right. Like, my understanding of what’s going on here is we have:
‘central’ consequence-desirability-maximizers, where there’s a simple utility function that they’re trying to maximize according to the VNM axioms
‘general’ consequence-desirability-maximizers, where there’s a complicated utility function that they’re trying to maximize, which is selected because it imitates some other behavior
The first is a narrow class, and depending on how strict you are with ‘maximize’, quite possibly no physically real agents will fall into it. The second is a universal class, which instantiates the ‘trivial claim’ that everything is utility maximization.
Put another way, the first is what happens if you hold utility fixed / keep utility simple, and then examine what behavior follows; the second is what happens if you hold behavior fixed / keep behavior simple, and then examine what utility follows.
Distance from the first is what I mean by “the further a robot’s behavior is from optimal”; I want to say that I should have said something like “VNM-optimal” but actually I think it needs to be closer to “simple utility VNM-optimal.”
I think you’re basically right in calling out a bait-and-switch that sometimes happens, where anyone who wants to talk about the universality of expected utility maximization in the trivial ‘general’ sense can’t get it to do any work, because it should all add up to normality, and in normality there’s a meaningful distinction between people who sort of pursue fuzzy goals and ruthless utility maximizers.