Goal-completeness doesn’t make much sense as a rigorous concept because of No-Free-Lunch theorems in optimisation. A goal is essentially a specification of a function to optimise, and all optimisation algorithms perform equally well (or rather poorly) when averaged across all functions.
There is no system that can take in an arbitrary goal specification (which is, say, a subset of the state space of the universe) and achieve that goal on average better than any other such system. My stupid random action generator is equally as bad as the superintelligence when averaged across all goals. Most goals are incredibly noisy, the ones that we care about form a tiny subset of the space of all goals, and any progress in AI we make is really about biasing our models to be good on the goals we care about.
The No Free Lunch theorem is irrelevant in worlds like ours that are a subset of possible data structures (world arrangements). I’m surprised this isn’t better understood. I think Steve Byrnes did a nice writeup of this logic. I can find the link if you like.
Hmm, but here the set of possible world states would be the domain of the function we’re optimising, not the function itself. Like, No-Free-Lunch states (from wikipedia):
Theorem 1: Given a finite set V and a finite set S of real numbers, assume that f:S→V is chosen at random according to uniform distribution on the set SV of all possible functions from V to S. For the problem of optimizing f over the set V, then no algorithm performs better than blind search.
Here V is the set of possible world arrangements, which is admittedly much smaller than all possible data structures, but the theorem still holds because we’re averaging over all possible value functions on this set of worlds, a set which is not physically restricted by anything.
I’d be very interested if you can find Byrnes’ writeup.
Here it is: The No Free Lunch theorem for dummies. See particularly the second section: Sidenote: Why NFL has basically nothing to do with AGI and the first link to Yudkowsky’s post on essentially the same thing.
I think the thing about your descripton is that S → V is not going to be chosen at random in our world.
The no free lunch theorem states in essence (I’m pretty sure) that no classifier can both classify a big gray thing with tusks and big ears as both an elephant and not-an-elephant. That’s fine, because the remainder of an AGI system can choose (by any other criteria) to make elephants either a goal or an anti-goal or neither.
If the NFL theorem applied to general intelligences, it seems like humans couldn’t love elephants at one time and hate them at a later time, with no major changes to their perceptual systems. It proves too much.
A goal is essentially a specification of a function to optimise, and all optimisation algorithms perform equally well (or rather poorly) when averaged across all functions.
Well, I’ve never met a monkey that has an “optimization algorithm” by your definition. I’ve only met humans who have such optimization algorithms. And that distinction is what I’m pointing at.
Goal-completeness points to the same thing as what most people mean by “AGI”.
E.g. I claim humans are goal-complete General Intelligences because you can give us any goal-specification and we’ll very often be able to steer the future closer toward it.
Currently, no other known organism or software program has this property to the degree that humans do. GPT-4 has it for an unprecedentedly large domain, by virtue of giving satisfying answers to a large fraction of arbitrary natural-language prompts.
E.g. I claim humans are goal-complete General Intelligences because you can give us any goal-specification and we’ll very often be able to steer the future closer toward it.
If you’re thinking of “goals” as easily specified natural-language things, then I agree with you, but the point is that turing-completeness is a rigorously defined concept, and if you want to get the same level of rigour for “goal-completeness”, then most goals will be of the form “atom 1 is a location x, atom 2 is at location y, …” for all atoms in the universe. And when averaged across all such goals, literally just acting randomly performs as well as a human or a monkey trying their best to achieve the goal.
Hmm it seems to me that you’re just being pedantic about goal-completeness in a way that you aren’t symmetrically being for Turing-completeness.
You could point out that “most” Turing machines output tapes full of 10^100 1s and 0s in a near-random configuration, and every computing device on earth is equally hopeless at doing that.
I’ll try to say the point some other way: you define “goal-complete” in the following way:
By way of definition: An AI whose input is an arbitrary goal, which outputs actions to effectively steer the future toward that goal, is goal-complete.
Suppose you give me a specification of a goal as a function f:S→{0,1} from a state space to a binary output. Is the AI which just tries out uniformly random actions in perpetuity until it hits one of the goal states “goal-complete”? After all, no matter the goal specification this AI will eventually hit it, though it might take a very long time.
I think the interesting thing you’re trying to point at is contained in what it means to “effectively” steer the future, not in goal-arbitrariness.
I agree that if a goal-complete AI steers the future very slowly, or very weakly—as by just trying every possible action one at a time—then at some point it becomes a degenerate case of the concept.
(Applying the same level of pedantry to Turing-completeness, you could similarly ask if the simple Turing machine that enumerates all possible output-tape configurations one-by-one is a UTM.)
The reason “goal-complete” (or “AGI”) is a useful coinage, is that there’s a large cluster in plausible-reality-space of goal-complete agents with a reasonable amount of goal-complete optimization power (e.g. humans, natural selection, and probably AI starting in a few years), and another large distinguishable cluster of non-goal-complete agents (e.g. the other animals, narrow AI).
I don’t get what point you’re trying to make about the takeaway of my analogy by bringing up the halting problem. There might not even be something analogous to the halting problem in my analogy of goal-completeness, but so what?
I also don’t get why you’re bringing up the detail that “single correct output” is not 100% the same thing as “single goal-specification with variable degrees of success measured on a utility function”. It’s in the nature of analogies that details are different yet we’re still able to infer an analogous conclusion on some dimension.
Humans are goal-complete, or equivalently “humans are general intelligences”, in the sense that many of us in the smartest quartile can output plans with the expectation of a much better than random score on a very broad range of utility functions over arbitrary domains.
I find the ideas you discuss interesting, but they leave me with more questions. I agree that we are moving toward a more generic AI that we can use for all kinds of tasks.
I have trouble understanding the goal-completeness concept. I’d reiterate @Razied ’s point. You mention “steers the future very slowly”, so there is an implicit concept of “speed of steering”. I don’t find the turing machine analogy helpful in infering an analogous conclusion because I don’t know what that conclusion is.
You’re making a qualitative distinction between humans (goal-complete) and other animals (non-goal complete) agents. I don’t understand what you mean by that distinction. I find the idea of goal completeness interesting to explore but quite fuzzy at this point.
Unlike the other animals, humans can represent any goal in a large domain like the physical universe, and then in a large fraction of cases, they can think of useful things to steer the universe toward that goal to an appreciable degree.
Some goals are more difficult than others / require giving the human control over more resources than others, and measurements of optimization power are hard to define, but this definition is taking a step toward formalizing the claim that humans are more of a “general intelligence” than animals. Presumably you agree with this claim?
It seems the crux of our disagreement factors down to a disagreement about whether this Optimization Power post by Eliezer is pointing at a sufficiently coherent concept.
Goal-completeness doesn’t make much sense as a rigorous concept because of No-Free-Lunch theorems in optimisation. A goal is essentially a specification of a function to optimise, and all optimisation algorithms perform equally well (or rather poorly) when averaged across all functions.
There is no system that can take in an arbitrary goal specification (which is, say, a subset of the state space of the universe) and achieve that goal on average better than any other such system. My stupid random action generator is equally as bad as the superintelligence when averaged across all goals. Most goals are incredibly noisy, the ones that we care about form a tiny subset of the space of all goals, and any progress in AI we make is really about biasing our models to be good on the goals we care about.
The No Free Lunch theorem is irrelevant in worlds like ours that are a subset of possible data structures (world arrangements). I’m surprised this isn’t better understood. I think Steve Byrnes did a nice writeup of this logic. I can find the link if you like.
Hmm, but here the set of possible world states would be the domain of the function we’re optimising, not the function itself. Like, No-Free-Lunch states (from wikipedia):
Theorem 1: Given a finite set V and a finite set S of real numbers, assume that f:S→V is chosen at random according to uniform distribution on the set SV of all possible functions from V to S. For the problem of optimizing f over the set V, then no algorithm performs better than blind search.
Here V is the set of possible world arrangements, which is admittedly much smaller than all possible data structures, but the theorem still holds because we’re averaging over all possible value functions on this set of worlds, a set which is not physically restricted by anything.
I’d be very interested if you can find Byrnes’ writeup.
Here it is: The No Free Lunch theorem for dummies. See particularly the second section: Sidenote: Why NFL has basically nothing to do with AGI and the first link to Yudkowsky’s post on essentially the same thing.
I think the thing about your descripton is that S → V is not going to be chosen at random in our world.
The no free lunch theorem states in essence (I’m pretty sure) that no classifier can both classify a big gray thing with tusks and big ears as both an elephant and not-an-elephant. That’s fine, because the remainder of an AGI system can choose (by any other criteria) to make elephants either a goal or an anti-goal or neither.
If the NFL theorem applied to general intelligences, it seems like humans couldn’t love elephants at one time and hate them at a later time, with no major changes to their perceptual systems. It proves too much.
Well, I’ve never met a monkey that has an “optimization algorithm” by your definition. I’ve only met humans who have such optimization algorithms. And that distinction is what I’m pointing at.
Goal-completeness points to the same thing as what most people mean by “AGI”.
E.g. I claim humans are goal-complete General Intelligences because you can give us any goal-specification and we’ll very often be able to steer the future closer toward it.
Currently, no other known organism or software program has this property to the degree that humans do. GPT-4 has it for an unprecedentedly large domain, by virtue of giving satisfying answers to a large fraction of arbitrary natural-language prompts.
If you’re thinking of “goals” as easily specified natural-language things, then I agree with you, but the point is that turing-completeness is a rigorously defined concept, and if you want to get the same level of rigour for “goal-completeness”, then most goals will be of the form “atom 1 is a location x, atom 2 is at location y, …” for all atoms in the universe. And when averaged across all such goals, literally just acting randomly performs as well as a human or a monkey trying their best to achieve the goal.
Hmm it seems to me that you’re just being pedantic about goal-completeness in a way that you aren’t symmetrically being for Turing-completeness.
You could point out that “most” Turing machines output tapes full of 10^100 1s and 0s in a near-random configuration, and every computing device on earth is equally hopeless at doing that.
I’ll try to say the point some other way: you define “goal-complete” in the following way:
Suppose you give me a specification of a goal as a function f:S→{0,1} from a state space to a binary output. Is the AI which just tries out uniformly random actions in perpetuity until it hits one of the goal states “goal-complete”? After all, no matter the goal specification this AI will eventually hit it, though it might take a very long time.
I think the interesting thing you’re trying to point at is contained in what it means to “effectively” steer the future, not in goal-arbitrariness.
I agree that if a goal-complete AI steers the future very slowly, or very weakly—as by just trying every possible action one at a time—then at some point it becomes a degenerate case of the concept.
(Applying the same level of pedantry to Turing-completeness, you could similarly ask if the simple Turing machine that enumerates all possible output-tape configurations one-by-one is a UTM.)
The reason “goal-complete” (or “AGI”) is a useful coinage, is that there’s a large cluster in plausible-reality-space of goal-complete agents with a reasonable amount of goal-complete optimization power (e.g. humans, natural selection, and probably AI starting in a few years), and another large distinguishable cluster of non-goal-complete agents (e.g. the other animals, narrow AI).
The turing machine enumeration analogy doesn’t work because the machine needs to halt.
Optimization is conceptually different than computation in that there is no single correct output.
What would humans not being goal-complete look like? What arguments are there for humans being goal-complete?
I don’t get what point you’re trying to make about the takeaway of my analogy by bringing up the halting problem. There might not even be something analogous to the halting problem in my analogy of goal-completeness, but so what?
I also don’t get why you’re bringing up the detail that “single correct output” is not 100% the same thing as “single goal-specification with variable degrees of success measured on a utility function”. It’s in the nature of analogies that details are different yet we’re still able to infer an analogous conclusion on some dimension.
Humans are goal-complete, or equivalently “humans are general intelligences”, in the sense that many of us in the smartest quartile can output plans with the expectation of a much better than random score on a very broad range of utility functions over arbitrary domains.
I find the ideas you discuss interesting, but they leave me with more questions. I agree that we are moving toward a more generic AI that we can use for all kinds of tasks.
I have trouble understanding the goal-completeness concept. I’d reiterate @Razied ’s point. You mention “steers the future very slowly”, so there is an implicit concept of “speed of steering”. I don’t find the turing machine analogy helpful in infering an analogous conclusion because I don’t know what that conclusion is.
You’re making a qualitative distinction between humans (goal-complete) and other animals (non-goal complete) agents. I don’t understand what you mean by that distinction. I find the idea of goal completeness interesting to explore but quite fuzzy at this point.
Unlike the other animals, humans can represent any goal in a large domain like the physical universe, and then in a large fraction of cases, they can think of useful things to steer the universe toward that goal to an appreciable degree.
Some goals are more difficult than others / require giving the human control over more resources than others, and measurements of optimization power are hard to define, but this definition is taking a step toward formalizing the claim that humans are more of a “general intelligence” than animals. Presumably you agree with this claim?
It seems the crux of our disagreement factors down to a disagreement about whether this Optimization Power post by Eliezer is pointing at a sufficiently coherent concept.