A goal is, fundamentally, an idea. As the final step in a plan, you can write it out as a symbolic representation of the “world state” you are trying to achieve, although it could represent other things as well. In a planning computer agent, this will probably terminate in a bunch of 1s and 0s stored in its memory.
In order for this symbolic representation to be meaningful, it must be comparable and distinct from other symbolic representations. World state A in the agent’s plan could be contrasted from world state B, C and D. This is a very fundamental fact about how information and meaning work, if World State A was indistinguishable from all the others, there would be no reason for the agent to act, because its goal would have been “accomplished”.
This has a logic error. There need not be one best world state, and a world state need not be distinguishable from all others—merely from some of them. (In fact, utility function yielding a real value compresses the world into a characteristic of things we care about in such a way.)
Also, with unbounded computations, utility optimizer could tell supremum (best outcome) for any set of world states you’d provide it; without that, it will have less granularity, work on set of close states (for instance, “easily coming to human mind”) or employ other optimization techniques.
I believe this underlies much of the disagreement, because then more knowledge or more intelligence might change only the relations of “final goal” sign but not its meaning (re: isomorphism).
Your series of posts also assume that signs have a fixed order. This is false. For instance, different fields of mathematics treat real number as either first order signs (atomic objects) or higher-order ones, defined as relations on rational numbers.
Or, for an easier example to work on: equality could be a second-order sign “object A is same as object B”, or it may be defined using third order expression “for any property P, A and B either both have the property or both not have it”. It is no coincidence that those definitions are identical; you cannot assume that if something is expressible using higher order signs, is not also expressible in lower order.
And this might undermine the rest of argument.
Engaging with the perspective of orthogonality thesis itself: rejecting it means that a change in intelligence will lead, in expectation, to change in final goals. Could you name the expected direction of such a change, like “more intelligent agents will act with less kindness”?
Thank you for the substantive response. I do think there’s a few misunderstandings here of what I’m saying.
There need not be one best world state, and a world state need not be distinguishable from all others—merely from some of them. (In fact, utility function yielding a real value compresses the world into a characteristic of things we care about in such a way.)
I’m not talking about world states which exist “out there” in the territory, which, is debatable whether they exist or not anyway, I’m talking about world states that exist within the agent, compressed however they like. Within the agent, each world state they consider as a possible goal is distinguished from the others in order for it to be meaningful in some way. The distinguishing characteristics can be decided by the agent in an arbitrary way.
Your series of posts also assume that signs have a fixed order. This is false. For instance, different fields of mathematics treat real number as either first order signs (atomic objects) or higher-order ones, defined as relations on rational numbers.
It is no coincidence that those definitions are identical; you cannot assume that if something is expressible using higher order signs, is not also expressible in lower order.
So when I’m talking about signs I’m talking about signifier/signified pairs, when we’re talking about real numbers for example, we’re taking about a signifier with two different signifieds, therefore two different signs. I talk about exactly this issue in my last post:
When your goal is to create something new, something novel, your goal is necessarily a higher order sign. Things which do not yet exist cannot be directly represented as a first order sign. And how do we know that this thing which doesn’t yet exist is the thing we seek? The only way is through reference to other signs, hence making it a higher order sign. For example, when we speak of a theory of quantum gravity, we are not speaking the name of an actual theory, but the theory which fulfills a role within the existing scientific framework of physics. This is different from known signs that are the output of an operation, for example a specific number that is the answer to a math question, in these cases sign function collapse is possible (we can think of 4 as either a proper name of a concept, or merely as the consequence as from a certain logical rule).
As I say, most signifiers do have both associated first order and higher order signs! But these are /not/ the same thing, they are not equivalent, as you say they are, from an information perspective. If you know the first order sign, there’s no reason you would automatically know the corresponding higher order sign, and the same for vice versa, as I show in my excerpt from my most recent blog.
My argument specifically hinges on whether it’s possible for an agent to have final goals without higher order signs: it’s not, precisely because first order and higher order signs do not contain the same information.
Engaging with the perspective of orthogonality thesis itself: rejecting it means that a change in intelligence will lead, in expectation, to change in final goals. Could you name the expected direction of such a change, like “more intelligent agents will act with less kindness”?
This has a logic error. There need not be one best world state, and a world state need not be distinguishable from all others—merely from some of them. (In fact, utility function yielding a real value compresses the world into a characteristic of things we care about in such a way.)
Also, with unbounded computations, utility optimizer could tell supremum (best outcome) for any set of world states you’d provide it; without that, it will have less granularity, work on set of close states (for instance, “easily coming to human mind”) or employ other optimization techniques.
I believe this underlies much of the disagreement, because then more knowledge or more intelligence might change only the relations of “final goal” sign but not its meaning (re: isomorphism).
Your series of posts also assume that signs have a fixed order. This is false. For instance, different fields of mathematics treat real number as either first order signs (atomic objects) or higher-order ones, defined as relations on rational numbers.
Or, for an easier example to work on: equality could be a second-order sign “object A is same as object B”, or it may be defined using third order expression “for any property P, A and B either both have the property or both not have it”. It is no coincidence that those definitions are identical; you cannot assume that if something is expressible using higher order signs, is not also expressible in lower order.
And this might undermine the rest of argument.
Engaging with the perspective of orthogonality thesis itself: rejecting it means that a change in intelligence will lead, in expectation, to change in final goals. Could you name the expected direction of such a change, like “more intelligent agents will act with less kindness”?
Thank you for the substantive response. I do think there’s a few misunderstandings here of what I’m saying.
I’m not talking about world states which exist “out there” in the territory, which, is debatable whether they exist or not anyway, I’m talking about world states that exist within the agent, compressed however they like. Within the agent, each world state they consider as a possible goal is distinguished from the others in order for it to be meaningful in some way. The distinguishing characteristics can be decided by the agent in an arbitrary way.
So when I’m talking about signs I’m talking about signifier/signified pairs, when we’re talking about real numbers for example, we’re taking about a signifier with two different signifieds, therefore two different signs. I talk about exactly this issue in my last post:
As I say, most signifiers do have both associated first order and higher order signs! But these are /not/ the same thing, they are not equivalent, as you say they are, from an information perspective. If you know the first order sign, there’s no reason you would automatically know the corresponding higher order sign, and the same for vice versa, as I show in my excerpt from my most recent blog.
My argument specifically hinges on whether it’s possible for an agent to have final goals without higher order signs: it’s not, precisely because first order and higher order signs do not contain the same information.
I couldn’t name a specific direction, but what I would say is that agents of similar intelligence and environment will tend towards similar final goals. Otherwise, I generally agree with this post on the topic. https://unstableontology.com/2024/09/19/the-obliqueness-thesis/