Thank you for the substantive response. I do think there’s a few misunderstandings here of what I’m saying.
There need not be one best world state, and a world state need not be distinguishable from all others—merely from some of them. (In fact, utility function yielding a real value compresses the world into a characteristic of things we care about in such a way.)
I’m not talking about world states which exist “out there” in the territory, which, is debatable whether they exist or not anyway, I’m talking about world states that exist within the agent, compressed however they like. Within the agent, each world state they consider as a possible goal is distinguished from the others in order for it to be meaningful in some way. The distinguishing characteristics can be decided by the agent in an arbitrary way.
Your series of posts also assume that signs have a fixed order. This is false. For instance, different fields of mathematics treat real number as either first order signs (atomic objects) or higher-order ones, defined as relations on rational numbers.
It is no coincidence that those definitions are identical; you cannot assume that if something is expressible using higher order signs, is not also expressible in lower order.
So when I’m talking about signs I’m talking about signifier/signified pairs, when we’re talking about real numbers for example, we’re taking about a signifier with two different signifieds, therefore two different signs. I talk about exactly this issue in my last post:
When your goal is to create something new, something novel, your goal is necessarily a higher order sign. Things which do not yet exist cannot be directly represented as a first order sign. And how do we know that this thing which doesn’t yet exist is the thing we seek? The only way is through reference to other signs, hence making it a higher order sign. For example, when we speak of a theory of quantum gravity, we are not speaking the name of an actual theory, but the theory which fulfills a role within the existing scientific framework of physics. This is different from known signs that are the output of an operation, for example a specific number that is the answer to a math question, in these cases sign function collapse is possible (we can think of 4 as either a proper name of a concept, or merely as the consequence as from a certain logical rule).
As I say, most signifiers do have both associated first order and higher order signs! But these are /not/ the same thing, they are not equivalent, as you say they are, from an information perspective. If you know the first order sign, there’s no reason you would automatically know the corresponding higher order sign, and the same for vice versa, as I show in my excerpt from my most recent blog.
My argument specifically hinges on whether it’s possible for an agent to have final goals without higher order signs: it’s not, precisely because first order and higher order signs do not contain the same information.
Engaging with the perspective of orthogonality thesis itself: rejecting it means that a change in intelligence will lead, in expectation, to change in final goals. Could you name the expected direction of such a change, like “more intelligent agents will act with less kindness”?
Thank you for the substantive response. I do think there’s a few misunderstandings here of what I’m saying.
I’m not talking about world states which exist “out there” in the territory, which, is debatable whether they exist or not anyway, I’m talking about world states that exist within the agent, compressed however they like. Within the agent, each world state they consider as a possible goal is distinguished from the others in order for it to be meaningful in some way. The distinguishing characteristics can be decided by the agent in an arbitrary way.
So when I’m talking about signs I’m talking about signifier/signified pairs, when we’re talking about real numbers for example, we’re taking about a signifier with two different signifieds, therefore two different signs. I talk about exactly this issue in my last post:
As I say, most signifiers do have both associated first order and higher order signs! But these are /not/ the same thing, they are not equivalent, as you say they are, from an information perspective. If you know the first order sign, there’s no reason you would automatically know the corresponding higher order sign, and the same for vice versa, as I show in my excerpt from my most recent blog.
My argument specifically hinges on whether it’s possible for an agent to have final goals without higher order signs: it’s not, precisely because first order and higher order signs do not contain the same information.
I couldn’t name a specific direction, but what I would say is that agents of similar intelligence and environment will tend towards similar final goals. Otherwise, I generally agree with this post on the topic. https://unstableontology.com/2024/09/19/the-obliqueness-thesis/