There is a problem that, other things equal, agents that care about the state of the world in the distant future, to the exclusion of everything else, will outcompete agents that lack that property. This is self-evident, because we can operationalize “outcompete” as “have more effect on the state of the world in the distant future”.
I am not sure about that!
One way this argument could fail: maybe agents who care exclusively about the state of the world in the distant future end up, as part of their optimizing, creating other agents who care in different ways from that.
In that case, they would “have more effect on the state of the world in the distant future”, but they might not “outcompete” other agents (in the common-sensical way of understanding “outcompete”).
A person might think this implausible, because they might think that a smart agent who cares exclusively about X can best achieve X by having all minds they create also be [smart agents who care exclusively about X.
But, I’m not sure this is true, basically for reasons of not trusting assumptions (1), (2), (3), and (4) that I listed here.
(As one possible sketch: a mind whose only goal is to map branch B of mathematics might find it instrumentally useful to map a bunch of other branches of mathematics. And, since supervision is not free, it might be more able to do this efficiently if it creates researchers who have an intrinsic interest in math-in-general, and who are not being fully supervised by exclusively-B-interested minds.)
I just paraphrased the OP for a friend who said he couldn’t decipher it. He said it helped, so I’m copy-pasting here in case it clarifies for others.
I’m trying to say:
A) There’re a lot of “theorems” showing that a thing is what agents will converge on, or something, that involve approximations (“assume a frictionless plane”) that aren’t quite true.
B) The “VNM utility theorem” is one such theorem, and involves some approximations that aren’t quite true. So does e.g. Steve Omohundro’s convergent instrumental drives, the “Gandhi folk theorems” showing that an agent will resist changes to its utility function, etc.
C) So I don’t think the VNM utility theorem means that all minds will necessarily want to become VNM agents, nor to follow instrumental drives, nor to resist changes to their “utility functions” (if indeed they have a “utility function”).
D) But “be a better VNM-agent” “follow the instrumental Omohundro drives” etc. might still be a self-fulfilling prophecy for some region, partially. Like, humans or other entities who think its rational to be VNM agents might become better VNM agents, who might become better VNM agents, for awhile.
E) And there might be other [mathematically describable mind-patterns] that can serve as alternative self-propagating patterns, a la D, that’re pretty different from “be a better VNM-agent.” E.g. “follow the god of nick land”.
F) And I want to know what are all the [mathematically describable mind-patterns, that a mind might decide to emulate, and that might make a kinda-stable attractor for awhile, where the mind and its successors keeps emulating that mind-pattern for awhile]. They’ll probably each have a “theorem” attached that involves some sort of approximation (a la “assume a frictionless plane”).