Epistemic status: probably wrong; intuitively, I feel like I’m onto something but I’m too uncertain about this framing to be confident in it
I refer to optimizers which can be identified by a measuring stick of utility as agenty optimizers
The measuring stick is optimization power. In particular, in the spirit of this sequence, it is the correlation between local optimization and optimization far away. If I have 4 basic actions available to me and each performs two bits of optimization on the universe, I am maximally powerful (for a structure with 4 basic actions) and most definitely either an agent or constructed by one. I speak and the universe trembles.
One might look at the life on Earth and see that it is unusually structured and unusually purposeful and conclude that it is the work of an agenty optimizer. And they would be wrong.
But if they looked closer, at the pipelines and wires and radio waves on Earth, they might conclude that they were the work of an agenty optimizer because they turn small actions (flipping a switch, pressing a key) into large, distant effects (water does or doesn’t arrive at a village, a purchase is confirmed and a bushel of apples is shipped across the planet). And they would be correct.
In this framing, resources under my control are structures which propagate and amplify my outputs out into large, distant effects (they needn’t be friendly, per se, they just have to be manipulable). Thus, a dollar (+ Amazon + a computer + …) is an invaluable resource because, with it, I can cause any of literally millions of distinct objects to move from one part of the world to another by moving my fingers in the right way. And I can do that because the world has been reshaped to bend to my will in a way that clearly indicates agency to anyone who knows how to look.
However, I haven’t the slightest idea how to turn this framework into a method for actually identifying agents (or resources) in a universe with weird physics.
Also, I have a sense that there is an important difference between accumulating asymmetric power (allies, secret AI) and creating approximately symmetrically empowering infrastructure (Elicit), which is not captured by this framework. Maybe the former is evidence of instrumental resource accumulation whereas the latter provides specific information about the creator’s goals? But both *are* clear signs of agenty optimization, so maybe it’s not relevant to this context?
Also possibly of note is that more optimization power is not strictly desirable because having too many choices might overwhelm your computational limitations.
I’m not sure if this is helpful, but I tend to think of the term “resources” as referring to things that are expended when used (like dollars or fuel cells). I think of reusable tools (like screwdrivers or light-switches) as being in a different category.
(I realize that approximately all tools will in fact wear out after some amount of use, but these still feel like naturally-distinct categories for most things in my actual life. I do not think of my screwdriver or my measuring tape or my silverware as having some finite number of charges that are being expended each time I use them.)
Epistemic status: probably wrong; intuitively, I feel like I’m onto something but I’m too uncertain about this framing to be confident in it
I refer to optimizers which can be identified by a measuring stick of utility as agenty optimizers
The measuring stick is optimization power. In particular, in the spirit of this sequence, it is the correlation between local optimization and optimization far away. If I have 4 basic actions available to me and each performs two bits of optimization on the universe, I am maximally powerful (for a structure with 4 basic actions) and most definitely either an agent or constructed by one. I speak and the universe trembles.
One might look at the life on Earth and see that it is unusually structured and unusually purposeful and conclude that it is the work of an agenty optimizer. And they would be wrong.
But if they looked closer, at the pipelines and wires and radio waves on Earth, they might conclude that they were the work of an agenty optimizer because they turn small actions (flipping a switch, pressing a key) into large, distant effects (water does or doesn’t arrive at a village, a purchase is confirmed and a bushel of apples is shipped across the planet). And they would be correct.
In this framing, resources under my control are structures which propagate and amplify my outputs out into large, distant effects (they needn’t be friendly, per se, they just have to be manipulable). Thus, a dollar (+ Amazon + a computer + …) is an invaluable resource because, with it, I can cause any of literally millions of distinct objects to move from one part of the world to another by moving my fingers in the right way. And I can do that because the world has been reshaped to bend to my will in a way that clearly indicates agency to anyone who knows how to look.
However, I haven’t the slightest idea how to turn this framework into a method for actually identifying agents (or resources) in a universe with weird physics.
Also, I have a sense that there is an important difference between accumulating asymmetric power (allies, secret AI) and creating approximately symmetrically empowering infrastructure (Elicit), which is not captured by this framework. Maybe the former is evidence of instrumental resource accumulation whereas the latter provides specific information about the creator’s goals? But both *are* clear signs of agenty optimization, so maybe it’s not relevant to this context?
Also possibly of note is that more optimization power is not strictly desirable because having too many choices might overwhelm your computational limitations.
I’m not sure if this is helpful, but I tend to think of the term “resources” as referring to things that are expended when used (like dollars or fuel cells). I think of reusable tools (like screwdrivers or light-switches) as being in a different category.
(I realize that approximately all tools will in fact wear out after some amount of use, but these still feel like naturally-distinct categories for most things in my actual life. I do not think of my screwdriver or my measuring tape or my silverware as having some finite number of charges that are being expended each time I use them.)
EDIT: Reworded for clarity.