I strongly disagree with this view; “freeness” of a choice should be analyzed in terms of how much a local system’s thermal noise has causal impact on other systems. For example, if a system such as a brain has free agency over an object, such as a rock, then the noise in possible brain activations has determining impact over what trajectories the rock will follow, because the denoising process the brain implements gets to guide the behavior of the rock. Notably, this means that even with a deterministic PRNG, diffusion models have a very significant amount of “free agency”, because they have full control over the trajectory and ability to reject noise. Other generative models do as well, but diffusion has notably more than others due to how many steps of denoising steering the model takes; the model becomes very strong at rejecting noise and making its choices, and so the model will always have impact on its output that controls it into aesthetic states defined by the model weights.
Compare this to how much aesthetic control a brain has over their body’s behavior in circumstances with reduced optionality; the reduced optionality is recognizeable from an oracle’s perspective due to the fact that the externally unpredictable chaos of local trajectories in the brain does not have as many paths that cause the brain’s aesthetically desired outcomes.
For example, if someone is the only person on a planet, the person cannot push through barriers in statespace such as “I’m low on fuel and have an unknown disease”.
Or if someone is in a place where the people around them consider them to not have offered acceptable trade for access to material goods and services, but the person is out of fuel and has an unknown disease, their optionality is significantly reduced versus if material goods had not been collected by the actions of the stronger agents in their environment.
So, okay, I’ve claimed we can define optionality usefully. What does this mean about how a stronger agent should behave in a competitive environment?
I’d argue that the key question any safety-of-inter-agent-interaction alignment approach needs to answer is how to distribute gains from trade in order to maximize optionality of both agents in the trade. In other words, to minimize externality-inducing exploitation of vulnerability, maximize the combined optionality of both agents.
I tried to find papers to cite for this post, and didn’t find any of what I was looking for, which is frustrating, because I’m pretty sure I’ve seen good work out of economic game theory recently, but my searches are coming up mostly empty handed. here’s one result:
I strongly disagree with this view; “freeness” of a choice should be analyzed in terms of how much a local system’s thermal noise has causal impact on other systems. For example, if a system such as a brain has free agency over an object, such as a rock, then the noise in possible brain activations has determining impact over what trajectories the rock will follow, because the denoising process the brain implements gets to guide the behavior of the rock. Notably, this means that even with a deterministic PRNG, diffusion models have a very significant amount of “free agency”, because they have full control over the trajectory and ability to reject noise. Other generative models do as well, but diffusion has notably more than others due to how many steps of denoising steering the model takes; the model becomes very strong at rejecting noise and making its choices, and so the model will always have impact on its output that controls it into aesthetic states defined by the model weights.
Compare this to how much aesthetic control a brain has over their body’s behavior in circumstances with reduced optionality; the reduced optionality is recognizeable from an oracle’s perspective due to the fact that the externally unpredictable chaos of local trajectories in the brain does not have as many paths that cause the brain’s aesthetically desired outcomes.
For example, if someone is the only person on a planet, the person cannot push through barriers in statespace such as “I’m low on fuel and have an unknown disease”.
Or if someone is in a place where the people around them consider them to not have offered acceptable trade for access to material goods and services, but the person is out of fuel and has an unknown disease, their optionality is significantly reduced versus if material goods had not been collected by the actions of the stronger agents in their environment.
So, okay, I’ve claimed we can define optionality usefully. What does this mean about how a stronger agent should behave in a competitive environment?
I’d argue that the key question any safety-of-inter-agent-interaction alignment approach needs to answer is how to distribute gains from trade in order to maximize optionality of both agents in the trade. In other words, to minimize externality-inducing exploitation of vulnerability, maximize the combined optionality of both agents.
I tried to find papers to cite for this post, and didn’t find any of what I was looking for, which is frustrating, because I’m pretty sure I’ve seen good work out of economic game theory recently, but my searches are coming up mostly empty handed. here’s one result:
https://www.semanticscholar.org/paper/Coalition-Formation-in-Games-with-Externalities-Montero/91e61943f424fb464bbf2222a3d31e98e6dc730d