Hmm, I was going to reply with something like “money-pumps don’t just say something about adversarial environments, they also say something about avoiding leaking resources” (e.g. if you have circular preferences between proximity to apples, bananas, and carrots, then if you encounter all three of them in a single room you might get trapped walking between them forever) but that’s also begging your original question—we can always just update to enjoy leaking resources, transmuting a “leak” into an “expenditure”.
Another frame here is that if you make/encounter an agent, and that agent self-modifies into/starts off as something which is happy to leak pretty fundamental resources like time and energy and material-under-control, then you’re not as worried about it? It’s certainly not competing as strongly for the same resources as you whenever it’s “under the influence” of its circular preferences.
Hmm, I was going to reply with something like “money-pumps don’t just say something about adversarial environments, they also say something about avoiding leaking resources” (e.g. if you have circular preferences between proximity to apples, bananas, and carrots, then if you encounter all three of them in a single room you might get trapped walking between them forever) but that’s also begging your original question—we can always just update to enjoy leaking resources, transmuting a “leak” into an “expenditure”.
Another frame here is that if you make/encounter an agent, and that agent self-modifies into/starts off as something which is happy to leak pretty fundamental resources like time and energy and material-under-control, then you’re not as worried about it? It’s certainly not competing as strongly for the same resources as you whenever it’s “under the influence” of its circular preferences.