If making decisions some way incentivizes other agents to become less like LDTs and more like uncooperative boulders, you can simply not make decisions that way.
Another way that those agents might handle the situation is not to become boulders themselves, but to send boulders to make the offer. That is, send a minion to present the offer without any authority to discuss terms. I believe this often happens in the real world, e.g. customer service staff whose main goal, for their own continued employment, is to send the aggrieved customer away empty-handed and never refer the call upwards.
A smart agent can simply make decisions like a negotiator with restrictions on the kinds of terms it can accept, without having to spawn a “boulder” to do that.
You can just do the correct thing, without having to separate yourself into parts that do things correctly and a part that tries to not look at the world and spawns correct-thing-doers.
In Parfit’s Hitchhiker, you can just pay once you’re there, without precommiting/rewriting yourself into an agent that pays. You can just do the thing that wins.
Some agents can’t do the things that win and would have to rewrite themselves into something better and still lose in some problems, but you can be an agent that wins, and gradient descent probably crystallizes something that wins into what is making the decisions in smart enough things.
Another way that those agents might handle the situation is not to become boulders themselves, but to send boulders to make the offer. That is, send a minion to present the offer without any authority to discuss terms. I believe this often happens in the real world, e.g. customer service staff whose main goal, for their own continued employment, is to send the aggrieved customer away empty-handed and never refer the call upwards.
A smart agent can simply make decisions like a negotiator with restrictions on the kinds of terms it can accept, without having to spawn a “boulder” to do that.
You can just do the correct thing, without having to separate yourself into parts that do things correctly and a part that tries to not look at the world and spawns correct-thing-doers.
In Parfit’s Hitchhiker, you can just pay once you’re there, without precommiting/rewriting yourself into an agent that pays. You can just do the thing that wins.
Some agents can’t do the things that win and would have to rewrite themselves into something better and still lose in some problems, but you can be an agent that wins, and gradient descent probably crystallizes something that wins into what is making the decisions in smart enough things.