This is pretty unsatisfying as an expansion of “incoherent yet not dominated” given that it just uses the phrase “not coherent” instead.
I find money-pump arguments to be the most compelling ones since they’re essentially tiny selection theorems for agents in adversarial environments, and we’ve got an example in the post of (the skeleton of) a proof that a lack-of-total-preferences doesn’t immediately lead to you being pumped. Perhaps there’s a more sophisticated argument that Actually No, You Still Get Pumped but I don’t think I’ve seen one in the comments here yet.
If there are things which cannot-be-money-pumped, and yet which are not utility-maximizers, and problems like corrigibility are almost certainly unsolvable for utility-maximizers, perhaps it’s somewhat worth looking at coherent non-pumpable non-EU agents?
...wait, you were just asking for an example of an agent being “incoherent but not dominated” in those two senses of being money-pumped? And this is an exercise meant to hint that such “incoherent” agents are always dominatable?
I continue to not see the problem, because the obvious examples don’t work. If I have (1apple,$0) as incomparable to (1banana,$0) that doesn’t mean I turn down the trade of −1apple,+1banana,+$10000 (which I assume is what you’re hinting at re. foregoing free money).
If one then says “ah but if I offer $9999 and you turn that down, then we have identified your secret equivalent utili-” no, this is just a bid/ask spread, and I’m pretty sure plenty of ink has been spilled justifying EUM agents using uncertainty to price inaction like this.
What’s an example of a non-EUM agent turning down free money which doesn’t just reduce to comparing against an EUM with reckless preferences/a low price of uncertainty?
This is pretty unsatisfying as an expansion of “incoherent yet not dominated” given that it just uses the phrase “not coherent” instead.
I find money-pump arguments to be the most compelling ones since they’re essentially tiny selection theorems for agents in adversarial environments, and we’ve got an example in the post of (the skeleton of) a proof that a lack-of-total-preferences doesn’t immediately lead to you being pumped. Perhaps there’s a more sophisticated argument that Actually No, You Still Get Pumped but I don’t think I’ve seen one in the comments here yet.
If there are things which cannot-be-money-pumped, and yet which are not utility-maximizers, and problems like corrigibility are almost certainly unsolvable for utility-maximizers, perhaps it’s somewhat worth looking at
coherentnon-pumpable non-EU agents?Things are dominated when they forego free money and not just when money gets pumped out of them.
How is the toy example agent sketched in the post dominated?
...wait, you were just asking for an example of an agent being “incoherent but not dominated” in those two senses of being money-pumped? And this is an exercise meant to hint that such “incoherent” agents are always dominatable?
I continue to not see the problem, because the obvious examples don’t work. If I have (1 apple,$0) as incomparable to (1 banana,$0) that doesn’t mean I turn down the trade of −1 apple,+1 banana,+$10000 (which I assume is what you’re hinting at re. foregoing free money).
If one then says “ah but if I offer $9999 and you turn that down, then we have identified your secret equivalent utili-” no, this is just a bid/ask spread, and I’m pretty sure plenty of ink has been spilled justifying EUM agents using uncertainty to price inaction like this.
What’s an example of a non-EUM agent turning down free money which doesn’t just reduce to comparing against an EUM with reckless preferences/a low price of uncertainty?
Want to bump this because it seems important? How do you see the agent in the post as being dominated?