The idea is that the universe offers you Dutch-book situations and you make and take bets on uncertain outcomes implicitly.
That said, I concur with your basic point: universal overarching utility functions—not just small ones for a given situation, but a single large one for you as a human—are something humans don’t, and I think can’t, do—and realising how mathematically helpful it would be if they did still doesn’t mean they can, and trying to turn oneself into an expected utility maximiser is unlikely to work.
(And, I suspect, will merely leave you vulnerable to everyday human-level exploits—remember that the actual threat model we evolved in is beating other humans, and as long as we’re dealing with humans we need to deal with humans.)
The idea is that the universe offers you Dutch-book situations
But does it in fact do that? To the extent that you believe that humans are bad Bayesians, you believe that the environment in which humans evolved wasn’t constantly Dutch-booking them, or that if it was then humans evolved some defense against this which isn’t becoming perfect Bayesians.
I do suspect that our thousand shards of desire being contradictory and not resolving is selected for, in that we are thus money-pumped into propagating our genes.
The idea is that the universe offers you Dutch-book situations and you make and take bets on uncertain outcomes implicitly.
That said, I concur with your basic point: universal overarching utility functions—not just small ones for a given situation, but a single large one for you as a human—are something humans don’t, and I think can’t, do—and realising how mathematically helpful it would be if they did still doesn’t mean they can, and trying to turn oneself into an expected utility maximiser is unlikely to work.
(And, I suspect, will merely leave you vulnerable to everyday human-level exploits—remember that the actual threat model we evolved in is beating other humans, and as long as we’re dealing with humans we need to deal with humans.)
But does it in fact do that? To the extent that you believe that humans are bad Bayesians, you believe that the environment in which humans evolved wasn’t constantly Dutch-booking them, or that if it was then humans evolved some defense against this which isn’t becoming perfect Bayesians.
I do suspect that our thousand shards of desire being contradictory and not resolving is selected for, in that we are thus money-pumped into propagating our genes.