I think that to contain the concept of Utility as it exists in me, you would have to do homework exercises I don’t know how to prescribe. Maybe one set of homework exercises like that would be showing you an agent, including a human, making some set of choices that allegedly couldn’t obey expected utility, and having you figure out how to pump money from that agent (or present it with money that it would pass up).
Like, just actually doing that a few dozen times.
Maybe it’s not helpful for me to say this? If you say it to Eliezer, he immediately goes, “Ah, yes, I could see how I would update that way after doing the homework, so I will save myself some time and effort and just make that update now without the homework”, but this kind of jumping-ahead-to-the-destination is something that seems to me to be… dramatically missing from many non-Eliezers.
If you already know how your future self’s credences (to be arrived at using your own reasoning process) will differ from your current credences, you should just trust your expected-future-self’s reasoning and immediately adopt his expected conclusions. If you can foresee how a reasoner like yourself will be led to think in some particular way, then you also have the ability to put yourself in that agent’s shoes and be so led right now. Whatever considerations will foreseeably later motivate you are equally moving considerations now.
In deep stock-markets, many traders are trying to make a buck by knowing something that no one else yet does. Because the current prices of various assets already take into consideration all public knowledge about them, you can’t expect to reliably make money as a trader unless you know something that the other traders don’t. Here, “knowing something” is basically synonymous with “being able to predict how public knowledge about the assets in question will evolve.” If a trader can foresee what everyone will believe in the near future, he can beat everyone else to it by betting on those expected beliefs now. Because of this, whenever a trader in a deep stock-market comes to know something about a relevant future belief, prices today are pushed around to reflect that belief that would’ve been held tomorrow. The deep stock-market is thus epistemically efficient: it factors in every relevant credence that any trader can presently scry, well before those credences become widely held.
You want to be like the epistemically efficient stock-market. Don’t leave any actionable knowledge on the table by turning a deaf ear to what your future self will think; don’t dutifully go through the motions of studying when you already know what you’ll think at the end of the process. Incorporate everything any good reasoning process would conclude right now; if you can currently see it, you’re already in a position to believe it. And so long as you keep doing that, the way your future self will update his credences should remain completely unpredictable to you.
Your Future Self’s Credences Should Be Unpredictable to You
My exposition of Yudkowsky’s below idea about skipping unnecessary studying:
If you already know how your future self’s credences (to be arrived at using your own reasoning process) will differ from your current credences, you should just trust your expected-future-self’s reasoning and immediately adopt his expected conclusions. If you can foresee how a reasoner like yourself will be led to think in some particular way, then you also have the ability to put yourself in that agent’s shoes and be so led right now. Whatever considerations will foreseeably later motivate you are equally moving considerations now.
In deep stock-markets, many traders are trying to make a buck by knowing something that no one else yet does. Because the current prices of various assets already take into consideration all public knowledge about them, you can’t expect to reliably make money as a trader unless you know something that the other traders don’t. Here, “knowing something” is basically synonymous with “being able to predict how public knowledge about the assets in question will evolve.” If a trader can foresee what everyone will believe in the near future, he can beat everyone else to it by betting on those expected beliefs now. Because of this, whenever a trader in a deep stock-market comes to know something about a relevant future belief, prices today are pushed around to reflect that belief that would’ve been held tomorrow. The deep stock-market is thus epistemically efficient: it factors in every relevant credence that any trader can presently scry, well before those credences become widely held.
You want to be like the epistemically efficient stock-market. Don’t leave any actionable knowledge on the table by turning a deaf ear to what your future self will think; don’t dutifully go through the motions of studying when you already know what you’ll think at the end of the process. Incorporate everything any good reasoning process would conclude right now; if you can currently see it, you’re already in a position to believe it. And so long as you keep doing that, the way your future self will update his credences should remain completely unpredictable to you.