″ I would say that there are some kinds of irrationality that will be self modified or subagented away, and others that will stay. ”
^ I agree; this is the point of my analogy with ordinal numbers.
A completely myopic agent (that doesn’t directly do planning over future time-steps, but only seeks to optimize its current decision) probably shouldn’t make any sub-agents in the first place (except incidentally).
″ I would say that there are some kinds of irrationality that will be self modified or subagented away, and others that will stay. ”
^ I agree; this is the point of my analogy with ordinal numbers.
A completely myopic agent (that doesn’t directly do planning over future time-steps, but only seeks to optimize its current decision) probably shouldn’t make any sub-agents in the first place (except incidentally).