I would say that there are some kinds of irrationality that will be self modified or subagented away, and others that will stay. A CDT agent will not make other CDT agents. A myopic agent, one that only cares about the next hour, will create a subagent that only cares about the first hour after it was created. (Aeons later it will have taken over the universe and put all the resources into time-travel and worrying that its clock is wrong.)
I am not aware of any irrationality that I would consider to make a safe, useful and stable under self modification—subagent creation.
″ I would say that there are some kinds of irrationality that will be self modified or subagented away, and others that will stay. ”
^ I agree; this is the point of my analogy with ordinal numbers.
A completely myopic agent (that doesn’t directly do planning over future time-steps, but only seeks to optimize its current decision) probably shouldn’t make any sub-agents in the first place (except incidentally).
I would say that there are some kinds of irrationality that will be self modified or subagented away, and others that will stay. A CDT agent will not make other CDT agents. A myopic agent, one that only cares about the next hour, will create a subagent that only cares about the first hour after it was created. (Aeons later it will have taken over the universe and put all the resources into time-travel and worrying that its clock is wrong.)
I am not aware of any irrationality that I would consider to make a safe, useful and stable under self modification—subagent creation.
″ I would say that there are some kinds of irrationality that will be self modified or subagented away, and others that will stay. ”
^ I agree; this is the point of my analogy with ordinal numbers.
A completely myopic agent (that doesn’t directly do planning over future time-steps, but only seeks to optimize its current decision) probably shouldn’t make any sub-agents in the first place (except incidentally).