A simple remark: we don’t have access to all of E, only E up until the current time.
So we have to make sure that we don’t get a degenerate pair which diverges wildly from the actual universe at some point in the future.
Maybe this is similar to the fact that we don’t want AIs to diverge from human values once we go off-distribution? But you’re definitely right that there’s a difference: we do want AIs to diverge from human behaviour (even in common situations).
A simple remark: we don’t have access to all of E, only E up until the current time. So we have to make sure that we don’t get a degenerate pair which diverges wildly from the actual universe at some point in the future.
Maybe this is similar to the fact that we don’t want AIs to diverge from human values once we go off-distribution? But you’re definitely right that there’s a difference: we do want AIs to diverge from human behaviour (even in common situations).