The idea of using an ordering function for knowledge is new to me, thanks!
For the example I think the order would look like this: A=a, S, U except for A, the source code of A.
The hard part is getting “U except for A”. Given the source code of U and the source code for A, we don’t know how to “factor” one program by another (or equivalently, find all correlates of the agent in the universe). If we knew how to do that, it would help a lot with UDT in general.
I guess you would actually have various knowledge items about the world, some of them implying things about A, and any that in conjunction with the others so far cause a contradiction with A=a that the agent can find would be discarded. Maybe that already would be enough; I’m not sure.
That’s a really difficult question. It’s hard to say what principles humans follow when evaluating counterfactuals, and even harder to say in how far that’s a reasonable example to follow.
I think higher level observational laws should usually have a higher priority than concrete data points they are based on, and all else equal they should be in descending order of generality and confidence. That the US. president can veto US federal legislation and that the person who can veto US federal legislation is the same person as the commander in chief of the US military forces should both have a higher priority than that George W. Bush could veto US federal legislation.
It would also depend on what the counterfactual is used for. For counterfactuals concerning the past timing would obviously extremely important.
In the case of considering the counterfactual implications of a decision the agent makes you could try ascending order of strength as Bayesian evidence about the agent as a secondary criterion, maybe? Or perhaps instead ratio of that strength to general importance? (Which would probably require nested counterfactuals? Are we concerned with computability yet?)
EDIT: I think the knowledge items would have redundancy so that even if the agent can derive itself directly from the laws of physics and needs to reject (one of) them it can reconstruct almost normal physics from various observational laws. It also seems redundancy could reduce the importance of the initial order somewhat.
The idea of using an ordering function for knowledge is new to me, thanks!
The hard part is getting “U except for A”. Given the source code of U and the source code for A, we don’t know how to “factor” one program by another (or equivalently, find all correlates of the agent in the universe). If we knew how to do that, it would help a lot with UDT in general.
I guess you would actually have various knowledge items about the world, some of them implying things about A, and any that in conjunction with the others so far cause a contradiction with A=a that the agent can find would be discarded. Maybe that already would be enough; I’m not sure.
What considerations should be used to order the knowledge items?
That’s a really difficult question. It’s hard to say what principles humans follow when evaluating counterfactuals, and even harder to say in how far that’s a reasonable example to follow.
I think higher level observational laws should usually have a higher priority than concrete data points they are based on, and all else equal they should be in descending order of generality and confidence. That the US. president can veto US federal legislation and that the person who can veto US federal legislation is the same person as the commander in chief of the US military forces should both have a higher priority than that George W. Bush could veto US federal legislation.
It would also depend on what the counterfactual is used for. For counterfactuals concerning the past timing would obviously extremely important.
In the case of considering the counterfactual implications of a decision the agent makes you could try ascending order of strength as Bayesian evidence about the agent as a secondary criterion, maybe? Or perhaps instead ratio of that strength to general importance? (Which would probably require nested counterfactuals? Are we concerned with computability yet?)
EDIT: I think the knowledge items would have redundancy so that even if the agent can derive itself directly from the laws of physics and needs to reject (one of) them it can reconstruct almost normal physics from various observational laws. It also seems redundancy could reduce the importance of the initial order somewhat.