Is there a particular reason to define U as a constant? What if it was defined as a function, taking the agent function as its argument? For example:
U(#X) = (a, 0) if X(#U, 0) = C; (b, 0) if X(#U, 0) = D
Then A could be defined as: A(#X, i) = y, such that there exists #z, such that z(#X, i)=y and forall #z’ p_i(X(#z)) >= p_i(X(#z’)) [Some uniquness condition on y can be added for the cases where several actions may have equal maximal utility]
Then it should be provable that a>b ⇒ A(#U,0)=C and a A(#U,0)=D
This should also work with the toy PD. (But not with the real one—the agent’s definition won’t work because it is impossible to impersonate some other agent if you have to present your own Goedelian number).
Continuing from here...
Is there a particular reason to define U as a constant? What if it was defined as a function, taking the agent function as its argument? For example:
U(#X) =
(a, 0) if X(#U, 0) = C;
(b, 0) if X(#U, 0) = D
Then A could be defined as:
A(#X, i) = y, such that there exists #z, such that z(#X, i)=y and forall #z’ p_i(X(#z)) >= p_i(X(#z’))
[Some uniquness condition on y can be added for the cases where several actions may have equal maximal utility]
Then it should be provable that a>b ⇒ A(#U,0)=C and a A(#U,0)=D
This should also work with the toy PD. (But not with the real one—the agent’s definition won’t work because it is impossible to impersonate some other agent if you have to present your own Goedelian number).