Random thought after reading “A model of UDT with a halting oracle”: imagine there are two super-intelligent AIs A and B, suitably modified to have access to their own and each other’s source codes. They are both competing to submit a python code of length at most N which prints the larger number, then halts (where N is orders of magnitude larger than the code lengths of A and B). A can try to “cheat” by submitting something like exec(run B on the query “submit a code of length N that prints a large number, then halts”) then print(0), but B can do this as well. Supposing they must submit to a halting oracle that will punish any AI that submits a non-halting program, what might A and B do?
Random thought after reading “A model of UDT with a halting oracle”: imagine there are two super-intelligent AIs A and B, suitably modified to have access to their own and each other’s source codes. They are both competing to submit a python code of length at most N which prints the larger number, then halts (where N is orders of magnitude larger than the code lengths of A and B). A can try to “cheat” by submitting something like exec(run B on the query “submit a code of length N that prints a large number, then halts”) then print(0), but B can do this as well. Supposing they must submit to a halting oracle that will punish any AI that submits a non-halting program, what might A and B do?
submit busy beaver
edit: wait nevermind, busy beaver itself takes a halting oracle to implement
Has to be a python code; allowing arbitrary non-computable natural language descriptions gets hairy fast