Let us say I am an AI, I want to replace myself with a new program, I want to be sure that this new program will perform the tasks I want done better than me (including creating new better copies of myself), I need to be able to predict how good a program is.
I don’t want to have to run it and see as I will have to run it for a very long time to see if the program is better than me a long time in the future. So I want a proof that the new program is better than me. Are such proofs possible? My argument is that they are not if you cannot constrain the environment and make it not reference your proof.
Let us say I am an AI, I want to replace myself with a new program, I want to be sure that this new program will perform the tasks I want done better than me (including creating new better copies of myself), I need to be able to predict how good a program is.
I don’t want to have to run it and see as I will have to run it for a very long time to see if the program is better than me a long time in the future. So I want a proof that the new program is better than me. Are such proofs possible? My argument is that they are not if you cannot constrain the environment and make it not reference your proof.