“Only people with mistaken ideas about “rationality” will find themselves believing that “irrationality” will help.”
Conjecture: Take two arbitrary, identical optimization processes. Pick some verifiable piece of information A. Give the first process the belief “A”, and the second process the belief “~A”. The average end utility of the first process will always be equal to or greater than the average end utility of the second process.
Attempted disproof: Suppose that both processes were programmed by naive human rationalists, who give the processes the supergoal of correcting mistaken beliefs. Also suppose that the universe is finite, and has a finite complexity. Both processes will improve their models of the universe, in accordance with the laws of rationality, until they eventually reach an asymptote due to the finite amount of modelable information. Because the utility is determined by the amount of new information that had to be added to the model, the second process will have a higher end utility, as it had to add the information “A” to correct the false information “~A”.
“Only people with mistaken ideas about “rationality” will find themselves believing that “irrationality” will help.”
Conjecture: Take two arbitrary, identical optimization processes. Pick some verifiable piece of information A. Give the first process the belief “A”, and the second process the belief “~A”. The average end utility of the first process will always be equal to or greater than the average end utility of the second process.
Attempted disproof: Suppose that both processes were programmed by naive human rationalists, who give the processes the supergoal of correcting mistaken beliefs. Also suppose that the universe is finite, and has a finite complexity. Both processes will improve their models of the universe, in accordance with the laws of rationality, until they eventually reach an asymptote due to the finite amount of modelable information. Because the utility is determined by the amount of new information that had to be added to the model, the second process will have a higher end utility, as it had to add the information “A” to correct the false information “~A”.