Shane, suppose your super-GZIP program was searching a space of arbitrary compressive Turing machines (only not classic TMs, efficient TMs) and it discovered an algorithm that was really good at predicting future input from past input, much better than all the standard algorithms built into its library. This is because the algorithm turns out to contain (a) a self-improving (unFriendly) AI or (b) a program that hacked the “safe” AI’s Internet connection (it doesn’t have any goals, right?) to take over unguarded machines or (c) both.
Wha? If I have a theorem prover, and run a search over all compressor algorithms trying to find/prove the ones with high “efficiency” (some function of its asymptotics of running time and output size), I expect to never create an unfriendly AI that takes over the Internet.
Shane, suppose your super-GZIP program was searching a space of arbitrary compressive Turing machines (only not classic TMs, efficient TMs) and it discovered an algorithm that was really good at predicting future input from past input, much better than all the standard algorithms built into its library. This is because the algorithm turns out to contain (a) a self-improving (unFriendly) AI or (b) a program that hacked the “safe” AI’s Internet connection (it doesn’t have any goals, right?) to take over unguarded machines or (c) both.
Wha? If I have a theorem prover, and run a search over all compressor algorithms trying to find/prove the ones with high “efficiency” (some function of its asymptotics of running time and output size), I expect to never create an unfriendly AI that takes over the Internet.