I think we have to be really careful when bringing AIXI/Kolmogorov complexity to the discussion. My strong intuition is that AIXI is specifically the type of alien intelligence we want to avoid here. Most programming languages have very short programs that behave in wild and complex ways when analyzed from a human perspective. To an AIXI mind based on those programming languages this kind of behavior is by definition simple and preferred by Occam’s Razor. The generalizations to n+1 dimensions that this kind of an intelligence makes are almost guaranteed to be incomprehensible and totally alien to a human mind.
I’m not saying that information theory is a bad approach in itself. I’m just saying that we have to be really careful what kind of primitives we want to give our AI to reason with. It’s important to remember that even though human concepts are only a constant factor away from concepts based on Turing machines, those constant factors are huge and the behavior and reasoning of an information theoretic mind are always exponentially dominated by what it finds simple in it’s own terms.
I think we have to be really careful when bringing AIXI/Kolmogorov complexity to the discussion. My strong intuition is that AIXI is specifically the type of alien intelligence we want to avoid here. Most programming languages have very short programs that behave in wild and complex ways when analyzed from a human perspective. To an AIXI mind based on those programming languages this kind of behavior is by definition simple and preferred by Occam’s Razor. The generalizations to n+1 dimensions that this kind of an intelligence makes are almost guaranteed to be incomprehensible and totally alien to a human mind.
I’m not saying that information theory is a bad approach in itself. I’m just saying that we have to be really careful what kind of primitives we want to give our AI to reason with. It’s important to remember that even though human concepts are only a constant factor away from concepts based on Turing machines, those constant factors are huge and the behavior and reasoning of an information theoretic mind are always exponentially dominated by what it finds simple in it’s own terms.