The mistake here is the assumption that a program that models the world better necessarily has a higher Kolmogorov complexity.
I think Anton assumes that we have the simplest program that predicts the world to a given standard, in which case this is not a mistake. He doesn’t explicitly say so, though, so I think we should wait for clarification.
But it’s a strange assumption; I don’t see why the minimum complexity predictor couldn’t carry out what we would interpret as RSI in the process of arriving at its prediction.
The thing about the Pareto frontier of Kolmogorov complexity vs prediction score is that most programs aren’t on it. In particular, it seems unlikely that p_1, the seed AI written by humans, is going to be on the frontier. Even p_2, the successor AI, might not be on it either. We can’t equovicate between all programs that get the same prediction score, differences between them will be observable in the way they make predictions.
I think Anton assumes that we have the simplest program that predicts the world to a given standard, in which case this is not a mistake. He doesn’t explicitly say so, though, so I think we should wait for clarification.
But it’s a strange assumption; I don’t see why the minimum complexity predictor couldn’t carry out what we would interpret as RSI in the process of arriving at its prediction.
The thing about the Pareto frontier of Kolmogorov complexity vs prediction score is that most programs aren’t on it. In particular, it seems unlikely that p_1, the seed AI written by humans, is going to be on the frontier. Even p_2, the successor AI, might not be on it either. We can’t equovicate between all programs that get the same prediction score, differences between them will be observable in the way they make predictions.
I don’t disagree with any of what you say here—I just read Anton as assuming we have a program on that frontier