Warning: this is not a part of the report I’m confident I understand all that well; I’m trying anyway and Paul/Mark can correct me if I messed something up here.
I think the idea here is like:
We assume there’s some actual true correspondence between the AI Bayes net and the human Bayes net (because they’re describing the same underlying reality that has diamonds and chairs and tables in it).
That means that if we have one of the Bayes nets, and the true correspondence, we should be able to use that rederive the other Bayes net. In particular the human Bayes net plus the true correspondence should let us reconstruct the AI Bayes net; false correspondences that just do inference from observations in the human Bayes net wouldn’t allow us to do this since they throw away all the intermediate info derived by the AI Bayes net.
If you assume that the human Bayes net plus the true correspondence are simpler than the AI Bayes net, then this “compresses” the AI Bayes net because you just wrote down a program that’s smaller than the AI Bayes net which “unfolds” into the AI Bayes net.
This is why the counterexample in that section focuses on the case where the AI Bayes net was already so simple to describe that there was nothing left to compress, and the human Bayes net + true correspondence had to be larger.
Warning: this is not a part of the report I’m confident I understand all that well; I’m trying anyway and Paul/Mark can correct me if I messed something up here.
I think the idea here is like:
We assume there’s some actual true correspondence between the AI Bayes net and the human Bayes net (because they’re describing the same underlying reality that has diamonds and chairs and tables in it).
That means that if we have one of the Bayes nets, and the true correspondence, we should be able to use that rederive the other Bayes net. In particular the human Bayes net plus the true correspondence should let us reconstruct the AI Bayes net; false correspondences that just do inference from observations in the human Bayes net wouldn’t allow us to do this since they throw away all the intermediate info derived by the AI Bayes net.
If you assume that the human Bayes net plus the true correspondence are simpler than the AI Bayes net, then this “compresses” the AI Bayes net because you just wrote down a program that’s smaller than the AI Bayes net which “unfolds” into the AI Bayes net.
This is why the counterexample in that section focuses on the case where the AI Bayes net was already so simple to describe that there was nothing left to compress, and the human Bayes net + true correspondence had to be larger.