I would like someone who understands Solomonoff Induction/the univeral prior/algorithmic probability theory to explain how the conclusions drawn in this post affect those drawn in this one. As I understand it, cousin_it’s post shows that the probability assigned by the univeral prior is not related to K-complexity; this basically negates the points Eliezer makes in Occam’s Razor and in this post. I’m pretty stupid with respect to mathematics, however, so I would like someone to clarify this for me.
Solomonoff’s universal prior assigns a probability to every individual Turing machine. Usually the interesting statements or hypotheses about which machine we are dealing with are more like “the 10th output bit is 1” than “the machine has the number 643653″. The first statement describes an infinite number of different machines, and its probability is the sum of the probabilities of those Turing machines that produce 1 as their 10th output bit (as the probabilities of mutually exclusive hypotheses can be summed). This probability is not directly related to the K-complexity of the statement “the 10th output bit is 1” in any obvious way. The second statement, on the other hand, has probability exactly equal to the probability assigned to the Turing machine number 643653, and its K-complexity is essentially (that is, up to an additive constant) equal to the K-complexity of the number 643653.
So the point is that generic statements usually describe a huge number of different specific individual hypotheses, and that the complexity of a statement needed to delineate a set of Turing machines is not (necessarily) directly related to the complexities of the individual Turing machines in the set.
I don’t think there’s very much conflict. The basic idea of cousin-it’s post is that the probabilities of generic statements are not described by a simplicity prior. Eliezer’s post is about the reasons why the probabilities of every mutually exclusive explanation for your data should look like a simplicity prior (an explanation is a sort of statement, but in order for the arguments to work, you can’t assign probabilities to any old explanations—they need to have this specific sort of structure).
I don’t think it’s a clear-cut issue. Algorithmic probability seems to be the justification for several Sequence posts, most notably this one and this one. But, again, I am stupid with respect to algorithmic probability theory and its applications.
Kolmogorov Complexity is defined with respect to a Turing complete language.
I think cousin_it is saying we should be careful what language our hypothesis is encoded in before checking its complexity. It’s easy to make mistakes when trying to explain something in terms of Solomonoff Induction.
The complexity of “A and B and C and D” is roughly equal to the complexity of “A or B or C or D”, but we know for certain that the former hypothesis can never be more probable than the latter, no matter what A, B, C and D are.
“A or B or C or D” is invalid to judge the complexity of directly because it is a set of different alternative hypothesis rather than a single one.
The hypothesis “the correct theory of everything is the lexicographically least algorithm with K-complexity 3^^^^3” is quite short, but the universal prior for it is astronomically low.
This one is invalid to judge the complexity of directly because K-complexity is not computable and the algorithm to compute it in this special case is very large. If the term K-complexity were expanded this statement would be astronomically complex.
The hypothesis “if my brother’s wife’s first son’s best friend flips a coin, it will fall heads” has quite high complexity, but should be assigned credence 0.5, just like its negation.
Here we must be careful to note that (presumably) all alternative low-complexity worlds where the “brother’s wife’s first son’s best friend” does not flip the coin have already been ruled out.
I would like someone who understands Solomonoff Induction/the univeral prior/algorithmic probability theory to explain how the conclusions drawn in this post affect those drawn in this one. As I understand it, cousin_it’s post shows that the probability assigned by the univeral prior is not related to K-complexity; this basically negates the points Eliezer makes in Occam’s Razor and in this post. I’m pretty stupid with respect to mathematics, however, so I would like someone to clarify this for me.
Solomonoff’s universal prior assigns a probability to every individual Turing machine. Usually the interesting statements or hypotheses about which machine we are dealing with are more like “the 10th output bit is 1” than “the machine has the number 643653″. The first statement describes an infinite number of different machines, and its probability is the sum of the probabilities of those Turing machines that produce 1 as their 10th output bit (as the probabilities of mutually exclusive hypotheses can be summed). This probability is not directly related to the K-complexity of the statement “the 10th output bit is 1” in any obvious way. The second statement, on the other hand, has probability exactly equal to the probability assigned to the Turing machine number 643653, and its K-complexity is essentially (that is, up to an additive constant) equal to the K-complexity of the number 643653.
So the point is that generic statements usually describe a huge number of different specific individual hypotheses, and that the complexity of a statement needed to delineate a set of Turing machines is not (necessarily) directly related to the complexities of the individual Turing machines in the set.
I don’t think there’s very much conflict. The basic idea of cousin-it’s post is that the probabilities of generic statements are not described by a simplicity prior. Eliezer’s post is about the reasons why the probabilities of every mutually exclusive explanation for your data should look like a simplicity prior (an explanation is a sort of statement, but in order for the arguments to work, you can’t assign probabilities to any old explanations—they need to have this specific sort of structure).
Stupid question: Does everyone agree that algorithmic probability is irrelevant to human epistemic practices?
I see it as a big open question.
I don’t think it’s a clear-cut issue. Algorithmic probability seems to be the justification for several Sequence posts, most notably this one and this one. But, again, I am stupid with respect to algorithmic probability theory and its applications.
Kolmogorov Complexity is defined with respect to a Turing complete language.
I think cousin_it is saying we should be careful what language our hypothesis is encoded in before checking its complexity. It’s easy to make mistakes when trying to explain something in terms of Solomonoff Induction.
“A or B or C or D” is invalid to judge the complexity of directly because it is a set of different alternative hypothesis rather than a single one.
This one is invalid to judge the complexity of directly because K-complexity is not computable and the algorithm to compute it in this special case is very large. If the term K-complexity were expanded this statement would be astronomically complex.
Here we must be careful to note that (presumably) all alternative low-complexity worlds where the “brother’s wife’s first son’s best friend” does not flip the coin have already been ruled out.
http://www.qwantz.com/index.php?comic=767&mobile=2