The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent’s posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.
That looks like the same position that Eliezer took, and I think I already refuted it. Let me know if you’ve read the one-logic thread and found my argument wrong or unconvincing.
The idea is that universal prior is really about observation-predicting algorithms that agents run, and not about prediction of what will happen in the world. So, for any agent that runs a given anticipation-defining algorithm and rewards/punishes the universal prior-based agent according to it, we have an anticipation-computing program that will obtain higher and higher probability in the universal prior-based agent.
This by the way again highlights the distinction between what will actually happen, and what a person anticipates—predictions are about capturing the concept of anticipation, an aspect of how people think, and are not about what in fact can happen.
That looks like the same position that Eliezer took, and I think I already refuted it. Let me know if you’ve read the one-logic thread and found my argument wrong or unconvincing.
The idea is that universal prior is really about observation-predicting algorithms that agents run, and not about prediction of what will happen in the world. So, for any agent that runs a given anticipation-defining algorithm and rewards/punishes the universal prior-based agent according to it, we have an anticipation-computing program that will obtain higher and higher probability in the universal prior-based agent.
This by the way again highlights the distinction between what will actually happen, and what a person anticipates—predictions are about capturing the concept of anticipation, an aspect of how people think, and are not about what in fact can happen.