If β is set to be so near 1 that the exponential time simulation of real physics can have the highest posterior within a reasonable time...
Epistemic status: shady
So I’m a bit baffled by the philosophy here, but here’s why I haven’t been concerned with the long time it would take BoMAI to entertain the true environment (and it might well, given a safe value of β).
There is relatively clear distinction one can make between objective probabilities and subjective ones. The asymptotic benignity result makes use of world-models that perfectly match the objective probabilities rising to the top.
Consider a new kind of probability: a “k-optimal subjective probability.” That is, the best (in the sense of KL divergence) approximation of the objective probabilities that can be sampled from using a UTM and using only k computation steps. Suspend disbelief for a moment, and suppose we thought of these probabilities as objective probabilities. My intuition here is that everything works just great when agents treat subjective probabilities like real probabilities, and to a k-bounded agent, it feels like there is some sense in which these might as well be objective probabilities; the more intricate structure is inaccessible. If no world-models were considered that allowed more than k computation steps per timestep (mk per episode I guess, whatever), then just by calling “k-optimal subjective probabilities” “objective,” the same benignity theorems would apply, where the role in the proofs of [the world-model that matches the objective probabilities] is replaced by [the world-model that matches the k-optimal subjective probabilities]. And in this version, i0 comes much sooner, and the limiting value of intelligence is reached much sooner.
Of course, “the limiting value of intelligence” is much less, because only fast world-models are considered. But that just goes to show that even if, on a human timescale, BoMAI basically never fields a world-model that actually matches objective probabilities, along the way, it will still be fielding the best ones available that use a more modest computation budget. Once the computation budget surpasses the human brain, that should suffice for it to be practically intelligent.
EDIT: if one sets β to be safe, then if this logic fails, BoMAI will be useless, not dangerous.
Epistemic status: shady
So I’m a bit baffled by the philosophy here, but here’s why I haven’t been concerned with the long time it would take BoMAI to entertain the true environment (and it might well, given a safe value of β).
There is relatively clear distinction one can make between objective probabilities and subjective ones. The asymptotic benignity result makes use of world-models that perfectly match the objective probabilities rising to the top.
Consider a new kind of probability: a “k-optimal subjective probability.” That is, the best (in the sense of KL divergence) approximation of the objective probabilities that can be sampled from using a UTM and using only k computation steps. Suspend disbelief for a moment, and suppose we thought of these probabilities as objective probabilities. My intuition here is that everything works just great when agents treat subjective probabilities like real probabilities, and to a k-bounded agent, it feels like there is some sense in which these might as well be objective probabilities; the more intricate structure is inaccessible. If no world-models were considered that allowed more than k computation steps per timestep (mk per episode I guess, whatever), then just by calling “k-optimal subjective probabilities” “objective,” the same benignity theorems would apply, where the role in the proofs of [the world-model that matches the objective probabilities] is replaced by [the world-model that matches the k-optimal subjective probabilities]. And in this version, i0 comes much sooner, and the limiting value of intelligence is reached much sooner.
Of course, “the limiting value of intelligence” is much less, because only fast world-models are considered. But that just goes to show that even if, on a human timescale, BoMAI basically never fields a world-model that actually matches objective probabilities, along the way, it will still be fielding the best ones available that use a more modest computation budget. Once the computation budget surpasses the human brain, that should suffice for it to be practically intelligent.
EDIT: if one sets β to be safe, then if this logic fails, BoMAI will be useless, not dangerous.