NFL theorems are about max-entropy worlds. Solomonoff induction works on highly lawful, simplicity-biased, low-entropy worlds.
The same (or a similar) point applies. If you limit yourself to the set of lawful worlds and use an Occamian prior, you will start off much worse than an algorithm that implictly assumes a prior that’s close to the true distribution. As Solomonoff induction works its way up through longer algorithms, it will hit some that run into an infinite loop. Even if you program a constraint that gets it past or out of these, the optimality is only present “after a long time”, which, in practice, means later than we need or want the results.
If you could actually do Solomonoff induction, you would become at least as smart as a human baby in roughly 0 seconds (some rounding error may have occurred).
What else can you tell us about the implications of being able to compute uncomputable functions?
As Solomonoff induction works its way up through longer algorithms, it will hit some that run into an infinite loop. Even you program a constraint that gets it past or out of these, the optimality is only present “after a long time”, which, in practice, means later than we need or want the results.
You are arguing against a strawman: it’s not obvious that there are no algorithms that approximate Solomonoff induction well enough in practical cases. Of course there are silly implementations that are way worse than magical oracles.
it’s not obvious that there are no algorithms that approximate Solomonoff induction well enough in practical cases.
Right, but any such approximation works by introducing a prior about which functions it can skip over. And for such knowledge to actually speed it up, it must involve knowledge (gained separately from S/I) about the true distribution.
But at that point, you’re optimizing for a narrower domain, not implementing universal intelligence. (In my naming convention, you’re bringing in type 2 intelligence.)
You can’t “speed up” an uncomputable non-algorithm.
Okay, we’re going in circles. You had just mentioned possible computable algorithms that approximate Solomonoff induction.
it’s not obvious that there are no algorithms that approximate Solomonoff induction well enough in practical cases. [emphasis added—SB]
So, we were talking about approximating algorithms. The point I was making, in response to this argument that “well, we can have working algorithms that are close enough to S/I”, was that to do so, you have to bring in knowledge of the distribution gained some other way, at which point it is no longer universal. (And, in which case talk of “speeding up” is meaningful.)
Demonstrating my point that universal intelligence has its limits and must combine with intelligence in a different sense of the term.
You introduce operations on the approximate algorithms (changing the algorithm by adding data), something absent from the original problem. What doesn’t make sense is to compare “speed” of non-algorithmic specification with the speed of algorithmic approximations. And absent any approximate algorithms, it’s also futile to compare their speed, much less propose mechanisms for their improvement that assume specific structure of these absent algorithms (if you are not serious about exploring the design space in this manner to obtain actual results).
You introduce operations on the approximate algorithms (changing the algorithm by adding data), something absent from the original problem.
What you call “the original problem” (pure Solomonoff induction) isn’t. It’s not a problem. It can’t be done, so it’s a moot point.
What doesn’t make sense is to compare “speed” of non-algorithmic specification with the speed of algorithmic approximations
Sure it does. The uncomputable Solomonoff induction has a speed of zero. Non-halting approximations have a speed greater than zero. Sounds comparable to me for the purposes of this discussion.
And absent any approximate algorithms, it’s also futile to compare their speed, much less propose mechanisms for their improvement that assume specific structure of these absent algorithms (if you are not serious about exploring the design space in this manner to obtain actual results).
There are approximate algorithms. Even Bayesian inference counts. And my point is that any time you add something to modify Solomonoff induction to make it useful is, directly or indirectly, introducing a prior unique to the search space—cleary showing the distinctness of type 2 intelligence.
I don’t understand why you’d continue arguing definitions about speed of Solomonoff induction or it being “the original problem”. It’s clear what we both mean.
I believe you are wrong about general statements about what needs to be done to implement approximate Solomonoff induction. Since we don’t technically define in what sense this approximation has to be general, there remain possibilities for a good technical definition that preserves “generality” in an approximate implementation.
don’t understand why you’d continue arguing definitions about speed of Solomonoff induction or it being “the original problem”. It’s clear what we both mean.
A better question would be why you brought up the issue. We both knew what the other meant before that, but you kept bringing it up.
I believe you are wrong … there remain possibilities for a good technical definition that preserves “generality” in an approximate implementation.
Okay, well, I’ll believe it when I see it. In the mean time, I suspect it will be far more productive to exploit whatever regularity we already know about the environment, and work on building that into the inference program’s prior. (Arguably, even the Occamian prior does that by using our hard-won belief in the universe’s preference for simplicity!)
The same (or a similar) point applies. If you limit yourself to the set of lawful worlds and use an Occamian prior, you will start off much worse than an algorithm that implictly assumes a prior that’s close to the true distribution. As Solomonoff induction works its way up through longer algorithms, it will hit some that run into an infinite loop. Even if you program a constraint that gets it past or out of these, the optimality is only present “after a long time”, which, in practice, means later than we need or want the results.
What else can you tell us about the implications of being able to compute uncomputable functions?
You are arguing against a strawman: it’s not obvious that there are no algorithms that approximate Solomonoff induction well enough in practical cases. Of course there are silly implementations that are way worse than magical oracles.
Right, but any such approximation works by introducing a prior about which functions it can skip over. And for such knowledge to actually speed it up, it must involve knowledge (gained separately from S/I) about the true distribution.
But at that point, you’re optimizing for a narrower domain, not implementing universal intelligence. (In my naming convention, you’re bringing in type 2 intelligence.)
It introduces a prior, period. Not a prior about “skipping over”. Universal induction doesn’t have to “run” anything in a trivial manner.
You can’t “speed up” an uncomputable non-algorithm.
Okay, we’re going in circles. You had just mentioned possible computable algorithms that approximate Solomonoff induction.
So, we were talking about approximating algorithms. The point I was making, in response to this argument that “well, we can have working algorithms that are close enough to S/I”, was that to do so, you have to bring in knowledge of the distribution gained some other way, at which point it is no longer universal. (And, in which case talk of “speeding up” is meaningful.)
Demonstrating my point that universal intelligence has its limits and must combine with intelligence in a different sense of the term.
You introduce operations on the approximate algorithms (changing the algorithm by adding data), something absent from the original problem. What doesn’t make sense is to compare “speed” of non-algorithmic specification with the speed of algorithmic approximations. And absent any approximate algorithms, it’s also futile to compare their speed, much less propose mechanisms for their improvement that assume specific structure of these absent algorithms (if you are not serious about exploring the design space in this manner to obtain actual results).
What you call “the original problem” (pure Solomonoff induction) isn’t. It’s not a problem. It can’t be done, so it’s a moot point.
Sure it does. The uncomputable Solomonoff induction has a speed of zero. Non-halting approximations have a speed greater than zero. Sounds comparable to me for the purposes of this discussion.
There are approximate algorithms. Even Bayesian inference counts. And my point is that any time you add something to modify Solomonoff induction to make it useful is, directly or indirectly, introducing a prior unique to the search space—cleary showing the distinctness of type 2 intelligence.
To wrap up (as an alternative to not replying):
I don’t understand why you’d continue arguing definitions about speed of Solomonoff induction or it being “the original problem”. It’s clear what we both mean.
I believe you are wrong about general statements about what needs to be done to implement approximate Solomonoff induction. Since we don’t technically define in what sense this approximation has to be general, there remain possibilities for a good technical definition that preserves “generality” in an approximate implementation.
A better question would be why you brought up the issue. We both knew what the other meant before that, but you kept bringing it up.
Okay, well, I’ll believe it when I see it. In the mean time, I suspect it will be far more productive to exploit whatever regularity we already know about the environment, and work on building that into the inference program’s prior. (Arguably, even the Occamian prior does that by using our hard-won belief in the universe’s preference for simplicity!)