I view that as more of an interesting discussion than entirely a criticism. I just gave it a reread—he raises a lot of good points, but there’s not exactly a central argument distinct from the ones I addressed as far as I can tell? He is mainly focused on digging into embeddedness issues, particularly discussing things I’d classify as “pain sensors” to prevent AIXI from destroying itself. My solution to this here is a little more thorough than the one that the pro-AIXI speaker comes up with.
The discussion of death is somewhat incorrect because it doesn’t consider Turing machines which (while never halting) produce only a finite percept sequence and then “hang” or loop indefinitely. This can be viewed as death and may be considered likely in some cases. Here is a paper on it.
The other criticism is that AIXI doesn’t self-improve—I mean, it learns of course, but doesn’t edit its own source code. There may be hacky ways around this but basically I agree—that’s just not the point of the AIXI model. It’s a specification for optimal intelligence and an optimal intelligence does not need to self-improve. Perhaps self-improvement is better viewed as a method of bootstrapping a weak AIXI approximation into a better one using external conceptual tools. It’s probably not a necessary ingredient up to human level though; certainly modern LLMs do not self-improve (yet) and since they are pretty much black-boxes it’s not clear that they will be able to until well past the point where they are smart enough to be dangerous.
I view that as more of an interesting discussion than entirely a criticism. I just gave it a reread—he raises a lot of good points, but there’s not exactly a central argument distinct from the ones I addressed as far as I can tell? He is mainly focused on digging into embeddedness issues, particularly discussing things I’d classify as “pain sensors” to prevent AIXI from destroying itself. My solution to this here is a little more thorough than the one that the pro-AIXI speaker comes up with.
The discussion of death is somewhat incorrect because it doesn’t consider Turing machines which (while never halting) produce only a finite percept sequence and then “hang” or loop indefinitely. This can be viewed as death and may be considered likely in some cases. Here is a paper on it.
The other criticism is that AIXI doesn’t self-improve—I mean, it learns of course, but doesn’t edit its own source code. There may be hacky ways around this but basically I agree—that’s just not the point of the AIXI model. It’s a specification for optimal intelligence and an optimal intelligence does not need to self-improve. Perhaps self-improvement is better viewed as a method of bootstrapping a weak AIXI approximation into a better one using external conceptual tools. It’s probably not a necessary ingredient up to human level though; certainly modern LLMs do not self-improve (yet) and since they are pretty much black-boxes it’s not clear that they will be able to until well past the point where they are smart enough to be dangerous.