Intuitively, this limitation could be addressed by hooking up the AIXItl’s output channel to its source code. Unfortunately, if you do that, the resulting formalism is no longer AIXItl.
I dispute this. Any robot which instantiates AIXI-tl must consist of two parts: First, there must be a component which performs the actual computations for AIXI-tl. Second, there is a router, which observes the robot’s environment and feeds it to the first compoment as input, and also reads the first component’s output and translates it into an action the robot performs. The design of the router must be neccessity make additional arbitrary choices not present in the pure description of AIXI-tl. For example, the original description of AIXI described the output as a bit-string, which in this scenario must somehow be converted into a constree for the output register. If the router is badly designed then it can make problems that no program of any intelligence can overcome. For example, imagine the router can’t perform the action ‘move right’.
The problem described here is not at all in AIXI-tl, but entirely in the design of the router. This can be seen from how at no point you look into the internal components of AIXI-tl or what output it would generate. If you allowed the router to change the internal registers of the robot, it would still be AIXI-tl, just that it would have a different output router.
I think that if the robot use such a router then it would kill itself in experimentation before it would have the chance to solve the problem, but you haven’t established that. I would like to see an argument against AIXI-tl that does not really rely what it is or is not physically capable of doing, but rather on what it is intelligent enough to choose to do. After all, humans, despite supposedly being capable of “naturalized induction”, would not do well in this problem either. A human cannot by force of will reprogram her brain into a static set of commands, nor can she make her brain stop emitting heat.
Finally, I want to say why I am making these arguments. It is not because I want to advocate for AIXI-tl and argue for its intelligence. The way I think of it AIXI is the dumbest program that is still capable of learning the right behavior eventually. Actually it’s worse than that; my argument here has convince me that even with exponential resources AIXI-tl can’t argue itself out of a paper bag (Note argument does look into the internals of AIXI-tl rather than treating it as a black-box). So if anything I think you might be overestimating the intelligence of AIXI-tl. However, my concern is that in addition to its usual stupidity, you think AIXI-tl has an additional obstacle in terms of some sort of ‘Cartesian boundary problem’, and that there exists some sort of ‘naturalized induction’ which humans have and which AIXI and AIXI-tl don’t have. I am unconvinced by this, and I think it is an unproductive line of research. Rather, I think any problem AIXI has in reasoning about itself is either one humans also have in reasoning about themselves or analogous to a problem it has reasoning about other things. In this case it is a problem humans also have.
Unfortunately, if you do that, the resulting formalism is no longer AIXItl.
I meant it literally and formally. The resulting machine may or may not be smart, but regardless, it does not necessarily obey the AIXI equations after the first timestep, and it lacks some of the formal properties of the AIXItl model. The AIXItl model assumes that the machine will continue to be AIXItl. Proofs about how AIXItl behaves (e.g., that its environment model improves over time) do not apply to AIXItls that can modify their code. I don’t know what properties this variant has, but I’m not yet convinced that they are nice ones.
Perhaps the variant can still act intelligently. Perhaps it cannot. Perhaps there’s a clever way to train it so that it happens to work in this particular game. I don’t know. My point is only that AIXItl was designed with an optimality property in mind (“superior to any other time t and length l bounded algorithm”), and that an embodied AIXItl lacks this property (regardless of the “router”).
This post attempts to explain why AIXItl is not a superior agent, and attempts to impart an intuition for why the field of AGI is not reducible to constructing better and better approximations of AIXI. It sounds like you already believe this point, so I won’t try to convince you of it further :-)
Rather, I think any problem AIXI has in reasoning about itself is either one humans also have in reasoning about themselves or analogous to a problem it has reasoning about other things. In this case it is a problem humans also have.
Be that as it may, the argument “humans can’t do X” is not a compelling reason to stop caring about X. It seems to me that an ideal artificial agent should be able to win at the HeatingUp game.
However, my concern is that in addition to its usual stupidity, you think AIXI-tl has an additional obstacle in terms of some sort of ‘Cartesian boundary problem’, and that there exists some sort of ‘naturalized induction’ which humans have and which AIXI and AIXI-tl don’t have. I am unconvinced by this, and I think it is an unproductive line of research.
I have made no claims in this post about research strategy, I’m only trying to point out why AIXI is not an ideal (as this concept still seems foreign to many people). Again, it seems you are already on board with this idea.
I hear your complaints about what you’ve assumed are my research goals, but I think we should save the research strategy discussion for another place and time :-)
Ack! I’m not sure what to think. When I wrote that comment, I had the impression that we had some sort of philosophical conflict, and I felt like I should make the case for my side. However, now I worry the comment was too aggressive. Moreover, it seems like we agree on most of the questions we can state precisely. I’m not sure how to deal with this situation.
I suppose I could turn some assumptions into questions: To what extent is it your goal in this inquiry to figure out ‘naturalized induction’? Do you think ‘naturalized induction’ is something humans naturally do when thinking, perhaps imperfectly?
However, now I worry the comment was too aggressive.
No worries :-)
To what extent is it your goal in this inquiry to figure out ‘naturalized induction’?
Zero. To be honest, I don’t spend much time thinking about AIXI. My inclination with regards to AIXI is to shrug and say “it’s not ideal for all the obvious reasons, and I can’t use it to study self-modification”, and then move on.
However, it turns out that what I think are the “obvious reasons” aren’t so obvious to some. While I’m not personally confident that AIXI can be modified to be useful for studying self-modification, ignoring AIXI entirely isn’t the most cunning strategy for forming relationships with other AGI researchers (who are researching different parts of the problem, and for whom AIXI may indeed be quite interesting and relevant).
If anything, my “goal with this inquiry” is to clearly sketch specific problems with AIXI that make it less useful to me and point towards directions where I’d be happy to discuss collaboration with researchers who are interested in AIXI.
It is not the case that I’m working on these problems in my free time: left to my own devices, I just use (or develop) toy models that better capture the part of the problem space I care about.
Do you think ‘naturalized induction’ is something humans naturally do when thinking, perhaps imperfectly?
I really don’t want to get dragged into a strategy discussion here. I’ll state a few points that I expect we both agree upon, but forgive me if I don’t answer further questions in this vein during this discussion.
Solomonoff induction would have trouble (or, at least, be non-optimal) in an uncomputable universe.
We’ve been pretty wrong about the rules of the universe in the past. (I wouldn’t have wanted scientists in 1750 to gamble on the universe being deterministic/single-branch, and I similarly don’t want scientists today to gamble on the universe being computable.)
Intuitively, it seems like there should be a computable program that can discover it’s inside an exotic universe (where ‘exotic’ includes ‘uncomputable’, but is otherwise a vague placeholder word).
I don’t think discussing how humans deal with this problem is relevant. Are there ways the universe could be that I can’t conceive of? Almost certainly. Can I figure out the laws of my universe as well as a perfect Solomonoff inductor? Probably not. Yet it does feel like I could be convinced that the universe is uncomputable, and so Solomonoff induction is probably not an idealization of whatever it is that I’m trying to do.
I don’t personally view this as an induction problem, but rather as a priors problem. And though I do indeed think it’s a problem, I’ll note that this does not imply that the problem captures any significant fraction of my research efforts.
A human cannot by force of will [...] make her brain stop emitting heat.
Disputed! I’m pretty sure if somebody studied under controlled conditions for enough years they’d be able to induce mild hypothermia through some sort of meditative exercise.
More realistically, albeit not through willpower alone, a human could sit in a bathtub full of ice water, dose themselves with ethanol or sodium thiopental, or otherwise use external, independently-verifiable means to temporarily reduce their cognitive capacity.
I dispute this. Any robot which instantiates AIXI-tl must consist of two parts: First, there must be a component which performs the actual computations for AIXI-tl. Second, there is a router, which observes the robot’s environment and feeds it to the first compoment as input, and also reads the first component’s output and translates it into an action the robot performs. The design of the router must be neccessity make additional arbitrary choices not present in the pure description of AIXI-tl. For example, the original description of AIXI described the output as a bit-string, which in this scenario must somehow be converted into a constree for the output register. If the router is badly designed then it can make problems that no program of any intelligence can overcome. For example, imagine the router can’t perform the action ‘move right’.
The problem described here is not at all in AIXI-tl, but entirely in the design of the router. This can be seen from how at no point you look into the internal components of AIXI-tl or what output it would generate. If you allowed the router to change the internal registers of the robot, it would still be AIXI-tl, just that it would have a different output router.
I think that if the robot use such a router then it would kill itself in experimentation before it would have the chance to solve the problem, but you haven’t established that. I would like to see an argument against AIXI-tl that does not really rely what it is or is not physically capable of doing, but rather on what it is intelligent enough to choose to do. After all, humans, despite supposedly being capable of “naturalized induction”, would not do well in this problem either. A human cannot by force of will reprogram her brain into a static set of commands, nor can she make her brain stop emitting heat.
Finally, I want to say why I am making these arguments. It is not because I want to advocate for AIXI-tl and argue for its intelligence. The way I think of it AIXI is the dumbest program that is still capable of learning the right behavior eventually. Actually it’s worse than that; my argument here has convince me that even with exponential resources AIXI-tl can’t argue itself out of a paper bag (Note argument does look into the internals of AIXI-tl rather than treating it as a black-box). So if anything I think you might be overestimating the intelligence of AIXI-tl. However, my concern is that in addition to its usual stupidity, you think AIXI-tl has an additional obstacle in terms of some sort of ‘Cartesian boundary problem’, and that there exists some sort of ‘naturalized induction’ which humans have and which AIXI and AIXI-tl don’t have. I am unconvinced by this, and I think it is an unproductive line of research. Rather, I think any problem AIXI has in reasoning about itself is either one humans also have in reasoning about themselves or analogous to a problem it has reasoning about other things. In this case it is a problem humans also have.
I think you’ve missed the point. When I said
I meant it literally and formally. The resulting machine may or may not be smart, but regardless, it does not necessarily obey the AIXI equations after the first timestep, and it lacks some of the formal properties of the AIXItl model. The AIXItl model assumes that the machine will continue to be AIXItl. Proofs about how AIXItl behaves (e.g., that its environment model improves over time) do not apply to AIXItls that can modify their code. I don’t know what properties this variant has, but I’m not yet convinced that they are nice ones.
Perhaps the variant can still act intelligently. Perhaps it cannot. Perhaps there’s a clever way to train it so that it happens to work in this particular game. I don’t know. My point is only that AIXItl was designed with an optimality property in mind (“superior to any other time t and length l bounded algorithm”), and that an embodied AIXItl lacks this property (regardless of the “router”).
This post attempts to explain why AIXItl is not a superior agent, and attempts to impart an intuition for why the field of AGI is not reducible to constructing better and better approximations of AIXI. It sounds like you already believe this point, so I won’t try to convince you of it further :-)
Be that as it may, the argument “humans can’t do X” is not a compelling reason to stop caring about X. It seems to me that an ideal artificial agent should be able to win at the HeatingUp game.
I have made no claims in this post about research strategy, I’m only trying to point out why AIXI is not an ideal (as this concept still seems foreign to many people). Again, it seems you are already on board with this idea.
I hear your complaints about what you’ve assumed are my research goals, but I think we should save the research strategy discussion for another place and time :-)
Ack! I’m not sure what to think. When I wrote that comment, I had the impression that we had some sort of philosophical conflict, and I felt like I should make the case for my side. However, now I worry the comment was too aggressive. Moreover, it seems like we agree on most of the questions we can state precisely. I’m not sure how to deal with this situation.
I suppose I could turn some assumptions into questions: To what extent is it your goal in this inquiry to figure out ‘naturalized induction’? Do you think ‘naturalized induction’ is something humans naturally do when thinking, perhaps imperfectly?
No worries :-)
Zero. To be honest, I don’t spend much time thinking about AIXI. My inclination with regards to AIXI is to shrug and say “it’s not ideal for all the obvious reasons, and I can’t use it to study self-modification”, and then move on.
However, it turns out that what I think are the “obvious reasons” aren’t so obvious to some. While I’m not personally confident that AIXI can be modified to be useful for studying self-modification, ignoring AIXI entirely isn’t the most cunning strategy for forming relationships with other AGI researchers (who are researching different parts of the problem, and for whom AIXI may indeed be quite interesting and relevant).
If anything, my “goal with this inquiry” is to clearly sketch specific problems with AIXI that make it less useful to me and point towards directions where I’d be happy to discuss collaboration with researchers who are interested in AIXI.
It is not the case that I’m working on these problems in my free time: left to my own devices, I just use (or develop) toy models that better capture the part of the problem space I care about.
I really don’t want to get dragged into a strategy discussion here. I’ll state a few points that I expect we both agree upon, but forgive me if I don’t answer further questions in this vein during this discussion.
Solomonoff induction would have trouble (or, at least, be non-optimal) in an uncomputable universe.
We’ve been pretty wrong about the rules of the universe in the past. (I wouldn’t have wanted scientists in 1750 to gamble on the universe being deterministic/single-branch, and I similarly don’t want scientists today to gamble on the universe being computable.)
Intuitively, it seems like there should be a computable program that can discover it’s inside an exotic universe (where ‘exotic’ includes ‘uncomputable’, but is otherwise a vague placeholder word).
I don’t think discussing how humans deal with this problem is relevant. Are there ways the universe could be that I can’t conceive of? Almost certainly. Can I figure out the laws of my universe as well as a perfect Solomonoff inductor? Probably not. Yet it does feel like I could be convinced that the universe is uncomputable, and so Solomonoff induction is probably not an idealization of whatever it is that I’m trying to do.
I don’t personally view this as an induction problem, but rather as a priors problem. And though I do indeed think it’s a problem, I’ll note that this does not imply that the problem captures any significant fraction of my research efforts.
Disputed! I’m pretty sure if somebody studied under controlled conditions for enough years they’d be able to induce mild hypothermia through some sort of meditative exercise.
More realistically, albeit not through willpower alone, a human could sit in a bathtub full of ice water, dose themselves with ethanol or sodium thiopental, or otherwise use external, independently-verifiable means to temporarily reduce their cognitive capacity.