AIXI isn’t computable. Assuming our universe is computable, that means you can’t have an AIXI laptop.
Well, I was referring to Eliezer’s hypercomputable laptop example. And we are assuming that somewhere in this machine there is a bit that causes ‘experiencing blue’. And I’m also not talking about the AIXI building a model of itself or its own behavior in any way.
Still AIXItl won’t be able to predict its own destruction, because destruction is not equivalent to any string of digits.
Destruction implies ‘no more reward after this point.’
Let me give a more thorough example. Let’s say our AIXI (or AIXItl) laptop has found the bit that corresponds to its reward. Whenever this bit is flipped, it experiences reward (there are good arguments that it will not incessantly flip this bit until it dies, but there’s nothing preventing it from realizing the bit’s existence). It is able to infer that if this bit is destroyed, that will not at all be conducive to maximizing its reward. Thus it makes it a priority to keep this bit intact, and by extension it makes it a priority to keep the laptop intact.
Nowhere in the above example have I included ‘self’. I suspect a similar process goes on with humans. If I had cancer in my leg, I would not object to my leg being cut off to save my life. If, however, one day, someone came over to me and showed me some new scientific literature conclusively demostrating that the leg is in fact the seat of intelligence and consciousness, and that the brain just does passive stuff like regulate heartbeat, I’d have a much more different attitude. I’d want to preserve my leg at all costs (imagine for a moment that I have not seen or heard of anyone with their leg amputated or brain dysfunctional, so I have no prior knowledge about such things). Nowhere has introspection entered this picture!
A bit has two possible values, 0 and 1, it doesn’t have “destroyed” value. And, of course, it’s not enough to prefer intact laptop over one wholly pulverised into plasma (the latter severs the cable connecting the laptop to the internet, after all). There’s necessarily some parts of the laptop which in reality are necessary to compute AIXI but which in AIXI’s model do not compute the output actions of AIXI (as potential actions are magic’d into the model rather than arise through processes in the model). Those parts are liable to being tampered with, especially as any changes in their behaviour would be grossly misunderstood (e.g. increase in clock speed would be misunderstood as world slowing down).
edit: And especially as existence of those parts is detrimental to the operation of the protected part of the laptop (due to their power consumption, heat, risk of short circuit failure taking out the protected part, etc). Somewhat simplifying, in reality there’s the CPU internals that compute AIXI, and there’s the bus controller that sends the actions onto the bus, to eventually act on the real world, and that reads the rewards. In AIXI’s model of self, there’s useless hardware (deep internals whose output is substituted for with potential actions) that is connected to the same power supply as the critically important components (that relay the actions and rewards), endangering their operation.
A bit has two possible values, 0 and 1, it doesn’t have “destroyed” value.
A physical bit does. Remember that we are talking about an actual bit stored inside a memory location on the computer (say, a capacitor in a DRAM cell).
And, of course, it’s not enough to prefer intact laptop over one wholly pulverised into plasma
Why not? Not recieving any future reward is such a huge negative utility that it would take a very large positive utility to carry out an action that would risk that occuring. Would you allow a surgeon to remove some section of your brain for $1,000,000 even if you knew that that section would not affect your reward pathways?
Would you allow a surgeon to remove some section of your brain for $1,000,000 even if you knew that that section would not affect your reward pathways?
If I had brain cancer or cerebral AVM or the like, I’d pay to have it removed. See my edit. The root issue is that in AIXI’s model, potential actions (that it iterates through) are not represented as output of some hardware, but are forced onto the model. Consequently the hardware that actually outputs those in the real world is not represented as critical. And it is connected in parallel onto the same power supply (as the understood-to-be-critically-important hardware which relays the actions). It literally thinks it got a brain parasite. Of course it won’t necessarily drop an anvil at the whole thing just because of experimenting—that’s patently stupid. It will surgically excise some parts with great caution.
Well, I was referring to Eliezer’s hypercomputable laptop example. And we are assuming that somewhere in this machine there is a bit that causes ‘experiencing blue’. And I’m also not talking about the AIXI building a model of itself or its own behavior in any way.
Destruction implies ‘no more reward after this point.’
Let me give a more thorough example. Let’s say our AIXI (or AIXItl) laptop has found the bit that corresponds to its reward. Whenever this bit is flipped, it experiences reward (there are good arguments that it will not incessantly flip this bit until it dies, but there’s nothing preventing it from realizing the bit’s existence). It is able to infer that if this bit is destroyed, that will not at all be conducive to maximizing its reward. Thus it makes it a priority to keep this bit intact, and by extension it makes it a priority to keep the laptop intact.
Nowhere in the above example have I included ‘self’. I suspect a similar process goes on with humans. If I had cancer in my leg, I would not object to my leg being cut off to save my life. If, however, one day, someone came over to me and showed me some new scientific literature conclusively demostrating that the leg is in fact the seat of intelligence and consciousness, and that the brain just does passive stuff like regulate heartbeat, I’d have a much more different attitude. I’d want to preserve my leg at all costs (imagine for a moment that I have not seen or heard of anyone with their leg amputated or brain dysfunctional, so I have no prior knowledge about such things). Nowhere has introspection entered this picture!
A bit has two possible values, 0 and 1, it doesn’t have “destroyed” value. And, of course, it’s not enough to prefer intact laptop over one wholly pulverised into plasma (the latter severs the cable connecting the laptop to the internet, after all). There’s necessarily some parts of the laptop which in reality are necessary to compute AIXI but which in AIXI’s model do not compute the output actions of AIXI (as potential actions are magic’d into the model rather than arise through processes in the model). Those parts are liable to being tampered with, especially as any changes in their behaviour would be grossly misunderstood (e.g. increase in clock speed would be misunderstood as world slowing down).
edit: And especially as existence of those parts is detrimental to the operation of the protected part of the laptop (due to their power consumption, heat, risk of short circuit failure taking out the protected part, etc). Somewhat simplifying, in reality there’s the CPU internals that compute AIXI, and there’s the bus controller that sends the actions onto the bus, to eventually act on the real world, and that reads the rewards. In AIXI’s model of self, there’s useless hardware (deep internals whose output is substituted for with potential actions) that is connected to the same power supply as the critically important components (that relay the actions and rewards), endangering their operation.
A physical bit does. Remember that we are talking about an actual bit stored inside a memory location on the computer (say, a capacitor in a DRAM cell).
Why not? Not recieving any future reward is such a huge negative utility that it would take a very large positive utility to carry out an action that would risk that occuring. Would you allow a surgeon to remove some section of your brain for $1,000,000 even if you knew that that section would not affect your reward pathways?
If I had brain cancer or cerebral AVM or the like, I’d pay to have it removed. See my edit. The root issue is that in AIXI’s model, potential actions (that it iterates through) are not represented as output of some hardware, but are forced onto the model. Consequently the hardware that actually outputs those in the real world is not represented as critical. And it is connected in parallel onto the same power supply (as the understood-to-be-critically-important hardware which relays the actions). It literally thinks it got a brain parasite. Of course it won’t necessarily drop an anvil at the whole thing just because of experimenting—that’s patently stupid. It will surgically excise some parts with great caution.