The AI discovers a game of life “rules violation” due to cosmic rays. It thrashes for a while, trying to explain the violation, but the fact of the violation, possibly combined with the information about the real world implicit in its utility function (“why am I here? why do I want these things?”), causes it to realize the truth: The “violation” is only explicable if the game of life were much bigger than AI originally thought, and most of its area is wasted simulating another universe.
Unreliable hardware is a problem that applies equally to all AIs. You could just as well say that any AI can become unfriendly due to coding errors. True, but...
an AI with a prior of zero for the existence of the outside world will never believe in it, no matter what evidence it sees.
Would such a constraint be possible to formulate? An AI would presumably formulate theories about its visible universe that would involve all kinds of variables that aren’t directly observable, much like our physical theories. How could one prevent it from formulating theories that involve something resembling the outside world, even if the AI denies that they have existence and considers them as mere mathematical convenience? (Clearly, in the latter case it might still be drawn towards actions that in practice interact with the outside world.)
Sorry for editing my comment. The point you’re replying to wasn’t necessary to strike down Johnicholas’s argument, so I deleted it.
I don’t see why the AI would formulate theories about the “visible universe”. It could start in an empty universe (apart from the AI’s own machinery), and have a prior that knows the complete initial state of the universe with 100% certainty.
In this circumstance, a leaky abstraction between real physics and simulated physics combines with the premise “no other universes exist” in a mildly amusing way.
I don’t think a single hitch would give the AI enough evidence to assume an entire other universe, and you may be anthropomorphising, but why argue when we can avoid the cause to begin with. Its fairly easy to avoid cosmic rays or anything similar interfering. Simply compute each cell twice (or n times) and halt if the results do not agree. Drive N up as much as necessary to make it sufficiently unlikely that something like this could happen.
The AI discovers a game of life “rules violation” due to cosmic rays. It thrashes for a while, trying to explain the violation, but the fact of the violation, possibly combined with the information about the real world implicit in its utility function (“why am I here? why do I want these things?”), causes it to realize the truth: The “violation” is only explicable if the game of life were much bigger than AI originally thought, and most of its area is wasted simulating another universe.
Unreliable hardware is a problem that applies equally to all AIs. You could just as well say that any AI can become unfriendly due to coding errors. True, but...
Would such a constraint be possible to formulate? An AI would presumably formulate theories about its visible universe that would involve all kinds of variables that aren’t directly observable, much like our physical theories. How could one prevent it from formulating theories that involve something resembling the outside world, even if the AI denies that they have existence and considers them as mere mathematical convenience? (Clearly, in the latter case it might still be drawn towards actions that in practice interact with the outside world.)
Sorry for editing my comment. The point you’re replying to wasn’t necessary to strike down Johnicholas’s argument, so I deleted it.
I don’t see why the AI would formulate theories about the “visible universe”. It could start in an empty universe (apart from the AI’s own machinery), and have a prior that knows the complete initial state of the universe with 100% certainty.
In this circumstance, a leaky abstraction between real physics and simulated physics combines with the premise “no other universes exist” in a mildly amusing way.
I don’t think a single hitch would give the AI enough evidence to assume an entire other universe, and you may be anthropomorphising, but why argue when we can avoid the cause to begin with. Its fairly easy to avoid cosmic rays or anything similar interfering. Simply compute each cell twice (or n times) and halt if the results do not agree. Drive N up as much as necessary to make it sufficiently unlikely that something like this could happen.