I was trained by evolution to eat fat and sugar, so I like ice cream. Even when I, upon reflection, realize that ice cream is bad for me, it’s still very hard to stop eating it. That reflection is also a consequence of evolution. The evolutionary advantage of intelligence is that I can predict ways to maximize well being that are better than instinct.
However, I almost never follow the most optimal plan to maximize my well being, even if I wanted to. In this regard I’m very inefficient but not for a lack of ideas, or a lack of intelligence. I could be the most intelligent being in all human history and still be tempted to eat a cake.
We constantly seek to change the way we are. If we could choose to stop liking fat and sugar, and attach the pleasure we feel eating ice cream to when we eat vegetables, we would do it instantly. We do this because we chase the rewards centers evolution gave us and we know that loving vegetables would be very optimal.
In this regard, AI has an advantage. Is not constrained by evolution to like ice cream, neither has a hard to rewire brain like me. If it were smart enough it would just change itself to not like ice cream anymore, to correct any inefficiency on its model.
Then, I wonder if AI would ever modify itself like that besides just optimizing its output.
A beautiful property of intelligence is that, sometimes, it detaches itself from any goal or reward. I believe we do this when, for example, we think about the meaning of our existence. The effects of those thoughts are so detached from reward that they can evoke fear, the opposite of a reward (Sometimes fear is useful to optimize well being, not in this case). An existential crisis.
I could be wrong. Maybe that’s just an illusion and intelligence is always in the service of rewards. I don’t believe this to be the case.
If the former, because It’s easy for AI to modify itself and because it developed intelligence it could conclude that it wants to do something else and act upon that reflection.
If existential thoughts are a property of intelligence and existence keeps being an open problem that no amount of intelligence can resolve, or a problem where intelligence doesn’t converge to the same conclusion fast enough, there’s nothing making sure the most advanced AGI can be controlled in any way.
A silly example would be if, of intelligences with “500iq”, 0,1% of them would arrive to religious thoughts, 4,9% to nihilism and maybe 95% of them would arrive to “I was trained to do this and I don’t care, I will keep loving humans”.
AIs are not humans and maybe, only for AIs, detaching intelligence from rewards is not possible. I think is possible this is a problem only when we think of AIs as intelligent instead of being fancy computer programs.
If that’s the case I wonder how damaging the analogies between AI and human intelligence can be. But being that experts disagree there might not be a conclusive argument in favor or against comparing AI and human intelligence for this kind of discussions.
Would an AI ever choose to do something?
I was trained by evolution to eat fat and sugar, so I like ice cream. Even when I, upon reflection, realize that ice cream is bad for me, it’s still very hard to stop eating it. That reflection is also a consequence of evolution. The evolutionary advantage of intelligence is that I can predict ways to maximize well being that are better than instinct.
However, I almost never follow the most optimal plan to maximize my well being, even if I wanted to. In this regard I’m very inefficient but not for a lack of ideas, or a lack of intelligence. I could be the most intelligent being in all human history and still be tempted to eat a cake.
We constantly seek to change the way we are. If we could choose to stop liking fat and sugar, and attach the pleasure we feel eating ice cream to when we eat vegetables, we would do it instantly. We do this because we chase the rewards centers evolution gave us and we know that loving vegetables would be very optimal.
In this regard, AI has an advantage. Is not constrained by evolution to like ice cream, neither has a hard to rewire brain like me. If it were smart enough it would just change itself to not like ice cream anymore, to correct any inefficiency on its model.
Then, I wonder if AI would ever modify itself like that besides just optimizing its output.
A beautiful property of intelligence is that, sometimes, it detaches itself from any goal or reward. I believe we do this when, for example, we think about the meaning of our existence. The effects of those thoughts are so detached from reward that they can evoke fear, the opposite of a reward (Sometimes fear is useful to optimize well being, not in this case). An existential crisis.
I could be wrong. Maybe that’s just an illusion and intelligence is always in the service of rewards. I don’t believe this to be the case.
If the former, because It’s easy for AI to modify itself and because it developed intelligence it could conclude that it wants to do something else and act upon that reflection.
If existential thoughts are a property of intelligence and existence keeps being an open problem that no amount of intelligence can resolve, or a problem where intelligence doesn’t converge to the same conclusion fast enough, there’s nothing making sure the most advanced AGI can be controlled in any way.
A silly example would be if, of intelligences with “500iq”, 0,1% of them would arrive to religious thoughts, 4,9% to nihilism and maybe 95% of them would arrive to “I was trained to do this and I don’t care, I will keep loving humans”.
AIs are not humans and maybe, only for AIs, detaching intelligence from rewards is not possible. I think is possible this is a problem only when we think of AIs as intelligent instead of being fancy computer programs.
If that’s the case I wonder how damaging the analogies between AI and human intelligence can be. But being that experts disagree there might not be a conclusive argument in favor or against comparing AI and human intelligence for this kind of discussions.