It has the ability to model and to investigate hypothetical possibilities that might negatively impact the utility function it is optimizing.
To be able to optimize it will have to know what it is supposed to optimize. You’ve to carefully specify what it output (utility function) is supposed to be or it won’t be able to tell how good it is at optimizing. If you just tell it to produce paperclips, it won’t be able to self-improve because it doesn’t know how paperclips look like etc., therefore it cannot judge its own success or that extreme heat would be a negative impact giving paperclips made out of plastic. You further assume that it has a detailed incentive, that it is given a detailed pathway that it tells to look for threats and eliminate them.
If it doesn’t, it is far below human intelligence and is non-threatening for the same reason a narrow AI is non-threatening (but it isn’t very useful either).
If it doesn’t it is what most researchers are working on, an intelligence with the potential to learn and make use of what it learnt, with the potential to become intelligent (educated). I’m getting the impression that people here assume that researchers are not working on an AGI but to hardcode a FOOM machine. If FOOM is simply part of your definition then there’s no arguing against it going FOOM. But what researchers like Goertzel are working on are systems with the potential to reach human level intelligence, that does not mean that they will by definition jailbreak their nursery school. Although I never tried to argue against the possibility but that there are many pathways where this won’t happen rather than the way it is portrayed by the SIAI, that any implementation of AGI will most likely consume humnanity.
The sorts of intelligences you are talking about are narrow AIs, not general intelligences. If you told a general intelligence to produce paperclips but it didn’t know what a paperclip was, then its first subgoal would be to find out. The sort of mind that would give up on a minor obstacle like that wouldn’t foom, but it wouldn’t be much of an AGI either.
And yes, most researchers today are working on narrow AIs, not on AGI. That means they’re less likely to successfully make a general intelligence, but it has no bearing on the question of what will happen if they do make one.
To be able to optimize it will have to know what it is supposed to optimize. You’ve to carefully specify what it output (utility function) is supposed to be or it won’t be able to tell how good it is at optimizing. If you just tell it to produce paperclips, it won’t be able to self-improve because it doesn’t know how paperclips look like etc., therefore it cannot judge its own success or that extreme heat would be a negative impact giving paperclips made out of plastic. You further assume that it has a detailed incentive, that it is given a detailed pathway that it tells to look for threats and eliminate them.
If it doesn’t it is what most researchers are working on, an intelligence with the potential to learn and make use of what it learnt, with the potential to become intelligent (educated). I’m getting the impression that people here assume that researchers are not working on an AGI but to hardcode a FOOM machine. If FOOM is simply part of your definition then there’s no arguing against it going FOOM. But what researchers like Goertzel are working on are systems with the potential to reach human level intelligence, that does not mean that they will by definition jailbreak their nursery school. Although I never tried to argue against the possibility but that there are many pathways where this won’t happen rather than the way it is portrayed by the SIAI, that any implementation of AGI will most likely consume humnanity.
The sorts of intelligences you are talking about are narrow AIs, not general intelligences. If you told a general intelligence to produce paperclips but it didn’t know what a paperclip was, then its first subgoal would be to find out. The sort of mind that would give up on a minor obstacle like that wouldn’t foom, but it wouldn’t be much of an AGI either.
And yes, most researchers today are working on narrow AIs, not on AGI. That means they’re less likely to successfully make a general intelligence, but it has no bearing on the question of what will happen if they do make one.