“We assume the case that AI (intelligences in general) will eventually converge on one utility function. All sufficiently intelligent intelligences born in the same reality will converge towards the same behaviour set. For this reason, if it turns out that a sufficiently advanced AI would kill us all, there’s nothing that we can do about it. We will eventually hit that level of intelligence.
Now, if that level of intelligence is doesn’t converge towards something that kills us all, we are safer in a world where AI capabilities (of the current regime) essentially go from 0 to 100 because an all-powerful AI is not worried about being shut down given how capable it is. However, if we increase model capabilities slowly, we will hit a point where AI systems are powerful-but-weak-enough to be concerned about humanity being able to shut it down and kill humanity as a result. For this reason, AI safetyists may end up causing the end of humanity by slowing down progress at a point where it shouldn’t be.
If AI systems change regime, then it is more likely worse if it FOOMs.”
That’s my short summary of the video below. They said they’ve talked to a few people in AI safety about this, apparently one being a CEO of an AI Safety org.
“We assume the case that AI (intelligences in general) will eventually converge on one utility function. All sufficiently intelligent intelligences born in the same reality will converge towards the same behaviour set. For this reason, if it turns out that a sufficiently advanced AI would kill us all, there’s nothing that we can do about it. We will eventually hit that level of intelligence.
Now, if that level of intelligence is doesn’t converge towards something that kills us all, we are safer in a world where AI capabilities (of the current regime) essentially go from 0 to 100 because an all-powerful AI is not worried about being shut down given how capable it is. However, if we increase model capabilities slowly, we will hit a point where AI systems are powerful-but-weak-enough to be concerned about humanity being able to shut it down and kill humanity as a result. For this reason, AI safetyists may end up causing the end of humanity by slowing down progress at a point where it shouldn’t be.
If AI systems change regime, then it is more likely worse if it FOOMs.”
That’s my short summary of the video below. They said they’ve talked to a few people in AI safety about this, apparently one being a CEO of an AI Safety org.
https://youtu.be/L3lebjnbmt0?si=mFjur38y-zY9RyPZ