Well, if he believes his AI will be specifically and precisely programmed so as to converge on exactly the right goals before they are solidified in the hard takeoff, then he’s working on a FAI. The remaining difference in opinions would be technical—about whether his AI will indeed converge, etc. It would not be about the Scary Idea itself.
I think it’s taken by Goertzel as part of the Scary Idea that it’s necessary to use several orders more precise understanding of AI’s goals for its behavior not to be disastrous.
It’s a direct logical consequence, isn’t it? If one doesn’t have a precise understanding of AI’s goals, then whatever goals one imparts into AI won’t be precise. And they must be precise, or (step3) ⇒ disaster.
He can’t think that god-powerfully optimizing for a forever-fixed not-precisely-correct goal would lead to anything but disaster. Not if he ever saw a non-human optimization process at work.
So he can only think precision is not important if he believes that (1) human values are an attractor in the goal space, and any reasonably close goals would converge there before solidifying, and/or (2) acceptable human values form a large convex region within the goal space, and optimizing for any point within this region is correct.
Without better understanding of AI goals, both can only be an article of faith...
4) Human values are not a natural category, there’s little to no chance that AI will converge on them by itself, unless specifically and precisely programmed.
Goertzel expressed doubt about step 4, saying that while it’s true that random AIs will have bad goals, he’s not working on random AIs.
That’s not really the same as asserting that human values are a natural category.
Goertzel expressed doubt about step 4, saying that while it’s true that random AIs will have bad goals, he’s not working on random AIs.
Well, if he believes his AI will be specifically and precisely programmed so as to converge on exactly the right goals before they are solidified in the hard takeoff, then he’s working on a FAI. The remaining difference in opinions would be technical—about whether his AI will indeed converge, etc. It would not be about the Scary Idea itself.
I think it’s taken by Goertzel as part of the Scary Idea that it’s necessary to use several orders more precise understanding of AI’s goals for its behavior not to be disastrous.
It’s a direct logical consequence, isn’t it? If one doesn’t have a precise understanding of AI’s goals, then whatever goals one imparts into AI won’t be precise. And they must be precise, or (step3) ⇒ disaster.
He doesn’t agree that they must be precise, so I guess step 3 is also out.
He can’t think that god-powerfully optimizing for a forever-fixed not-precisely-correct goal would lead to anything but disaster. Not if he ever saw a non-human optimization process at work.
So he can only think precision is not important if he believes that
(1) human values are an attractor in the goal space, and any reasonably close goals would converge there before solidifying, and/or
(2) acceptable human values form a large convex region within the goal space, and optimizing for any point within this region is correct.
Without better understanding of AI goals, both can only be an article of faith...
From the conversation with Luke, he apparently accepts faith.
That’s not really the same as asserting that human values are a natural category.