A paperclip maximizer also has a dead-end goal—maximizing paperclips—which is what I object to. An non-Friendly AI just has no particular interest in humans; there is no other necessary claim you can make about it.
Even more hairy. Any primary goal will, I think, eventually end up with a paperclipper. We need more research into how intelligent beings (ie, humans) actually function. I do not think people, with rare exceptions, actually have primary goals, only temporary, contingent goals to meet temporary ends. That is one reason I don’t think much of utilitarianism—peoples “utilities” are almost always temporary, contingent, and self-limiting.
This is also one reason why I have said that I think provably Friendly AI is impossible. I will be glad to be proven wrong if it does turn out to be possible.
Ah, I see. I misunderstood your definition of “paperclip maximiser”; I assumed paperclip maximiser and Unfriendly AI were equivalent. Sorry.
Next question: if maximising paperclips or relentless self-optimisation is a dead-end goal, what is an example of a non-dead-end goal? Is there a clear border between the two, which will be obvious to the AI?
To my mind, if paperclip maximisation is a dead end, then so is everything else. The Second Law of Thermodynamics will catch up with you eventually. Nothing you create will endure forever. The only thing you can do is try to maximise your utility for as long as possible, and if that means paperclips, then so be it.
A paperclip maximizer also has a dead-end goal—maximizing paperclips—which is what I object to. An non-Friendly AI just has no particular interest in humans; there is no other necessary claim you can make about it.
You have merely redefined the goal from ‘the benefit of humanity’ to ‘non dead-end goal’, which may just be equally hairy.
Even more hairy. Any primary goal will, I think, eventually end up with a paperclipper. We need more research into how intelligent beings (ie, humans) actually function. I do not think people, with rare exceptions, actually have primary goals, only temporary, contingent goals to meet temporary ends. That is one reason I don’t think much of utilitarianism—peoples “utilities” are almost always temporary, contingent, and self-limiting.
This is also one reason why I have said that I think provably Friendly AI is impossible. I will be glad to be proven wrong if it does turn out to be possible.
Ah, I see. I misunderstood your definition of “paperclip maximiser”; I assumed paperclip maximiser and Unfriendly AI were equivalent. Sorry.
Next question: if maximising paperclips or relentless self-optimisation is a dead-end goal, what is an example of a non-dead-end goal? Is there a clear border between the two, which will be obvious to the AI?
To my mind, if paperclip maximisation is a dead end, then so is everything else. The Second Law of Thermodynamics will catch up with you eventually. Nothing you create will endure forever. The only thing you can do is try to maximise your utility for as long as possible, and if that means paperclips, then so be it.
What would be a non-friendly goal that isn’t dead-end? (N.B. Not a rhetorical question.)
Deciding that humanity was a poor choice for the dominant sapient species and should be replaced by (the improved descendants of) dolphins or octopi?