Paperclip maximization doesn’t seem like a stable goal, though I could be wrong about that. Let’s say Clippy reproduces to create a bunch of clippys trying to maximize total paperclips (let’s call this collective ClippyBorg). If one of ClippyBorg’s subClippys had some variety of mutation that changed its goal set to one more suited for reproduction, it would outcompete the other clippys. Now ClippyBorg could destroy cancerClippy, but whether it would successfully do so every time is an open question.
One additional confounding factor is that if ClippyBorg’s subClippys are identical, they will not occupy every available niche optimally and could well be outcompeted by dumber but more adaptable agents (much like humans don’t completely dominate bacteria, despite vastly greater intelligence, due to lower adaptability).
A self-copying clippy would have the handicap of having to retain it’s desire to maximize paperclips, something other self-copiers wouldn’t have to do. I think the notion of Clippys not dominating does make sense, even if it’s not necessarily right. (my personal intuition is that whichever replicating optimizer with a stable goal set begins expansion first will dominate).
Paperclip maximization doesn’t seem like a stable goal, though I could be wrong about that. Let’s say Clippy reproduces to create a bunch of clippys trying to maximize total paperclips (let’s call this collective ClippyBorg). If one of ClippyBorg’s subClippys had some variety of mutation that changed its goal set to one more suited for reproduction, it would outcompete the other clippys. Now ClippyBorg could destroy cancerClippy, but whether it would successfully do so every time is an open question.
One additional confounding factor is that if ClippyBorg’s subClippys are identical, they will not occupy every available niche optimally and could well be outcompeted by dumber but more adaptable agents (much like humans don’t completely dominate bacteria, despite vastly greater intelligence, due to lower adaptability).
A self-copying clippy would have the handicap of having to retain it’s desire to maximize paperclips, something other self-copiers wouldn’t have to do. I think the notion of Clippys not dominating does make sense, even if it’s not necessarily right. (my personal intuition is that whichever replicating optimizer with a stable goal set begins expansion first will dominate).