General note: I’m confident Luke and Anna wouldn’t endorse Robin’s characterization of their position here. (Nor do I think Robin’s trying to summarize their view in a way they’d endorse. Rather, he’s deliberately reframing their view in other terms to try to encourage original seeing.)
A quick response on Luke and Anna’s behalf (though I’m sure their own response would look very different):
You can call a nuclear weapon design or a computer virus our ‘child’ or ‘descendant’, but this wouldn’t imply that we should have similar policies for nukes or viruses as the ones we have for our actual descendants. There needs to be a specific argument for why we should expect AGI systems to be like descendants on the dimensions that matter.
(Or, if we have the choice of building AGI systems that are more descendant-like versus less descendant-like, there needs to be some argument for why we ought to choose to build highly descendant-like AGI systems. E.g., if we have the option of building sentient AGI vs. nonsentient AGI, is there a reason we should need to choose “sentient”?)
It’s true we shouldn’t mistreat sentient AGI systems any more than we should mistreat humans; but we’re in the position of having to decide what kind of AGI systems to build, with finite resources.
If approach U would produce human-unfriendly AGI and approach F would produce human-friendly AGI, you can object that choosing F over U is “brainwashing” AGI or cruelly preventing U’s existence; but if you instead chose U over F, you could equally object that you’re brainwashing the AGI to be human-unfriendly, or cruelly preventing F’s existence. It’s true that we should expect U to be a lot easier than F, but I deny that this or putting on a blindfold creates a morally relevant asymmetry. In both cases, you’re just making a series of choices that determine what AGI systems look like.
It’s true we shouldn’t mistreat sentient AGI systems any more than we should mistreat humans; but we’re in the position of having to decide what kind of AGI systems to build, with finite resources.
That’s not how R&D works. The early versions of something, you need the freedom to experiment, and early versions of an idea need to be both simple and well instrumented.
One reason your first AI might be a ‘paperclip maximizer’ is simply that’s less code to fight with. Certainly, OpenAI’s papers that’s basically what all their systems are. (they don’t have the ability or capacity to allocate additional resources which seems to be the key step that makes a paperclip maximizer dangerous)
General note: I’m confident Luke and Anna wouldn’t endorse Robin’s characterization of their position here. (Nor do I think Robin’s trying to summarize their view in a way they’d endorse. Rather, he’s deliberately reframing their view in other terms to try to encourage original seeing.)
A quick response on Luke and Anna’s behalf (though I’m sure their own response would look very different):
You can call a nuclear weapon design or a computer virus our ‘child’ or ‘descendant’, but this wouldn’t imply that we should have similar policies for nukes or viruses as the ones we have for our actual descendants. There needs to be a specific argument for why we should expect AGI systems to be like descendants on the dimensions that matter.
(Or, if we have the choice of building AGI systems that are more descendant-like versus less descendant-like, there needs to be some argument for why we ought to choose to build highly descendant-like AGI systems. E.g., if we have the option of building sentient AGI vs. nonsentient AGI, is there a reason we should need to choose “sentient”?)
It’s true we shouldn’t mistreat sentient AGI systems any more than we should mistreat humans; but we’re in the position of having to decide what kind of AGI systems to build, with finite resources.
If approach U would produce human-unfriendly AGI and approach F would produce human-friendly AGI, you can object that choosing F over U is “brainwashing” AGI or cruelly preventing U’s existence; but if you instead chose U over F, you could equally object that you’re brainwashing the AGI to be human-unfriendly, or cruelly preventing F’s existence. It’s true that we should expect U to be a lot easier than F, but I deny that this or putting on a blindfold creates a morally relevant asymmetry. In both cases, you’re just making a series of choices that determine what AGI systems look like.
It’s true we shouldn’t mistreat sentient AGI systems any more than we should mistreat humans; but we’re in the position of having to decide what kind of AGI systems to build, with finite resources.
That’s not how R&D works. The early versions of something, you need the freedom to experiment, and early versions of an idea need to be both simple and well instrumented.
One reason your first AI might be a ‘paperclip maximizer’ is simply that’s less code to fight with. Certainly, OpenAI’s papers that’s basically what all their systems are. (they don’t have the ability or capacity to allocate additional resources which seems to be the key step that makes a paperclip maximizer dangerous)