Just wanted to remark that this is one of the most scissory things I’ve ever seen on LW, and that fact surprises me. The karma level of the OP hovers between −10 to +10 with 59 total votes as of this moment. Many of the comments are similarly quite chaotic karma-wise.
The reason the controversy surprises me is that this seems like the sort of thing that I would have expected Less Wrong to coordinate around in the early phase of the Singularity, where we are now. Of course we should advocate for shutting down and/or restricting powerful AI agents released to the wide world without adequate safety measures employed. I would have thought this would be something we could all agree on.
It seems from the other comments on this post that people are worried that this is in some sense premature, or “crying wolf”, or that we are squandering our political capital, or something along those lines. If you are seeing very strong signs of misaligned near-AGI-level being released to the public, and you still think we should keep our powder dry, then I am not sure at what point exactly you would think it appropriate to exert our influence and spend our political capital. Like, what are we doing here, folks?
For my part I just viewed this petition as simply a good idea with no significant drawbacks. I would like to see companies that release increasingly powerful agents with obvious, glaring, film-antagonist level villain tendencies be punished in the court of public opinion, and this is a good way to do that.
The value of such a petition seems obviously positive on net. Consider the likely outcomes of this petition:
* If the petition gets a ton of votes and makes a big splash in the public consciousness, good, a company will have received unambiguous and undeniable PR backlash for prematurely releasing a powerful AI product, and all actors in this space will think just a little bit more carefully about deploying powerful AI technologies to prod without more attention paid to alignment. If this results in 1 more AI programmer working in alignment instead of pure capabilities, the petition was a win.
* If the petition gets tons of votes and is then totally ignored by Microsoft, its existence serves as future rhetorical ammunition for the Alignment camp in making the argument that even obviously misaligned agents are not being treated with due caution, and the “we will just unplug it if it misbehaves” argument can be forever dismissed.
* If the petition gets a moderate amount of votes, it raises awareness of the current increasingly dangerous level of capabilities.
* If the petition gets very few votes and is totally ignored by the world at large, then who cares, null result.
Frankly I think the con positions laid out in the sibling comments on this post are too clever by half. To my mind they sum up to an argument that we shouldn’t argue for shutting down dangerous AI because it makes us look weird. Sorry, folks, we always looked weird! I only hope that we have the courage to continue to look weird until the end!
I think the proper way for the LessWrong community to react on this situation would be to have a discussion first, and the petition optionally later. This is not the kind of situation where delaying our response by a few days would be fatal.
If someone writes a petition alone and then shares it on LessWrong, that gives the rest of us essentially two options: (a) upvote the post, and have LessWrong associated with a petition whose text we could not influence, or (b) downvote the post. Note that if we downvote the post, we still have the option to discuss the topic, and optionally make a new petition later.
But yes, it is a scissor statement. My own opinion on it would be a coinflip.
Which is why I would prefer to have a discussion first, and then have a version that I could unambiguously support.
My issue is simply that I think LW has probably jumped the gun, because I think there is counter evidence on the misalignment example, and I’m starting to get a lot more worried about LW epistemic responses.
Just wanted to remark that this is one of the most scissory things I’ve ever seen on LW, and that fact surprises me. The karma level of the OP hovers between −10 to +10 with 59 total votes as of this moment. Many of the comments are similarly quite chaotic karma-wise.
The reason the controversy surprises me is that this seems like the sort of thing that I would have expected Less Wrong to coordinate around in the early phase of the Singularity, where we are now. Of course we should advocate for shutting down and/or restricting powerful AI agents released to the wide world without adequate safety measures employed. I would have thought this would be something we could all agree on.
It seems from the other comments on this post that people are worried that this is in some sense premature, or “crying wolf”, or that we are squandering our political capital, or something along those lines. If you are seeing very strong signs of misaligned near-AGI-level being released to the public, and you still think we should keep our powder dry, then I am not sure at what point exactly you would think it appropriate to exert our influence and spend our political capital. Like, what are we doing here, folks?
For my part I just viewed this petition as simply a good idea with no significant drawbacks. I would like to see companies that release increasingly powerful agents with obvious, glaring, film-antagonist level villain tendencies be punished in the court of public opinion, and this is a good way to do that.
The value of such a petition seems obviously positive on net. Consider the likely outcomes of this petition:
* If the petition gets a ton of votes and makes a big splash in the public consciousness, good, a company will have received unambiguous and undeniable PR backlash for prematurely releasing a powerful AI product, and all actors in this space will think just a little bit more carefully about deploying powerful AI technologies to prod without more attention paid to alignment. If this results in 1 more AI programmer working in alignment instead of pure capabilities, the petition was a win.
* If the petition gets tons of votes and is then totally ignored by Microsoft, its existence serves as future rhetorical ammunition for the Alignment camp in making the argument that even obviously misaligned agents are not being treated with due caution, and the “we will just unplug it if it misbehaves” argument can be forever dismissed.
* If the petition gets a moderate amount of votes, it raises awareness of the current increasingly dangerous level of capabilities.
* If the petition gets very few votes and is totally ignored by the world at large, then who cares, null result.
Frankly I think the con positions laid out in the sibling comments on this post are too clever by half. To my mind they sum up to an argument that we shouldn’t argue for shutting down dangerous AI because it makes us look weird. Sorry, folks, we always looked weird! I only hope that we have the courage to continue to look weird until the end!
I think the proper way for the LessWrong community to react on this situation would be to have a discussion first, and the petition optionally later. This is not the kind of situation where delaying our response by a few days would be fatal.
If someone writes a petition alone and then shares it on LessWrong, that gives the rest of us essentially two options: (a) upvote the post, and have LessWrong associated with a petition whose text we could not influence, or (b) downvote the post. Note that if we downvote the post, we still have the option to discuss the topic, and optionally make a new petition later.
But yes, it is a scissor statement. My own opinion on it would be a coinflip.
Which is why I would prefer to have a discussion first, and then have a version that I could unambiguously support.
My issue is simply that I think LW has probably jumped the gun, because I think there is counter evidence on the misalignment example, and I’m starting to get a lot more worried about LW epistemic responses.