You have done incredible, beautiful, courageous work I have immense appreciation and admiration for. And I recognise the pain, fear and frustration that must have brought you to this point.
But I hate, no, loathe, everything about this text. The unjustified assumptions behind it, the impact it will have on the perception of alignment research, the impact it will have on alignment researchers.
I do not want to do stuff that would have me remembered by nonexistent surviving observers as someone who valiantly tried and dramatically warned, and looked good doing it, so I can tell the corpses that I told them so while I choke. Strategies that achieve this are generally not the most effective strategies for actually improving our chances, they alienate the people we need to convince.
And death is never fucking dignified. The idea of dying with dignity repels me. It has always been code for surrender and acceptance. The very idea that death could be made dignified is a comforting illusion, and I will have none of it.
Yes, the risk is existential, and the situation is seriously fucking dire.
But no, you do not know we are doomed. You cannot know. And it is not ethical to claim you do, either. You are predicting how a future mind will act, a mind we do not understand and do not know. You are predicting its emergence as a certainty, while in a volatile, unpredictable world at the edge of climate collapse, emerging from a pandemic, and threatened by war that may well upend this develoopment. There is a huge difference between stating our immense uncertainty and concern on whether an AI will be aligned in light of the lack of a strategy ensuring it, and claiming to know it will be evil and wipe us all out, when none of us intend this outcome. You are disregarding the massive changes that might result from a hell of a lot of people suddenly taking AI safety seriously now.
In contrast, I would like to quote Rebecca Solnits “Hope in the Dark”, based on her studying unexpected, massive social transformations in response to historical crises:
“People have always been good at imagining the end of the world, which is much easier to picture than the strange sidelong paths of change in a world without end. (...) Cause-and-effect assumes history marches forward, but history is not an army. It is a crab scuttling sideways, a drip of soft water wearing away stone, an earthquake breaking centuries of tension. Sometimes one person inspires a movement, or her words do decades later, sometimes a few passionate people change the world; sometimes they start a mass movement and millions do; sometimes those millions are stirred by the same outrage or the same ideal, and change comes upon us like a change of weather. All that these transformations have in common is that they begin in the imagination, in hope. (...) The future is dark, with a darkness as much of the womb as the grave. (...) Inside the word “emergency” is “emerge”; from an emergency new things come forth. The old certainties are crumbling fast, but danger and possibility are sisters. (...) Hope locates itself in the premises that we don’t know what will happen and that in the spaciousness of uncertainty is room to act. When you recognize uncertainty, you recognize that you may be able to influence the outcomes–you alone or you in concert with a few dozen or several million others. Hope is an embrace of the unknown and knowable, a alternative to the certainty of both optimists and pessimists. Optimists think it will all be fine without our involvement; pessimists take the opposite position; both excuse themselves from acting. It’s the belief that what we do matters even though how and when it may matter, who and what is may impact, are not things we can know beforehand. We may not, in fact, know them afterward either, but they matter all the same, and history is full of people whose influence was most powerful after they were gone. (...) Hope just means another world might be possible, not promise, not guaranteed. Hope calls for action; action is impossible without hope.”
Sure, we are very probably screwed. On two fronts, honestly; we are simultaneously developing unaligned AI, and destroying the planetary climate. Like, that shit is bad, really bad, it is already killing people and unravelling right in front of our eyes. But frankly, none of us can give objective figures on the probability of this wiping us out, the systems involved are too complex, and too rapidly shifting. And, no matter how dire, they are probabilities, not certainties, and that is a world of difference. As both situations have become acute and impeding threats, the responses to them have also mobilised and are getting on track for mass level.
To quote Joanna Macy, author of “Active Hope”:
“These are warning signals that we live in a world that can end, at least as a home for conscious life. This is not to say that it will end, but it can end. That very possibility changes everything for us. (...) “The crisis is not going to be resolved by me, or you, or any one of us, but rather all of us. (...) If my teacher had told me how this whole situation was going to turn out, I wouldn’t have believed the story. It’s that uncertainty that keeps you alive.”
The climate movement has been struggling with this longer than you guys have. We’ve at this point put extensive thought into how to stay sane and helpful when dealing with a problem so big, so threatening, and escalating so badly. I think there is a lot to be learned here. https://www.resilience.org/stories/2018-07-02/the-varieties-of-hope/
There is still a chance that we will end up with AGI that is friendly, and mitigate the worst of climate change, catch ourselves in a stable state, and reverse without all being lost, while fundamentally changing and improving our way of life in the process, beating death instead of falling into extinction. The two—friendly AI, and stopping climate change—might in fact be entwined. And them both being resolved would be… fuck, more amazing than I can say. The risk is terrifying. But the opportunities are breathtaking. And it is not decided yet. It will not be decided until we are all dead.
And you are not alone. It is legit, necessary, and ethical for you to take breaks, you have done so fucking much. But this movement is bigger than you, and there are a lot of people who have been working on this from other angles, and new people joining. You have no idea what they will do. While we are still breathing, all is not lost.
You have done incredible, beautiful, courageous work I have immense appreciation and admiration for. And I recognise the pain, fear and frustration that must have brought you to this point.
But I hate, no, loathe, everything about this text. The unjustified assumptions behind it, the impact it will have on the perception of alignment research, the impact it will have on alignment researchers.
I do not want to do stuff that would have me remembered by nonexistent surviving observers as someone who valiantly tried and dramatically warned, and looked good doing it, so I can tell the corpses that I told them so while I choke. Strategies that achieve this are generally not the most effective strategies for actually improving our chances, they alienate the people we need to convince.
And death is never fucking dignified. The idea of dying with dignity repels me. It has always been code for surrender and acceptance. The very idea that death could be made dignified is a comforting illusion, and I will have none of it.
Yes, the risk is existential, and the situation is seriously fucking dire.
But no, you do not know we are doomed. You cannot know. And it is not ethical to claim you do, either. You are predicting how a future mind will act, a mind we do not understand and do not know. You are predicting its emergence as a certainty, while in a volatile, unpredictable world at the edge of climate collapse, emerging from a pandemic, and threatened by war that may well upend this develoopment. There is a huge difference between stating our immense uncertainty and concern on whether an AI will be aligned in light of the lack of a strategy ensuring it, and claiming to know it will be evil and wipe us all out, when none of us intend this outcome. You are disregarding the massive changes that might result from a hell of a lot of people suddenly taking AI safety seriously now.
In contrast, I would like to quote Rebecca Solnits “Hope in the Dark”, based on her studying unexpected, massive social transformations in response to historical crises:
“People have always been good at imagining the end of the world, which is much easier to picture than the strange sidelong paths of change in a world without end. (...) Cause-and-effect assumes history marches forward, but history is not an army. It is a crab scuttling sideways, a drip of soft water wearing away stone, an earthquake breaking centuries of tension. Sometimes one person inspires a movement, or her words do decades later, sometimes a few passionate people change the world; sometimes they start a mass movement and millions do; sometimes those millions are stirred by the same outrage or the same ideal, and change comes upon us like a change of weather. All that these transformations have in common is that they begin in the imagination, in hope. (...) The future is dark, with a darkness as much of the womb as the grave. (...) Inside the word “emergency” is “emerge”; from an emergency new things come forth. The old certainties are crumbling fast, but danger and possibility are sisters. (...) Hope locates itself in the premises that we don’t know what will happen and that in the spaciousness of uncertainty is room to act. When you recognize uncertainty, you recognize that you may be able to influence the outcomes–you alone or you in concert with a few dozen or several million others. Hope is an embrace of the unknown and knowable, a alternative to the certainty of both optimists and pessimists. Optimists think it will all be fine without our involvement; pessimists take the opposite position; both excuse themselves from acting. It’s the belief that what we do matters even though how and when it may matter, who and what is may impact, are not things we can know beforehand. We may not, in fact, know them afterward either, but they matter all the same, and history is full of people whose influence was most powerful after they were gone. (...) Hope just means another world might be possible, not promise, not guaranteed. Hope calls for action; action is impossible without hope.”
Sure, we are very probably screwed. On two fronts, honestly; we are simultaneously developing unaligned AI, and destroying the planetary climate. Like, that shit is bad, really bad, it is already killing people and unravelling right in front of our eyes. But frankly, none of us can give objective figures on the probability of this wiping us out, the systems involved are too complex, and too rapidly shifting. And, no matter how dire, they are probabilities, not certainties, and that is a world of difference. As both situations have become acute and impeding threats, the responses to them have also mobilised and are getting on track for mass level.
To quote Joanna Macy, author of “Active Hope”:
“These are warning signals that we live in a world that can end, at least as a home for conscious life. This is not to say that it will end, but it can end. That very possibility changes everything for us. (...) “The crisis is not going to be resolved by me, or you, or any one of us, but rather all of us. (...) If my teacher had told me how this whole situation was going to turn out, I wouldn’t have believed the story. It’s that uncertainty that keeps you alive.”
The climate movement has been struggling with this longer than you guys have. We’ve at this point put extensive thought into how to stay sane and helpful when dealing with a problem so big, so threatening, and escalating so badly. I think there is a lot to be learned here. https://www.resilience.org/stories/2018-07-02/the-varieties-of-hope/
There is still a chance that we will end up with AGI that is friendly, and mitigate the worst of climate change, catch ourselves in a stable state, and reverse without all being lost, while fundamentally changing and improving our way of life in the process, beating death instead of falling into extinction. The two—friendly AI, and stopping climate change—might in fact be entwined. And them both being resolved would be… fuck, more amazing than I can say. The risk is terrifying. But the opportunities are breathtaking. And it is not decided yet. It will not be decided until we are all dead.
And you are not alone. It is legit, necessary, and ethical for you to take breaks, you have done so fucking much. But this movement is bigger than you, and there are a lot of people who have been working on this from other angles, and new people joining. You have no idea what they will do. While we are still breathing, all is not lost.