I’m really glad you wrote this post, because Tsvi’s post is different and touches on very different concepts! That post is mainly about fun and exploration being undervalued as a human being. Your post seems to have one goal: ensure that up-and-coming alignment researchers do not burn themselves out or hyperfocus on only one strategy for contributing to reducing AI extinction risk.
Note, this passage seems to be a bit… off to me.
This one is slightly different from the last because it is an injunction to take care of your mental health. You are more useful to us when you are not stressed. I won’t deny that you are personally responsible for the entire destiny of the universe, because I won’t lie to you: but we have no use for a broken tool.
People aren’t broken tools. People have limited agency, and claiming they are “personally responsible for the entire destiny of the universe” is misleading. One must have an accurate sense of the agency and influence they have when it comes to reducing extinction risk if they want to be useful.
The notion that alignment researchers and people supporting them are “heroes” is a beautiful and intoxicating fantasy. One must be careful that it doesn’t lead to corruption in our epistemics, just because we want to maintain our belief in this narrative.
The passage on “you are responsible for the entire destiny of the universe” was mostly addressing the way it seems many EAs feel about the nature of responsibility. We indeed have limited agency in the world but people around here tend to feel they are personally responsible for literally saving the world alone. The idea was not to frontally deny that or to run against heroic responsibility but rather to say that while the responsibility won’t go away, there’s no point in becoming consumed by it. You are a less effective tool if you are too heavily burdened by responsibility to function properly. I wrote it that way because I’m hoping the harsh and utilitarian tone will reach the target audience better than something more clichèd would. There’s enough romanticization as it is here.
I definitely romanticized the alignment researchers being heroes part. I’ll add a disclaimer to mention that the choice of words was meant to paint the specific approach, the specific picture that up-and-coming alignment researchers might have when they arrive here.
As for which narrative to follow, this one might be as good as any. As the mental health post I referenced here mentioned, the “dying with dignity” approach Eliezer is following might not sit well with a number of people even when it is in line with his own predictions. I’m not sure to what degree what I described is a fantasy. In a universe where alignment is solved, would this picture be inacurate?
I’m really glad you wrote this post, because Tsvi’s post is different and touches on very different concepts! That post is mainly about fun and exploration being undervalued as a human being. Your post seems to have one goal: ensure that up-and-coming alignment researchers do not burn themselves out or hyperfocus on only one strategy for contributing to reducing AI extinction risk.
Note, this passage seems to be a bit… off to me.
People aren’t broken tools. People have limited agency, and claiming they are “personally responsible for the entire destiny of the universe” is misleading. One must have an accurate sense of the agency and influence they have when it comes to reducing extinction risk if they want to be useful.
The notion that alignment researchers and people supporting them are “heroes” is a beautiful and intoxicating fantasy. One must be careful that it doesn’t lead to corruption in our epistemics, just because we want to maintain our belief in this narrative.
The passage on “you are responsible for the entire destiny of the universe” was mostly addressing the way it seems many EAs feel about the nature of responsibility. We indeed have limited agency in the world but people around here tend to feel they are personally responsible for literally saving the world alone. The idea was not to frontally deny that or to run against heroic responsibility but rather to say that while the responsibility won’t go away, there’s no point in becoming consumed by it. You are a less effective tool if you are too heavily burdened by responsibility to function properly. I wrote it that way because I’m hoping the harsh and utilitarian tone will reach the target audience better than something more clichèd would. There’s enough romanticization as it is here.
I definitely romanticized the alignment researchers being heroes part. I’ll add a disclaimer to mention that the choice of words was meant to paint the specific approach, the specific picture that up-and-coming alignment researchers might have when they arrive here.
As for which narrative to follow, this one might be as good as any. As the mental health post I referenced here mentioned, the “dying with dignity” approach Eliezer is following might not sit well with a number of people even when it is in line with his own predictions. I’m not sure to what degree what I described is a fantasy. In a universe where alignment is solved, would this picture be inacurate?
Thanks for the feedback!