We’ve noticed that new alignment researchers (and sometimes experienced alignment researchers) often fall into similar traps.
Here are 7 traps that we often see in new alignment researchers:
1. They think they need to be up-to-date on the literature before they can start contributing their own ideas.
Suggestion: Staying up-to-to-date on the literature is useful. But you don’t need to read everything before you contribute your own ideas. You can write naive hypotheses and try to generate new ideas. For many people, it’s easier to come up with (certain types of) new ideas before reading all of the literature. Once you’ve read the literature, it can be harder to ignore the ideas and frames of others. (see also Cached Thoughts)
2. They end up pursuing proxy goals.
A. They end up starting projects (often projects that take 3+ months) without a clear theory of change. They forget the terminal goal (e.g., reducing x-risk) and end up pursuing a proxy goal that looks like it’s helping (e.g., skill-up in ML). To be clear, many proxy goals (like skilling up in ML) are not inherently bad. But they don’t have a sense of why they’re learning ML, which subproblems they’re hoping to solve with ML, and which specific subskills in ML (and other fields) might be helpful. They don’t have a sense of how long they should spend skilling up, what else they should be learning, or what evidence they should be looking for to tell them to stop (or keep going).
B. They lose sight of the terminal goal. The real goal is not to skill-up in ML. The real goal is not to replicate the results of a paper. The real goal is not even to “solve inner alignment.” The real goal is to not die & not lose the value of the far-future.
Suggestion: Keep the terminal goal in mind. Try to have a clear idea of how your actions are getting you closer to the terminal goal. Sometimes, you won’t have clear answers (and indeed if you rely on having a clear end-to-end impact story before doing anything, you may fall into the trap of doing nothing). But notice when you’re doing things that don’t have a clear path toward the terminal goal. Occasionally ask yourself if there are projects that seem useful but don’t actually matter. Be cautious about spending many months on projects unless they have a justifiable theory of change.
3. They assume that if they don’t understand something, it’s because they are dumb (as opposed to thinking that the writer explained it unclearly, that the writer doesn’t understand it, or that the claim is wrong).
Suggestion: If you don’t understand something, have some probability on “This thing actually makes sense and I simply don’t understand it.” But do not discard hypotheses like “this thing is poorly written”, “this thing is confusing”, “the author doesn’t even understand this thing fully yet”, or “this thing is wrong.” Ask people you respect if they understand the thing. Ask them to explain it to you. Ask them to explain any jargon they use. Ask them to explain it as if they were talking to an intelligent high school student. If there are ideas that consistently fail the “high school student” check, stay open to the possibility that the idea hasn’t been properly understood, explained, or justified. If you like writing, you can also post comments on LessWrong, but I find talking to usually be ~10x better because it is so much higher bandwidth.
4. They rarely challenge the ideas and frames of authority figures.
For example, people hear that there is “inner alignment and outer alignment” or they hear that “inner alignment is the most important problem”. And then they start trying to solve inner alignment and outer alignment. And they don’t realize that this is a model. This is not the truth. This is a model.
Suggestion: Remember that the things you read are claims and the frames you hear are models. Some of these frames are unhelpful. Some of these frames are helpful but suboptimal. Notice when you are relying on the same frames as others.
5. They don’t distinguish between “intuitions” and “hypotheses”.
If Alice and Bob disagree (e.g., about whether or not evolution analogies are useful), it’s easy for them to say “ah, we just have different intuitions about the problem” and then they move on. This is unacceptable in other scientific fields and often serves as a curiosity-stopper.
Suggestion: Intuitions should be processed through reasoning and logic to figure out if intuitions are fallacies or hypotheses. If you realize that you have “intuitions” about a topic, take that as an opportunity to examine these intuitions more clearly. Where do they come from? Are there any hypotheses you can make based on them?
6. They end up working on a specific research agenda given by a senior researcher (e.g., Circuits or Infra-Bayesianism), without understanding why this is useful for solving alignment as a whole.
Ending up in this situation is a) not helpful for building up your inside views & b) makes it harder to research, because you don’t understand the constraints on your solution.
Suggestion: Try to solve the whole alignment problem. In doing this, think about 1) the key barriers that all of your proposed solutions are running into, and 2) the tools that you have. These solutions (probably) won’t be good, but they are super useful for building inside views. A useful exercise is to build an ‘alignment game tree’, where you (and maybe a few friends) propose solutions to alignment, then break those solutions, then create patches, iteratively.
7. They spend too much time on the fundamental math and CS behind alignment (e.g., trying to complete all of MIRI’s course recommendations or John Wentworth’s study guide) or getting degrees in Math/CS.
These normally take multiple years, and yet I claim that you can get near the frontier of alignment knowledge in ~6 months to a year.
Suggestion: Definitely learn linear algebra, multivariable calculus (the differentiation part, integration doesn’t come up often), probability theory, and basic ML very well. Past that, I recommend learning things as they come up in the course of working on alignment.
While we believe these traps are common, and we want more researchers looking out for them, we also encourage you to consider the law of equal and opposite advice. For each of these traps, it is possible to fall too far in the opposite direction (e.g., never spending time learning relevant math/CS concepts).
7 traps that (we think) new alignment researchers often fall into
We’ve noticed that new alignment researchers (and sometimes experienced alignment researchers) often fall into similar traps.
Here are 7 traps that we often see in new alignment researchers:
1. They think they need to be up-to-date on the literature before they can start contributing their own ideas.
Suggestion: Staying up-to-to-date on the literature is useful. But you don’t need to read everything before you contribute your own ideas. You can write naive hypotheses and try to generate new ideas. For many people, it’s easier to come up with (certain types of) new ideas before reading all of the literature. Once you’ve read the literature, it can be harder to ignore the ideas and frames of others. (see also Cached Thoughts)
2. They end up pursuing proxy goals.
A. They end up starting projects (often projects that take 3+ months) without a clear theory of change. They forget the terminal goal (e.g., reducing x-risk) and end up pursuing a proxy goal that looks like it’s helping (e.g., skill-up in ML). To be clear, many proxy goals (like skilling up in ML) are not inherently bad. But they don’t have a sense of why they’re learning ML, which subproblems they’re hoping to solve with ML, and which specific subskills in ML (and other fields) might be helpful. They don’t have a sense of how long they should spend skilling up, what else they should be learning, or what evidence they should be looking for to tell them to stop (or keep going).
B. They lose sight of the terminal goal. The real goal is not to skill-up in ML. The real goal is not to replicate the results of a paper. The real goal is not even to “solve inner alignment.” The real goal is to not die & not lose the value of the far-future.
Suggestion: Keep the terminal goal in mind. Try to have a clear idea of how your actions are getting you closer to the terminal goal. Sometimes, you won’t have clear answers (and indeed if you rely on having a clear end-to-end impact story before doing anything, you may fall into the trap of doing nothing). But notice when you’re doing things that don’t have a clear path toward the terminal goal. Occasionally ask yourself if there are projects that seem useful but don’t actually matter. Be cautious about spending many months on projects unless they have a justifiable theory of change.
3. They assume that if they don’t understand something, it’s because they are dumb (as opposed to thinking that the writer explained it unclearly, that the writer doesn’t understand it, or that the claim is wrong).
Suggestion: If you don’t understand something, have some probability on “This thing actually makes sense and I simply don’t understand it.” But do not discard hypotheses like “this thing is poorly written”, “this thing is confusing”, “the author doesn’t even understand this thing fully yet”, or “this thing is wrong.” Ask people you respect if they understand the thing. Ask them to explain it to you. Ask them to explain any jargon they use. Ask them to explain it as if they were talking to an intelligent high school student. If there are ideas that consistently fail the “high school student” check, stay open to the possibility that the idea hasn’t been properly understood, explained, or justified. If you like writing, you can also post comments on LessWrong, but I find talking to usually be ~10x better because it is so much higher bandwidth.
4. They rarely challenge the ideas and frames of authority figures.
For example, people hear that there is “inner alignment and outer alignment” or they hear that “inner alignment is the most important problem”. And then they start trying to solve inner alignment and outer alignment. And they don’t realize that this is a model. This is not the truth. This is a model.
Suggestion: Remember that the things you read are claims and the frames you hear are models. Some of these frames are unhelpful. Some of these frames are helpful but suboptimal. Notice when you are relying on the same frames as others.
5. They don’t distinguish between “intuitions” and “hypotheses”.
If Alice and Bob disagree (e.g., about whether or not evolution analogies are useful), it’s easy for them to say “ah, we just have different intuitions about the problem” and then they move on. This is unacceptable in other scientific fields and often serves as a curiosity-stopper.
Suggestion: Intuitions should be processed through reasoning and logic to figure out if intuitions are fallacies or hypotheses. If you realize that you have “intuitions” about a topic, take that as an opportunity to examine these intuitions more clearly. Where do they come from? Are there any hypotheses you can make based on them?
6. They end up working on a specific research agenda given by a senior researcher (e.g., Circuits or Infra-Bayesianism), without understanding why this is useful for solving alignment as a whole.
Ending up in this situation is a) not helpful for building up your inside views & b) makes it harder to research, because you don’t understand the constraints on your solution.
Suggestion: Try to solve the whole alignment problem. In doing this, think about 1) the key barriers that all of your proposed solutions are running into, and 2) the tools that you have. These solutions (probably) won’t be good, but they are super useful for building inside views. A useful exercise is to build an ‘alignment game tree’, where you (and maybe a few friends) propose solutions to alignment, then break those solutions, then create patches, iteratively.
7. They spend too much time on the fundamental math and CS behind alignment (e.g., trying to complete all of MIRI’s course recommendations or John Wentworth’s study guide) or getting degrees in Math/CS.
These normally take multiple years, and yet I claim that you can get near the frontier of alignment knowledge in ~6 months to a year.
Suggestion: Definitely learn linear algebra, multivariable calculus (the differentiation part, integration doesn’t come up often), probability theory, and basic ML very well. Past that, I recommend learning things as they come up in the course of working on alignment.
While we believe these traps are common, and we want more researchers looking out for them, we also encourage you to consider the law of equal and opposite advice. For each of these traps, it is possible to fall too far in the opposite direction (e.g., never spending time learning relevant math/CS concepts).