There’s something deeply discouraging about being told “you’re an X% researcher, and if X>Y, then you should stay in alignment. Otherwise, do a different intervention.” No other effective/productive community does this. (Emphasis added.) I don’t know how to put this, but the vibes are deeply off.
I think this is common, actually.
We apply the logic of only taking top people to other areas. Take medicine. The cost of doing medicine badly is significant, so tons of filters exist. Don’t do well in organic chemistry? You can’t be a doctor. Low GPA? Nope. Can’t get a pretty good score on the MCAT? No again. Get into med school but can’t get an internship? Not gonna be able to practice.
It’s similar for many other high-stakes fields. The US military has a multi-decade long weeding out process to end up as a general. Most corporations effectively do the same. Academic research is brutal in similar ways. All of these systems are broken, but not because they have a filter, more because the filter works poorly, and moral mazes, etc.
Alignment work that’s not good can be costly, and can easily be very net negative. But it’s currently mostly happening outside of institutions with well-defined filters. So I agree that people should probably try to improve their skills if they want to help, but they should also self filter to some extent.
We apply the logic of only taking top people to other areas. Take medicine. The cost of doing medicine badly is significant, so tons of filters exist. Don’t do well in organic chemistry? You can’t be a doctor. Low GPA? Nope. Can’t get a pretty good score on the MCAT? No again. Get into med school but can’t get an internship? Not gonna be able to practice.
I think most fields don’t state it as explicitly as the intersection of EAs and AI Safety researchers tend to do. For example, I’ve definitely heard less explicit “yeah, you might not be smart enough”s in every single other community I’ve been in, even other communities that are very selected in other ways. Most other fields/communities tend to have more of a veneer of a growth mindset, I guess?
I do think it’s true that filtering is important. Given this fact, it probably does make sense to encourage people to be realistic. But my guess is too many people run into the jarring “you might not be smart enough” attitude and self filter way too aggressively, which is what the post is pushing up against.
All of these systems are broken, but not because they have a filter, more because the filter works poorly, and moral mazes, etc.
A complementary explanation is that if the system can’t train people (because nobody there knows how to; or because it’s genuinely hard, e.g. despite the Sequences we don’t have a second Yudkowsky), then the only way to find competent people is to filter for outliers. E.g. if you can’t meaningfully raise the IQ of your recruits, instead filter for higher-IQ recruits.
As pointed out in the linked essay, this strategy makes sense if outcomes are heavy-tailed, i.e. if exceptional people provide most of the value. E.g. if an exceptional general is 1000x as valuable as a good one, then it makes sense to filter for exceptional generals; not so much if the difference is only 3x.
For instance by being acknowledged by the first Yudkowsky as the second one. I was referring here mostly to the difficulty of trying to impart expertise from one person to another. Experts can write down and teach legible insights, but the rest of their expertise (which is often the most important stuff) is very hard to teach.
So filter for exceptional loyalty. To which extent that’s worth it depends on how relatively valuable an exceptionally loyal general is to a merely very loyal one, and on the degree to which you can train loyalty.
I think this is common, actually.
We apply the logic of only taking top people to other areas. Take medicine. The cost of doing medicine badly is significant, so tons of filters exist. Don’t do well in organic chemistry? You can’t be a doctor. Low GPA? Nope. Can’t get a pretty good score on the MCAT? No again. Get into med school but can’t get an internship? Not gonna be able to practice.
It’s similar for many other high-stakes fields. The US military has a multi-decade long weeding out process to end up as a general. Most corporations effectively do the same. Academic research is brutal in similar ways. All of these systems are broken, but not because they have a filter, more because the filter works poorly, and moral mazes, etc.
Alignment work that’s not good can be costly, and can easily be very net negative. But it’s currently mostly happening outside of institutions with well-defined filters. So I agree that people should probably try to improve their skills if they want to help, but they should also self filter to some extent.
I think most fields don’t state it as explicitly as the intersection of EAs and AI Safety researchers tend to do. For example, I’ve definitely heard less explicit “yeah, you might not be smart enough”s in every single other community I’ve been in, even other communities that are very selected in other ways. Most other fields/communities tend to have more of a veneer of a growth mindset, I guess?
I do think it’s true that filtering is important. Given this fact, it probably does make sense to encourage people to be realistic. But my guess is too many people run into the jarring “you might not be smart enough” attitude and self filter way too aggressively, which is what the post is pushing up against.
A complementary explanation is that if the system can’t train people (because nobody there knows how to; or because it’s genuinely hard, e.g. despite the Sequences we don’t have a second Yudkowsky), then the only way to find competent people is to filter for outliers. E.g. if you can’t meaningfully raise the IQ of your recruits, instead filter for higher-IQ recruits.
As pointed out in the linked essay, this strategy makes sense if outcomes are heavy-tailed, i.e. if exceptional people provide most of the value. E.g. if an exceptional general is 1000x as valuable as a good one, then it makes sense to filter for exceptional generals; not so much if the difference is only 3x.
How would you identify a second Yudkowsky? I really don’t like this trope.
By writing ability?
For instance by being acknowledged by the first Yudkowsky as the second one. I was referring here mostly to the difficulty of trying to impart expertise from one person to another. Experts can write down and teach legible insights, but the rest of their expertise (which is often the most important stuff) is very hard to teach.
No military big enough to require multiple layers of general level positions filters for exceptional generals, they all filter for loyalty.
So filter for exceptional loyalty. To which extent that’s worth it depends on how relatively valuable an exceptionally loyal general is to a merely very loyal one, and on the degree to which you can train loyalty.