One of the problems here is that, as of right now, there isn’t much of a middle path between “Stay at Google and do nothing” and “quit your job to do alignment work full-time”. Then there’s the issue of status-seeking vs. altruism as a case of revealed vs. stated preferences. If there was a way to make $750k a year and save the world, people would be all over that. I, personally, would be all over that.
But there isn’t. If we use johnswentworth as an optimistic case, those who would go into independent AI alignment work full-time would make about $90k per year. Of course, anyone that complains about the prospect of ‘only’ making 90k would be derided as a snot-nosed asshole, because we live in a world where hundreds of millions of people subsist upon less than two dollars a day. However, people’s internal calculus chugs on just the same, and Charlie decides to stay at Google.
If you think that we have a fair shot at stopping AI apocalypse, and that AGI is a short-term risk, then it is absolutely rational to optimize for solving AI safety. This is true even if you are entirely selfish.
Also, this essay is about advice given to ambitious people. It’s not about individual people choosing unambitious paths (wheeling). Charlie is a sad example of what can happen to you. I’m not complaining about him.
Yes, selfish agents want to not get turned into paperclips. But they have other goals too. You can prefer alignment be solved, while not wanting to dedicate your mind, body, and soul to waging a jihad against it. Where can Charlie effectively donate, say, 10% of his salary to best mitigate x-risk? Not MIRI (according to MIRI).
One of the problems here is that, as of right now, there isn’t much of a middle path between “Stay at Google and do nothing” and “quit your job to do alignment work full-time”. Then there’s the issue of status-seeking vs. altruism as a case of revealed vs. stated preferences. If there was a way to make $750k a year and save the world, people would be all over that. I, personally, would be all over that.
But there isn’t. If we use johnswentworth as an optimistic case, those who would go into independent AI alignment work full-time would make about $90k per year. Of course, anyone that complains about the prospect of ‘only’ making 90k would be derided as a snot-nosed asshole, because we live in a world where hundreds of millions of people subsist upon less than two dollars a day. However, people’s internal calculus chugs on just the same, and Charlie decides to stay at Google.
If you think that we have a fair shot at stopping AI apocalypse, and that AGI is a short-term risk, then it is absolutely rational to optimize for solving AI safety. This is true even if you are entirely selfish.
Also, this essay is about advice given to ambitious people. It’s not about individual people choosing unambitious paths (wheeling). Charlie is a sad example of what can happen to you. I’m not complaining about him.
Yes, selfish agents want to not get turned into paperclips. But they have other goals too. You can prefer alignment be solved, while not wanting to dedicate your mind, body, and soul to waging a jihad against it. Where can Charlie effectively donate, say, 10% of his salary to best mitigate x-risk? Not MIRI (according to MIRI).