Agree that this is definitely a plausible strategy, and that it doesn’t get anywhere near as much attention as it seemingly deserves, for reasons unknown to me. Strong upvote for the post, I want to see some serious discussion on this. Some preliminary thoughts:
How did we get here?
If I had to guess, the lack of discussion on this seems likely due to a founder effect. The people pulling the alarm in the early days of AGI safety concerns were disproportionately to the technical/philosophical side rather than to the policy/outreach/activism side.
In early days, focus on the technical problem makes sense. When you are the only person in the world working on AGI, all the delay in the world won’t help unless the alignment problem gets solved. But we are working at very different margins nowadays.
There’s also an obvious trap which makes motivated reasoning really easy. Often, the first thing that occurs when thinking about slowing down AGI development is sabotage—maybe because this feels urgent and drastic? It’s an obviously bad idea, and maybe that lets us to motivated stopping.
Maybe the “technical/policy” dichotomy is keeping us from thinking of obvious ways we could be making the future much safer? The outreach org you propose seems like not really either. Would be interested in brainstorming other major ways to affect the world, but not gonna do that in this comment.
HEY! FTX! OVER HERE!!
You should submit this to the Future Fund’s ideas competition, even though it’s technically closed. I’m really tempted to do it myself just to make sure it gets done, and very well might submit something in this vein once I’ve done a more detailed brainstorm.
You should submit this to the Future Fund’s ideas competition, even though it’s technically closed. I’m really tempted to do it myself just to make sure it gets done, and very well might submit something in this vein once I’ve done a more detailed brainstorm.
Probably a good idea, though I’m less optimistic about the form being checked. I’ll plan on writing something up today. If I don’t end up doing that today for whatever reason, akrasia, whatever, I’ll DM you.
I am confident that they would not let it being technically closed stop them from considering the proposal if someone they respected pointed at the proposal (and likely even if they didn’t), and I’d be happy to do the pointing for you if necessary. If that seems necessary, DM me.
Please DM me if you end up starting some sort of project along these lines. I have some (admittedly limited) experience working in media/public relations, and can probably help a bit.
If I had to guess, the lack of discussion on this seems likely due to a founder effect. The people pulling the alarm in the early days of AGI safety concerns were disproportionately to the technical/philosophical side rather than to the policy/outreach/activism side.
Also, the people pulling the alarm in the early days of AGI safety concerns, are also people interested in AGI. They find it cool. I get the impression that some of them think aligned people should also try to win the AGI race, so doing capabilities research and being willing to listen to alignment concerns is good. (I disagree with this position and I don’t think it’s a strawman, but it might be a bit unfair.)
Many of the people that got interested in AGI safety later on also find AGI cool, or have done some capabilities research (e.g. me), so thinking that what we’ve done is evil is counterintuitive.
Yeah most people in the AGI safety concern group are also technological progress enthusiasts (with good reason, technological progress is generally awesome). And the recent string of “alignement startup” fondations points toward an unhealthy mix of capacity and safety research.
Agree that this is definitely a plausible strategy, and that it doesn’t get anywhere near as much attention as it seemingly deserves, for reasons unknown to me. Strong upvote for the post, I want to see some serious discussion on this. Some preliminary thoughts:
How did we get here?
If I had to guess, the lack of discussion on this seems likely due to a founder effect. The people pulling the alarm in the early days of AGI safety concerns were disproportionately to the technical/philosophical side rather than to the policy/outreach/activism side.
In early days, focus on the technical problem makes sense. When you are the only person in the world working on AGI, all the delay in the world won’t help unless the alignment problem gets solved. But we are working at very different margins nowadays.
There’s also an obvious trap which makes motivated reasoning really easy. Often, the first thing that occurs when thinking about slowing down AGI development is sabotage—maybe because this feels urgent and drastic? It’s an obviously bad idea, and maybe that lets us to motivated stopping.
Maybe the “technical/policy” dichotomy is keeping us from thinking of obvious ways we could be making the future much safer? The outreach org you propose seems like not really either. Would be interested in brainstorming other major ways to affect the world, but not gonna do that in this comment.
HEY! FTX! OVER HERE!!
You should submit this to the Future Fund’s ideas competition, even though it’s technically closed. I’m really tempted to do it myself just to make sure it gets done, and very well might submit something in this vein once I’ve done a more detailed brainstorm.
Probably a good idea, though I’m less optimistic about the form being checked. I’ll plan on writing something up today. If I don’t end up doing that today for whatever reason, akrasia, whatever, I’ll DM you.
I am confident that they would not let it being technically closed stop them from considering the proposal if someone they respected pointed at the proposal (and likely even if they didn’t), and I’d be happy to do the pointing for you if necessary. If that seems necessary, DM me.
Please DM me if you end up starting some sort of project along these lines. I have some (admittedly limited) experience working in media/public relations, and can probably help a bit.
For anyone else also interested, you should add yourself on this spreadsheet. https://docs.google.com/spreadsheets/d/1WEsiHjTub9y28DLtGVeWNUyPO6tIm_75bMF1oeqpJpA/edit?usp=sharing
It’s very useful for people building such an organisation to know of interested people, and vice versa.
If you don’t want to use the spreadsheet, you can also DM me and I’ll keep you in the loop privately.
If you’re making such an organisation, please contact me. I’d like to work with you.
Also, the people pulling the alarm in the early days of AGI safety concerns, are also people interested in AGI. They find it cool. I get the impression that some of them think aligned people should also try to win the AGI race, so doing capabilities research and being willing to listen to alignment concerns is good. (I disagree with this position and I don’t think it’s a strawman, but it might be a bit unfair.)
Many of the people that got interested in AGI safety later on also find AGI cool, or have done some capabilities research (e.g. me), so thinking that what we’ve done is evil is counterintuitive.
Yeah most people in the AGI safety concern group are also technological progress enthusiasts (with good reason, technological progress is generally awesome). And the recent string of “alignement startup” fondations points toward an unhealthy mix of capacity and safety research.