This seems great in principle. The below is meant in the spirit of [please consider these things while moving forward with this], and not [please don’t move forward until you have good answers on everything].
That said:
First, I think it’s important to clearly distinguish:
A great world would have a lot more AI safety orgs. (yes)
Conditional on many new AI safety orgs starting, the world is in a better place. (maybe)
Intervening to facilitate the creation of new AI safety orgs makes the world better. (also maybe)
This program would be doing (3), so it’s important to be aware that (1) is not in itself much of an argument. I expect that it’s very hard to do (3) well, and that even a perfect version doesn’t allow us to jump to the (1) of our dreams. But I still think it’s a good idea!
Some thoughts that might be worth considering (very incomplete, I’m sure):
Impact of potential orgs will vary hugely.
Your impact will largely come down to [how much you increase (/reduce) the chance that positive (/negative) high-impact orgs get created].
This may be best achieved by aiming to create many orgs. It may not.
Of course [our default should be zero new orgs] is silly, but so would be [we’re aiming to create as many new orgs as possible].
You’ll have a bunch of information, time and levers that funders don’t have, so I don’t think such considerations can be left to funders.
In the below I’ll be mostly assuming that you’re not agnostic to the kind of orgs you’re facilitating (since this would be foolish :)). However, I note that even if you were agnostic, you’d inevitably make choices that imply significant tradeoffs.
Consider the incentive landscape created by current funding sources.
Consider how this compares to a highly-improved-by-your-lights incentive landscape.
Consider what you can do to change things for the better in this regard.
If anything seems clearly suboptimal as things stand, consider spending significant effort making this case to funders as soon as possible.
Consider what data you could gather on potential failure modes, or simply on dynamics that are non-obvious at the outset. (anonymized appropriately) Gather as much data as possible.
If you don’t have the resources to do a good job at experimentation, data gathering etc., make this case to funders and get those resources. Make the case that the cost of this is trivial relative to the opportunity cost of failing to gather the information.
The most positive-for-the-world orgs are likely among the hardest to create.
By default, orgs created are likely to be doing not-particularly-neglected things (similar selection pressures that created the current field act on new orgs; non-neglected areas of the field correlate positively with available jobs and in-demand skills...).
By default, it’s much more likely to select for [org that moves efficiently in some direction] than [org that picks a high-EV-given-what’s-currently-known direction].
Given that impact can easily vary by a couple of orders of magnitude (and can be negative), direction is important.
It’s long-term direction that’s important. In principle, an org that moves efficiently in some direction could radically alter that direction later. In practice, that’s uncommon—unless this mindset existed at the outset.
Perhaps facilitating this is another worthwhile intervention?? - i.e. ensuring that safety orgs have an incentive to pivot to higher-EV approaches, rather than to continue with a [low EV-relative-to-counterfactual, but high comparative advantage] approach.
Making it easier to create any kind of safety org doesn’t change the selection pressures much (though I do think it’s a modest improvement). If all the branches are a little lower, it’s still the low-hanging-fruit that tends to be picked first. It may often be easier to lower the low branches too.
If possible, you’d want to disproportionately lower the highest branches. Clearly this is easier said than done. (e.g. spending a lot of resources on helping those with hard-to-make-legible ideas achieve legibility, [on a process level, if necessary], so that there’s not strong selection for [easy to make legible])
Ground truth feedback on the most important kinds of progress is sparse-to-non-existent.
You’ll be using proxies (for [what seems important], [what impact we’d expect org x to have], [what impact direction y has had], [what impact new org z has had] etc. etc.).
Most proxies aren’t great.
The most natural proxies and metrics will tend to be the same ones others are using. This may help to get a project funded. It tends to act against neglectedness.
Using multiple, non-obvious proxies is worth a thought.
However, note that you don’t have the True Name of [AI safety], [alignment]… in your head: you have a vague, confused proxy.
One person coming up with multiple proxies, will tend to mean creating various proxies to their own internal proxy. That’s still a single point of failure.
If you all clearly understand the importance of all the proxies you’re using, that’s probably a bad sign.
It’s much better to create a great org slowly, than a mediocre org quickly. This can easily happen with (some of) the same people, entailing a high opportunity cost.
I think one of the most harmful dynamics at present is the expectation that people/orgs should have a concretely mapped out agenda/path-to-impact within a few months. This strongly selects against neglectedness.
Even Marius’ response to this seems to have the wrong emphasis: ”Second, a great agenda just doesn’t seem like a necessary requirement. It seems totally fine for me to replicate other people’s work, extend existing agendas, or ask other orgs if they have projects to outsource (usually they do) for a year or so and build skills during that time. After a while, people naturally develop their own new ideas and then start developing their own agendas.” I.e. that the options are:
Have a great agenda.
Replicate existing work, extend existing agenda, grab existing ideas to work on.
Where is the [spend time focusing on understanding the problem more deeply, and forming new ideas / approaches]? Of course this may sometime entail some replication/extension, but that shouldn’t be the motivation.
Financial pressures and incentives are important here: [We’ll fund you for six months to focus on coming up with new approaches] amounts to [if you pick a high-variance approach, your org may well cease to exist in six months]. If the aim is to get an org to focus on exploration for six months, guaranteed funding for two years is a more realistic minimum.
Of course this isn’t directly within your control—but it’s the kind of thing you might want to make a case for to funders.
Again, the more you’re able to shape the incentive landscape for future orgs, the more you’ll be able to avoid unhelpful instrumental constraints, and focus on facilitating the creation of the kind of orgs that should exist.
Also worth considering that the requirement for this kind of freedom is closer to [the people need near-guaranteed financial support for 2+ years]. Where an org is uncertain/experimental, it may still make sense to give the org short-term funding, but all the people involved medium-term funding.
Hi Joe, thanks a lot for your thoughtful comment! We think you’re making some valid points here and will take your suggestions and questions into consideration
This seems great in principle.
The below is meant in the spirit of [please consider these things while moving forward with this], and not [please don’t move forward until you have good answers on everything].
That said:
First, I think it’s important to clearly distinguish:
A great world would have a lot more AI safety orgs. (yes)
Conditional on many new AI safety orgs starting, the world is in a better place. (maybe)
Intervening to facilitate the creation of new AI safety orgs makes the world better. (also maybe)
This program would be doing (3), so it’s important to be aware that (1) is not in itself much of an argument. I expect that it’s very hard to do (3) well, and that even a perfect version doesn’t allow us to jump to the (1) of our dreams. But I still think it’s a good idea!
Some thoughts that might be worth considering (very incomplete, I’m sure):
Impact of potential orgs will vary hugely.
Your impact will largely come down to [how much you increase (/reduce) the chance that positive (/negative) high-impact orgs get created].
This may be best achieved by aiming to create many orgs. It may not.
Of course [our default should be zero new orgs] is silly, but so would be [we’re aiming to create as many new orgs as possible].
You’ll have a bunch of information, time and levers that funders don’t have, so I don’t think such considerations can be left to funders.
In the below I’ll be mostly assuming that you’re not agnostic to the kind of orgs you’re facilitating (since this would be foolish :)). However, I note that even if you were agnostic, you’d inevitably make choices that imply significant tradeoffs.
Consider the incentive landscape created by current funding sources.
Consider how this compares to a highly-improved-by-your-lights incentive landscape.
Consider what you can do to change things for the better in this regard.
If anything seems clearly suboptimal as things stand, consider spending significant effort making this case to funders as soon as possible.
Consider what data you could gather on potential failure modes, or simply on dynamics that are non-obvious at the outset. (anonymized appropriately)
Gather as much data as possible.
If you don’t have the resources to do a good job at experimentation, data gathering etc., make this case to funders and get those resources. Make the case that the cost of this is trivial relative to the opportunity cost of failing to gather the information.
The most positive-for-the-world orgs are likely among the hardest to create.
By default, orgs created are likely to be doing not-particularly-neglected things (similar selection pressures that created the current field act on new orgs; non-neglected areas of the field correlate positively with available jobs and in-demand skills...).
By default, it’s much more likely to select for [org that moves efficiently in some direction] than [org that picks a high-EV-given-what’s-currently-known direction].
Given that impact can easily vary by a couple of orders of magnitude (and can be negative), direction is important.
It’s long-term direction that’s important. In principle, an org that moves efficiently in some direction could radically alter that direction later. In practice, that’s uncommon—unless this mindset existed at the outset.
Perhaps facilitating this is another worthwhile intervention?? - i.e. ensuring that safety orgs have an incentive to pivot to higher-EV approaches, rather than to continue with a [low EV-relative-to-counterfactual, but high comparative advantage] approach.
Making it easier to create any kind of safety org doesn’t change the selection pressures much (though I do think it’s a modest improvement). If all the branches are a little lower, it’s still the low-hanging-fruit that tends to be picked first. It may often be easier to lower the low branches too.
If possible, you’d want to disproportionately lower the highest branches. Clearly this is easier said than done. (e.g. spending a lot of resources on helping those with hard-to-make-legible ideas achieve legibility, [on a process level, if necessary], so that there’s not strong selection for [easy to make legible])
Ground truth feedback on the most important kinds of progress is sparse-to-non-existent.
You’ll be using proxies (for [what seems important], [what impact we’d expect org x to have], [what impact direction y has had], [what impact new org z has had] etc. etc.).
Most proxies aren’t great.
The most natural proxies and metrics will tend to be the same ones others are using. This may help to get a project funded. It tends to act against neglectedness.
Using multiple, non-obvious proxies is worth a thought.
However, note that you don’t have the True Name of [AI safety], [alignment]… in your head: you have a vague, confused proxy.
One person coming up with multiple proxies, will tend to mean creating various proxies to their own internal proxy. That’s still a single point of failure.
If you all clearly understand the importance of all the proxies you’re using, that’s probably a bad sign.
It’s much better to create a great org slowly, than a mediocre org quickly. This can easily happen with (some of) the same people, entailing a high opportunity cost.
I think one of the most harmful dynamics at present is the expectation that people/orgs should have a concretely mapped out agenda/path-to-impact within a few months. This strongly selects against neglectedness.
Even Marius’ response to this seems to have the wrong emphasis:
”Second, a great agenda just doesn’t seem like a necessary requirement. It seems totally fine for me to replicate other people’s work, extend existing agendas, or ask other orgs if they have projects to outsource (usually they do) for a year or so and build skills during that time. After a while, people naturally develop their own new ideas and then start developing their own agendas.”
I.e. that the options are:
Have a great agenda.
Replicate existing work, extend existing agenda, grab existing ideas to work on.
Where is the [spend time focusing on understanding the problem more deeply, and forming new ideas / approaches]? Of course this may sometime entail some replication/extension, but that shouldn’t be the motivation.
Financial pressures and incentives are important here: [We’ll fund you for six months to focus on coming up with new approaches] amounts to [if you pick a high-variance approach, your org may well cease to exist in six months]. If the aim is to get an org to focus on exploration for six months, guaranteed funding for two years is a more realistic minimum.
Of course this isn’t directly within your control—but it’s the kind of thing you might want to make a case for to funders.
Again, the more you’re able to shape the incentive landscape for future orgs, the more you’ll be able to avoid unhelpful instrumental constraints, and focus on facilitating the creation of the kind of orgs that should exist.
Also worth considering that the requirement for this kind of freedom is closer to [the people need near-guaranteed financial support for 2+ years]. Where an org is uncertain/experimental, it may still make sense to give the org short-term funding, but all the people involved medium-term funding.
Hi Joe, thanks a lot for your thoughtful comment! We think you’re making some valid points here and will take your suggestions and questions into consideration