Under “Understand cooperation”, you should add Metagov (many relevant projects under this umbrella, please visit the website, in particular, DAO Science), “ecosystems of intelligence” agenda (itself pursued by Verses, Active Inference Institute, Gaia Consortium, Digital Gaia, and Bioform Labs). This is more often more practical than theoretical work though, so the category names (“Theory” > “Understanding cooperation”) wouldn’t be totally reasonable for it, but this is also true for a lot of entires already on the post.
In general, the science of cooperation, game theory, digital assets and money, and governance is mature, with a lot of academics working in it in different countries. Picking up just a few projects “familiar to the LessWrong crowd” is just reinforcing the bubble.
Also, speaking about pure “technical alignment/AI safety” agendas that “nobody on LW knows and talks about”, check out the 16 projects already funded by the “Safe Learning-Enabled Systems” NSF grant program. All these projects have received grants from $250k to $800k and are staffed with teams of academics in American universities.
I’ve added a line about the ecosystems. Nothing else in the umbrella strikes me as direct work (Public AI is cool but not alignment research afaict). (I liked your active inference paper btw, see ACS.)
A quick look suggests that the stable equilibrium things aren’t in scope—not because they’re outgroup but because this post is already unmanageable without handling policy, governance, political economy and ideology. The accusation of site bias against social context or mechanism was perfectly true last year, but no longer, and my personal scoping should not be taken as indifference.
Of the NSF people, only Sharon Li strikes me as doing things relevant to AGI.
I’m talking about science of governance, digitalised governance, and theories of contracting, rather than not-so-technical object-level policy and governance work that is currently done at institutions. And this is absolutely not to the detriment of that work, but just as a selection criteria for this post, which could decide to focus on technical agendas where technical visitors of LW may contribute to.
The view that there is a sharp divide between “AGI-level safety” and “near-term AI safety and ethics” is itself controversial, e.g., Scott Aaronson doesn’t share it. I guess this isn’t a justification for including all AI ethics work that is happening, but of the NSF projects, definitely more than one (actually, most of them) appear to me upon reading abstracts as potentially relevant for AGI safety. Note that this grant program of NSF is in a partnership with Open Philanthropy and OpenPhil staff participate in the evaluation of the projects. So, I don’t think they would select a lot of projects irrelevant for AGI safety.
Under “Understand cooperation”, you should add Metagov (many relevant projects under this umbrella, please visit the website, in particular, DAO Science), “ecosystems of intelligence” agenda (itself pursued by Verses, Active Inference Institute, Gaia Consortium, Digital Gaia, and Bioform Labs). This is more often more practical than theoretical work though, so the category names (“Theory” > “Understanding cooperation”) wouldn’t be totally reasonable for it, but this is also true for a lot of entires already on the post.
In general, the science of cooperation, game theory, digital assets and money, and governance is mature, with a lot of academics working in it in different countries. Picking up just a few projects “familiar to the LessWrong crowd” is just reinforcing the bubble.
The “LessWrong bias” is also felt in the decision to omit all the efforts that contribute to the creation of the stable equilibrium for the civilisation on which an ASI can land. Here’s my stab at what is going into that from one month ago; and this is Vitalik Buterin’s stab from yesterday.
Also, speaking about pure “technical alignment/AI safety” agendas that “nobody on LW knows and talks about”, check out the 16 projects already funded by the “Safe Learning-Enabled Systems” NSF grant program. All these projects have received grants from $250k to $800k and are staffed with teams of academics in American universities.
Ta!
I’ve added a line about the ecosystems. Nothing else in the umbrella strikes me as direct work (Public AI is cool but not alignment research afaict). (I liked your active inference paper btw, see ACS.)
A quick look suggests that the stable equilibrium things aren’t in scope—not because they’re outgroup but because this post is already unmanageable without handling policy, governance, political economy and ideology. The accusation of site bias against social context or mechanism was perfectly true last year, but no longer, and my personal scoping should not be taken as indifference.
Of the NSF people, only Sharon Li strikes me as doing things relevant to AGI.
Happy to be corrected if you know better!
I’m talking about science of governance, digitalised governance, and theories of contracting, rather than not-so-technical object-level policy and governance work that is currently done at institutions. And this is absolutely not to the detriment of that work, but just as a selection criteria for this post, which could decide to focus on technical agendas where technical visitors of LW may contribute to.
The view that there is a sharp divide between “AGI-level safety” and “near-term AI safety and ethics” is itself controversial, e.g., Scott Aaronson doesn’t share it. I guess this isn’t a justification for including all AI ethics work that is happening, but of the NSF projects, definitely more than one (actually, most of them) appear to me upon reading abstracts as potentially relevant for AGI safety. Note that this grant program of NSF is in a partnership with Open Philanthropy and OpenPhil staff participate in the evaluation of the projects. So, I don’t think they would select a lot of projects irrelevant for AGI safety.
If the funder comes through I’ll consider a second review post I think