This was a very solid post and I’ve curated it. Here are some of the reasons:
I think that the post is a far more careful analysis of questions around what research to do, what research is scalable, and what are the potential negative effects, than most any other proposals I’ve seen, whilst also containing clear ideas and practical recommendations. (Many posts that optimize for this level of carefulness end up not saying much at all, or at least little of any practical utility, yet this post says quite a lot of interesting things that are practically useful.) There kx a lot of valuable advice, not merely to try to help making narrow superhuman models useful, but how to do it in a way that is helpful for alignment. The section “What kind of projects do and don’t “count”″ is really helpful here.
I appreciate the efforts that Ajeya has made to understand and build consensus around these ideas, talking to people at various orgs (OpenAI, MIRI, more), and this again makes me feel more confident signal-boosting it, given that it contains information about many others’ perspectives on the topic. And more broadly, the whole “Objections and responses” section felt like it did a great job at perspective-taking on others’ concerns and addressing them head on.
I like a lot of the discussion around this post, both in the comments section here and also in the comments by Eliezer and Evan in Robby’s post. (I recommend everyone who reads the OP also reads the discussion in the linked post.)
The main hesitation I have around curating this is that I’m kind of scared of all recommendations for “interesting ideas for using machine learning that might be really important for AGI”. This part of me feels like everything has the potential to be used for capabilities, and that someone pursuing this line of work may do so quite inadvertently (and it will not be possible to “put the ball back in the urn”), or just end up proving their competence at building scalably useful models and then getting hired by a research lab to do work that will give them lots of money to do stuff with big models.
I am scared about most everything in this space, yet I don’t think I endorse “no action”, and this does seem to me like one of the most promising and careful posts and approaches. For me the most cruxy sections were “Isn’t this not neglected because lots of people want useful AI” and “Will this cause harm by increasing investment in scaling AI?”. For the first, I think if I believed the claims Ajeya makes as strongly as I think Ajeya does, I’d feel notably more relieved overall about encouraging this kind of work. For the second I didn’t feel persuaded by the arguments. I think that there are few people who are respected remotely on these sorts of questions or who are thinking strategically about them (especially in public). I think the x-risk and alignment communities have in the past had 100:1 outsized impact with actions taken and research directions pursued.
In sum, I continue to be generally terrified, but this was an excellent post and one of the relatively least terrifying things. Thank you very much for the post.
This was a very solid post and I’ve curated it. Here are some of the reasons:
I think that the post is a far more careful analysis of questions around what research to do, what research is scalable, and what are the potential negative effects, than most any other proposals I’ve seen, whilst also containing clear ideas and practical recommendations. (Many posts that optimize for this level of carefulness end up not saying much at all, or at least little of any practical utility, yet this post says quite a lot of interesting things that are practically useful.) There kx a lot of valuable advice, not merely to try to help making narrow superhuman models useful, but how to do it in a way that is helpful for alignment. The section “What kind of projects do and don’t “count”″ is really helpful here.
I appreciate the efforts that Ajeya has made to understand and build consensus around these ideas, talking to people at various orgs (OpenAI, MIRI, more), and this again makes me feel more confident signal-boosting it, given that it contains information about many others’ perspectives on the topic. And more broadly, the whole “Objections and responses” section felt like it did a great job at perspective-taking on others’ concerns and addressing them head on.
I like a lot of the discussion around this post, both in the comments section here and also in the comments by Eliezer and Evan in Robby’s post. (I recommend everyone who reads the OP also reads the discussion in the linked post.)
The main hesitation I have around curating this is that I’m kind of scared of all recommendations for “interesting ideas for using machine learning that might be really important for AGI”. This part of me feels like everything has the potential to be used for capabilities, and that someone pursuing this line of work may do so quite inadvertently (and it will not be possible to “put the ball back in the urn”), or just end up proving their competence at building scalably useful models and then getting hired by a research lab to do work that will give them lots of money to do stuff with big models.
I am scared about most everything in this space, yet I don’t think I endorse “no action”, and this does seem to me like one of the most promising and careful posts and approaches. For me the most cruxy sections were “Isn’t this not neglected because lots of people want useful AI” and “Will this cause harm by increasing investment in scaling AI?”. For the first, I think if I believed the claims Ajeya makes as strongly as I think Ajeya does, I’d feel notably more relieved overall about encouraging this kind of work. For the second I didn’t feel persuaded by the arguments. I think that there are few people who are respected remotely on these sorts of questions or who are thinking strategically about them (especially in public). I think the x-risk and alignment communities have in the past had 100:1 outsized impact with actions taken and research directions pursued.
In sum, I continue to be generally terrified, but this was an excellent post and one of the relatively least terrifying things. Thank you very much for the post.