I’ve also been thinking about how to boost reviewing in the alignment field. Unsure if AF is the right venue, but it might be. I was more thinking along the lines of academic peer review. Main advantages of reviewing generally I see are: - Encourages sharper/clearer thinking and writing; - Makes research more inter-operable between groups; - Catches some errors; - Helps filter the most important results.
Obviously peer review is imperfect at all of these. But so is upvoting or not doing review systematically.
I think the main reasons alignment researchers currently don’t submit their work to peer reviewed venues are: - Existing peer reviewed venues are super slow (something like 4 month turnaround is considered good). - Existing peer reviewed venues have few expert reviewers in alignment, so reviews are low quality and complain about things which are distractions. - Existing peer reviewed venues often have pretty low-effort reviews. - Many alignment researchers have not been trained in how to write ML papers that get accepted, so they have bad experiences at ML conferences that turn them off.
One hypothesis I’ve heard from people is that actually alignment researchers are great at sending out their work for feedback from actual peers, and the AF is good for getting feedback as well, so there’s no problem that needs fixing. This seems unlikely. Critical feedback from people who aren’t already thinking on your wavelength is uncomfortable to get and effortful to integrate, so I’d expect natural demand to be lower than optimal. Giving careful feedback is also effortful so I’d expect it to be undersupplied.
I’ve been considering a high-effort ‘journal’ for alignment research. It would be properly funded and would pay for high-effort reviews, aiming for something like a 1 week desk-reject and a 2 week initial review time. By focusing on AGI safety/Alignment you could maintain a pool of actually relevant expert reviewers. You’d probably want to keep some of the practice of academic review process (e.g., structured fields for feedback from reviewers), ranking or sorting papers for significance and quality; but not others (e.g., allow markdown or google doc submissions).
In my dream version of this, you’d use prediction markets about the ultimate impact of the paper, and then uprate the reviews from profitable impact forecasters.
Would be good to talk with people who are interested in this or variants. I’m pretty uncertain about the right format, but I think we can probably build something better than what we have now and the potential for value is large. I’m especially worried about the alignment community forming cliques that individually feel good about their work and don’t engage with concerns from other researchers and people feeling so much urgency that they make sloppy logical mistakes that end up being extremely costly.
I’ve talked to a few people who have suggested journal or conference ideas, but they never happened. I think it was mostly a mix of not knowing how to do it well and (mostly) they were busy with other stuff. Someone probably actually needs to take initiative on this if we want our research to be more ‘academic’.
Regardless of whether a journal is created or not, I’ve certainly wished I had more academic collaborators or someone who could teach me how to publish work that ends up being accepted within the ML community. As an indepedent researcher, it feels like the gap is too large and causes too much friction to figure things out and get started.
Yeah, I think it requires some specialist skills, time, and a bit of initiative. But it’s not deeply super hard.
Sadly, I think learning how to write papers for ML conferences is pretty time consuming. It’s one of the main things a phd student spends time learning in the first year or two of their phd. I do think there’s a lot that’s genuinely useful about that practice though, it’s not just jumping through hoops.
Did you see LeCun’s proposal about how to improve academic review here? It strikes me as very good and I’d love if AI safety/x-risk community had a system like this.
I’m suspicious about creating a separate journal, rather than concentrating efforts around existing institutions: LW/AF. I think it would be better to fund LW exactly for this purpose and add monetary incentives for providing good reviews of research writing on LW/AF (and, of course, the research writing itself could be incentivised in this way, too).
Then, turn AF in exactly the kind of “journal” that you proposed, as I described here.
Yeah, LeCun’s proposal seems interesting. I was actually involved in an attempt to modify OpenReview to push along those lines a couple years ago. But it became very much a ‘perfect is the enemy of the good’ situation where the technical complexity grew too fast relative to the amount of engineering effort devoted to it.
What makes you suspicious about a separate journal? Diluting attention? Hard to make new things? Or something else? I’m sympathetic to diluting attention, but bet that making a new thing wouldn’t be that hard.
Attention dilution, exactly. Ultimately, I want (because I think this will be more effective) all relevant work to be syndicated on LW/AF (via Linkposts, and review posts), not the other way around: AI safety researchers had to subscribe to arxiv sanity, google AI blog, all relevant standalone blogs such as Bengio’s and Scott Aaronson’s, etc. etc. etc., all by themselves and separately.
I even think if LW hired part-time staff dedicated to doing this would be very valuable.
Also, alignment newsletters, to further pre-process information, don’t live. Shah tried to revive his newsletter mid last year, but it didn’t survive for long. Part-time could also curate such an “AF newsletter”, I don’t think it takes Shah’s competence to do this well.
FWIW I think doing something like the newsletter well actually does take very rare skills. Summarizing well is really hard. Having relevant/interesting opinions about the papers is even harder.
I’ve also been thinking about how to boost reviewing in the alignment field. Unsure if AF is the right venue, but it might be. I was more thinking along the lines of academic peer review. Main advantages of reviewing generally I see are:
- Encourages sharper/clearer thinking and writing;
- Makes research more inter-operable between groups;
- Catches some errors;
- Helps filter the most important results.
Obviously peer review is imperfect at all of these. But so is upvoting or not doing review systematically.
I think the main reasons alignment researchers currently don’t submit their work to peer reviewed venues are:
- Existing peer reviewed venues are super slow (something like 4 month turnaround is considered good).
- Existing peer reviewed venues have few expert reviewers in alignment, so reviews are low quality and complain about things which are distractions.
- Existing peer reviewed venues often have pretty low-effort reviews.
- Many alignment researchers have not been trained in how to write ML papers that get accepted, so they have bad experiences at ML conferences that turn them off.
One hypothesis I’ve heard from people is that actually alignment researchers are great at sending out their work for feedback from actual peers, and the AF is good for getting feedback as well, so there’s no problem that needs fixing. This seems unlikely. Critical feedback from people who aren’t already thinking on your wavelength is uncomfortable to get and effortful to integrate, so I’d expect natural demand to be lower than optimal. Giving careful feedback is also effortful so I’d expect it to be undersupplied.
I’ve been considering a high-effort ‘journal’ for alignment research. It would be properly funded and would pay for high-effort reviews, aiming for something like a 1 week desk-reject and a 2 week initial review time. By focusing on AGI safety/Alignment you could maintain a pool of actually relevant expert reviewers. You’d probably want to keep some of the practice of academic review process (e.g., structured fields for feedback from reviewers), ranking or sorting papers for significance and quality; but not others (e.g., allow markdown or google doc submissions).
In my dream version of this, you’d use prediction markets about the ultimate impact of the paper, and then uprate the reviews from profitable impact forecasters.
Would be good to talk with people who are interested in this or variants. I’m pretty uncertain about the right format, but I think we can probably build something better than what we have now and the potential for value is large. I’m especially worried about the alignment community forming cliques that individually feel good about their work and don’t engage with concerns from other researchers and people feeling so much urgency that they make sloppy logical mistakes that end up being extremely costly.
I’ve talked to a few people who have suggested journal or conference ideas, but they never happened. I think it was mostly a mix of not knowing how to do it well and (mostly) they were busy with other stuff. Someone probably actually needs to take initiative on this if we want our research to be more ‘academic’.
Regardless of whether a journal is created or not, I’ve certainly wished I had more academic collaborators or someone who could teach me how to publish work that ends up being accepted within the ML community. As an indepedent researcher, it feels like the gap is too large and causes too much friction to figure things out and get started.
Yeah, I think it requires some specialist skills, time, and a bit of initiative. But it’s not deeply super hard.
Sadly, I think learning how to write papers for ML conferences is pretty time consuming. It’s one of the main things a phd student spends time learning in the first year or two of their phd. I do think there’s a lot that’s genuinely useful about that practice though, it’s not just jumping through hoops.
I strongly agree with most of this.
Did you see LeCun’s proposal about how to improve academic review here? It strikes me as very good and I’d love if AI safety/x-risk community had a system like this.
I’m suspicious about creating a separate journal, rather than concentrating efforts around existing institutions: LW/AF. I think it would be better to fund LW exactly for this purpose and add monetary incentives for providing good reviews of research writing on LW/AF (and, of course, the research writing itself could be incentivised in this way, too).
Then, turn AF in exactly the kind of “journal” that you proposed, as I described here.
Yeah, LeCun’s proposal seems interesting. I was actually involved in an attempt to modify OpenReview to push along those lines a couple years ago. But it became very much a ‘perfect is the enemy of the good’ situation where the technical complexity grew too fast relative to the amount of engineering effort devoted to it.
What makes you suspicious about a separate journal? Diluting attention? Hard to make new things? Or something else? I’m sympathetic to diluting attention, but bet that making a new thing wouldn’t be that hard.
Attention dilution, exactly. Ultimately, I want (because I think this will be more effective) all relevant work to be syndicated on LW/AF (via Linkposts, and review posts), not the other way around: AI safety researchers had to subscribe to arxiv sanity, google AI blog, all relevant standalone blogs such as Bengio’s and Scott Aaronson’s, etc. etc. etc., all by themselves and separately.
I even think if LW hired part-time staff dedicated to doing this would be very valuable.
Also, alignment newsletters, to further pre-process information, don’t live. Shah tried to revive his newsletter mid last year, but it didn’t survive for long. Part-time could also curate such an “AF newsletter”, I don’t think it takes Shah’s competence to do this well.
FWIW I think doing something like the newsletter well actually does take very rare skills. Summarizing well is really hard. Having relevant/interesting opinions about the papers is even harder.