Disclaimer: I’m not particularly confident in any of my views here or in my original comment. I mostly commented on the post because the post implied that I supported the idea of a review process whereas my actual opinion is mixed (not uniformly negative). If I hadn’t been named explicitly I wouldn’t have said anything; I don’t want to stop people from trying this out if they think it would be good; it’s quite plausible that someone thinking about this full time would have a vision for it that would be good that I haven’t even considered yet (given how little I’ve thought about it).
I think how excited I’d be would depend a lot more on the details (e.g. who are the reviewers, how much time do they spend, how are they incentivized, what happens after the reviews are completed). But if we just imagine the LessWrong Review extended to the Alignment Forum, I’m not that excited, because I predict (not confidently) that the reviews just wouldn’t be good at engaging with the details. (Mostly because LW / AF comments don’t seem very good at engaging with details on existing LW / AF posts, and because typical LW / AF commenters don’t seem familiar enough with ML to judge details in ML papers.)
My impression is that academic peer-reviewers generally do both of these.
Academic peer review does do both in principle, but I’d say that typically most of the emphasis is on Section Two. Generally the Section One style review is just “yup, this is in fact trying to make progress on a problem academia has previously deemed important, and is not just regurgitating things that people previously said” (i.e. it is significant and novel).
(It is common for bad reviews to just say “this is not significant / not novel” and then ignore Section One entirely, but this is pretty commonly thought of as explicitly “this was a bad review”, unless they actually justified “not significant / not novel” well enough that most others would agree with them.)
Disclaimer: I’m not particularly confident in any of my views here or in my original comment. I mostly commented on the post because the post implied that I supported the idea of a review process whereas my actual opinion is mixed (not uniformly negative). If I hadn’t been named explicitly I wouldn’t have said anything; I don’t want to stop people from trying this out if they think it would be good; it’s quite plausible that someone thinking about this full time would have a vision for it that would be good that I haven’t even considered yet (given how little I’ve thought about it).
I think how excited I’d be would depend a lot more on the details (e.g. who are the reviewers, how much time do they spend, how are they incentivized, what happens after the reviews are completed). But if we just imagine the LessWrong Review extended to the Alignment Forum, I’m not that excited, because I predict (not confidently) that the reviews just wouldn’t be good at engaging with the details. (Mostly because LW / AF comments don’t seem very good at engaging with details on existing LW / AF posts, and because typical LW / AF commenters don’t seem familiar enough with ML to judge details in ML papers.)
Academic peer review does do both in principle, but I’d say that typically most of the emphasis is on Section Two. Generally the Section One style review is just “yup, this is in fact trying to make progress on a problem academia has previously deemed important, and is not just regurgitating things that people previously said” (i.e. it is significant and novel).
(It is common for bad reviews to just say “this is not significant / not novel” and then ignore Section One entirely, but this is pretty commonly thought of as explicitly “this was a bad review”, unless they actually justified “not significant / not novel” well enough that most others would agree with them.)