Sorry, isn’t this the current system? Or do you mean something automated? See next comment, which I left automation out from.
Sorry, I was suggesting a system in which instead of first posting to LW via the LW interface, you just directly submit to the AIAF, without ever having to think about or go to LW. Then, there is a submission queue that is only visible to some moderators of the AIAF that decides whether your content shows up on both LW and the AIAF, or on neither. This would make it more similar to classical moderated comment-systems. I think a system like this would be clearer to users, since it’s relatively common on the internet, but would also have the problems I described.
It seems like you’re predicting that users would feel angry/powerless based on a kind of system with limited recourse/feedback, and hence everyone would be worse off with this system.
One specific problem with having a submission + admin-review system is that the user has to invest a lot of resources into writing a post, and then only after they invested all of those resources do they get to know whether they get any benefit from what they produced and whether their content (which they might have spent dozens of hours writing) is accepted. This is I think one of the primary things that creates a lot of resentment, and when I talk to people considering publishing in various journals, this is often one of the primary reasons they cite for not doing so.
When designing systems like this, I try to think of ways in which we can give the user feedback at the earliest level of investment, and make incremental benefit available as early as possible. The current system is designed that even if your post doesn’t get promoted to the AIAF, you will likely still get some feedback and benefit from having it on LW. And also, it tries to set expectations that getting a post onto the AIAF is more like a bonus, and the immediate level of reward to expect for the average user, is what you get from posting on LW, which in my experience from user-interviews causes people to publish earlier and faster and get more feedback before getting really invested, in a way that I think results in less resentment overall if it doesn’t get promoted.
I do think some people see very little reward in posting to LW instead of the AIAF, and for those this system is much worse than for the others. Those users still feel like they have to invest all of this upfront labor to get something onto the AIAF, and then have even less certainty than a normal submission system would provide on whether their content gets promoted, and then have even less recourse than a usual academic submission system would provide. I think it is pretty important for us to think more through the experience of those users, of which I think you are a good representative example.
Maybe this would be a good point to recap, from the mod team’s perspective, what are some ways the AF+LW could more clearly set user expectations about how things work. I think it would also be valuable to specify what happens when things don’t go how users want them to go, and to assess whether any reasonable steps should be taken to increase the transparency of AF content moderation. No need to re-do the whole discussion in the post+comments (i.e. no need to justify any decisions) — I just want to make sure this discussion turns into action items as the mods think are appropriate.
I am still thinking through what the right changes we want to make to the system are, but here is a guess on a system that feels good to me:
We do a trial where non-AF members get a button for “submit a comment to the AIAF” and “submit a post to the AIAF” when they log into the alignmentforum.org website
When they click that button a tiny box shows up that explains the setup of posting to the AIAF to them. It says something like the following:
“When you submit a comment or post to the AI Alignment Forum two things happen:
The post/comment is immediately public and commentable on our sister-platform LessWrong.com, where researchers can immediately provide feedback and thoughts on your submission. You can immediately link to your submission and invite others to comment on it.
The post/comment enters a review queue that is reviewed within three business days by an admin on whether to accept your submission to the AI Alignment Forum, and if it does not get accepted, the admin will provide you with a short one-sentence explanation for why they made that decision. The admin uses the discussion and reaction on LessWrong to help us judge whether the content is a good fit for the AI Alignment Forum.
The AI Alignment Forum admins are monitoring all activity on the site, and after you participated in the discussion on the AI Alignment Forum and LessWrong this way, an admin might promote you to a full member of the AI Alignment Forum, who can post to the forum without the need for review, and who can promote other people’s comments and posts from LessWrong.com to the AI Alignment Forum. If you have questions about full membership, or any part of this process, please don’t hesitate to reach out to us (the AIAF admins) via the Intercom in the bottom right corner of the forum.”
When you finish submitting your comment or post you automatically get redirected to the LW version of the corresponding page where you can see your comment/post live, and it will show (just to you) a small badge saying “awaiting AI Alignment Forum review”
I think we probably have the capacity to actually handle this submission queue and provide feedback, though this assumption might just turn out to be wrong, in which case I would revert those changes.
Alternatively, we could provide an option for “either show this content on the AIAF, or show it nowhere”, but I think that would actually end up being kind of messy and complicated, and the setup above strikes me as better. But it does point people quite directly to LessWrong.com in a way that strengthens the association between the two sites in a way that might be costly.
The post/comment enters a review queue that is reviewed within three business days by an admin on whether to accept your submission to the AI Alignment Forum
If you believe in Alignment Forum participants making review judgements, how about using a review queue that works more like StackOverflow for this then admin labor?
I would expect a system that allows Alignment Forum participants to work through the queue to lead to faster reviews and be more easy to scale.
My general philosophy for things like this is “do it in-house for a while so you understand what kinds of problems come up and make sure you have a good experience. After you really have it down maybe consider outsourcing it, or requesting volunteer labor for it.”
So I think eventually asking for a more crowdsourced solution seems reasonable, though I think that would come a few months after.
Sorry, I was suggesting a system in which instead of first posting to LW via the LW interface, you just directly submit to the AIAF, without ever having to think about or go to LW. Then, there is a submission queue that is only visible to some moderators of the AIAF that decides whether your content shows up on both LW and the AIAF, or on neither. This would make it more similar to classical moderated comment-systems. I think a system like this would be clearer to users, since it’s relatively common on the internet, but would also have the problems I described.
One specific problem with having a submission + admin-review system is that the user has to invest a lot of resources into writing a post, and then only after they invested all of those resources do they get to know whether they get any benefit from what they produced and whether their content (which they might have spent dozens of hours writing) is accepted. This is I think one of the primary things that creates a lot of resentment, and when I talk to people considering publishing in various journals, this is often one of the primary reasons they cite for not doing so.
When designing systems like this, I try to think of ways in which we can give the user feedback at the earliest level of investment, and make incremental benefit available as early as possible. The current system is designed that even if your post doesn’t get promoted to the AIAF, you will likely still get some feedback and benefit from having it on LW. And also, it tries to set expectations that getting a post onto the AIAF is more like a bonus, and the immediate level of reward to expect for the average user, is what you get from posting on LW, which in my experience from user-interviews causes people to publish earlier and faster and get more feedback before getting really invested, in a way that I think results in less resentment overall if it doesn’t get promoted.
I do think some people see very little reward in posting to LW instead of the AIAF, and for those this system is much worse than for the others. Those users still feel like they have to invest all of this upfront labor to get something onto the AIAF, and then have even less certainty than a normal submission system would provide on whether their content gets promoted, and then have even less recourse than a usual academic submission system would provide. I think it is pretty important for us to think more through the experience of those users, of which I think you are a good representative example.
I am still thinking through what the right changes we want to make to the system are, but here is a guess on a system that feels good to me:
We do a trial where non-AF members get a button for “submit a comment to the AIAF” and “submit a post to the AIAF” when they log into the alignmentforum.org website
When they click that button a tiny box shows up that explains the setup of posting to the AIAF to them. It says something like the following:
“When you submit a comment or post to the AI Alignment Forum two things happen:
The post/comment is immediately public and commentable on our sister-platform LessWrong.com, where researchers can immediately provide feedback and thoughts on your submission. You can immediately link to your submission and invite others to comment on it.
The post/comment enters a review queue that is reviewed within three business days by an admin on whether to accept your submission to the AI Alignment Forum, and if it does not get accepted, the admin will provide you with a short one-sentence explanation for why they made that decision. The admin uses the discussion and reaction on LessWrong to help us judge whether the content is a good fit for the AI Alignment Forum.
The AI Alignment Forum admins are monitoring all activity on the site, and after you participated in the discussion on the AI Alignment Forum and LessWrong this way, an admin might promote you to a full member of the AI Alignment Forum, who can post to the forum without the need for review, and who can promote other people’s comments and posts from LessWrong.com to the AI Alignment Forum. If you have questions about full membership, or any part of this process, please don’t hesitate to reach out to us (the AIAF admins) via the Intercom in the bottom right corner of the forum.”
When you finish submitting your comment or post you automatically get redirected to the LW version of the corresponding page where you can see your comment/post live, and it will show (just to you) a small badge saying “awaiting AI Alignment Forum review”
I think we probably have the capacity to actually handle this submission queue and provide feedback, though this assumption might just turn out to be wrong, in which case I would revert those changes.
Alternatively, we could provide an option for “either show this content on the AIAF, or show it nowhere”, but I think that would actually end up being kind of messy and complicated, and the setup above strikes me as better. But it does point people quite directly to LessWrong.com in a way that strengthens the association between the two sites in a way that might be costly.
If you believe in Alignment Forum participants making review judgements, how about using a review queue that works more like StackOverflow for this then admin labor?
I would expect a system that allows Alignment Forum participants to work through the queue to lead to faster reviews and be more easy to scale.
My general philosophy for things like this is “do it in-house for a while so you understand what kinds of problems come up and make sure you have a good experience. After you really have it down maybe consider outsourcing it, or requesting volunteer labor for it.”
So I think eventually asking for a more crowdsourced solution seems reasonable, though I think that would come a few months after.