Your posts should be on arXiv
TL;DR: There are many posts on the Alignment Forum/LessWrong that could easily be on arXiv. Putting them on arXiv has several large benefits and (sometimes) very low costs.
Benefits of having posts on arXiv
There are several large benefits of putting posts on arXiv:
1. Much better searchability, shows up in google scholar searches.
2. Additional reads (arXiv sanity, arXiv newsletters, and so on).
3. The article can accumulate citations, which are shown in google/google scholar search results.
1) − 3) lead to more people reading your research, which hopefully leads to more people building on it and maybe useful feedback from outside of the established alignment community. In particular, if people see that the paper already has citations, this will lead to more people reading it, which will lead to more citations, and so on.
You’d gain even more of 2) and 3) from publishing it at a conference, but, unlike arXiv, that’s significant additional work (often still worth it).
There are also some smaller benefits from publishing on arXiv:
4. firmly establishes this as your contribution (not sure, but I think if you only have an alignment forum post, someone could build a bit on it and then claim the whole thing as their contribution because alignment forum posts don’t count?).
5. better citability (e.g. if somebody writes an ML paper to be published in ML venues, it gives more credibility to cite arXiv papers than Alignment Forum/LessWrong posts. The same goes for people writing e.g. Wikipedia articles about alignment.)
How much work is it to submit to arXiv?
Citing DavidHolmes from the comments: “There is a certain amount of moderation on arXiv. This is a little opaque [...]. In writing this I don’t want to give the impression that posting things to arXiv is hard; I have currently 28 papers there, have never had a single problem or delay with moderation, and the submission process generally takes me <15 mins these days.”
Sometimes, I think getting your forum post ready for submission can be as easy as creating a pdf of your post (although if your post was written in LaTeX, they’ll want the tex file). If everything goes well, the submission takes less than an hour.
However, if your post doesn’t look like a research article, you might have to format it more like one (and even then it’s not guaranteed to get in, see this comment thread).
If you are submitting to arXiv for the first time, you might have to get an endorsement from someone who has already published on arXiv. The endorsement sometimes won’t be required if you have an academic email address, so be sure to use that one for submission.
If you’re a Very Busy Alignment Researcher, I’m sure you can outsource large parts of this. E.g. FAR’s comms staff could probably help. I also have worked with a freelancer on similar things in the past (like making publications look nice in LaTeX), feel free to reach out for the contact data.
Highlights from the comments:
DavidHolmes’s tips for how to submit to arXiv.
Dan Hendrycks’s suggestions on which sections of arXiv are suitable for alignment work.
This comment thread suggests that academic formatting/layout will probably be useful.
What types of posts should be on arXiv?
To be clear, I think that most LessWrong posts should not be on arXiv. The bar for submitting to arXiv should be higher than that for submitting to LessWrong/AF. Still, there are many research contributions on the Alignment Forum/LessWrong that wouldn’t look out of place on arXiv. Submitting to arXiv is particularly useful if the post’s target audience is wider than the LessWrong readership.
Here are some examples of posts that also are on arXiv, or, IMO, should be (skewed by what I read and remembered):
Conceptual and theoretical research:
Joe Carlsmith’s Draft report on existential risk from power-seeking AI (it’s on arXiv now, but >1 year after it was published)
Ajeya Cotra’s Draft report on AI timelines (not on arXiv but should be)
Richard Ngo’s AGI safety from first principles (not on arXiv but should be)
I haven’t read it, but from the name of it, it sounds as if ARC’s first technical report: Eliciting Latent Knowledge should maybe be on arXiv
The Truthful AI work by Lin, Evans, Cotton-Barratt, and others (conceptual paper on arXiv)
Hendrycks’s and Mazeika’s X-risk Analysis for AI Research (on arXiv)
Empirical ML research:
High-stakes alignment via adversarial training [Redwood Research report] (on arXiv)
The Truthful AI work by Lin, Evans, Cotton-Barratt, and others (empirical paper on arXiv)
Neel Nanda’s and Tom Lieberum’s A Mechanistic Interpretability Analysis of Grokking (To the surprise of me and several others, this post was actually not accepted by arXiv.)
Lots of ML alignment and safety work by Hendrycks et al., e.g. this, this, and this (all on arXiv)
Language models seem to be much better than humans at next-token prediction could be on arXiv, but would probably need some edits to adhere more to “scientific paper style”
Highlights from the comments:
Dan Hendrycks’s suggestions on what to put (and not put) on arXiv.
Davidmanheim with three more examples of alignment papers he put on arXiv.
If arXiv doesn’t fit
There are also some other posts on the Alignment Forum/LessWrong whose target audience is the wider AI community. I think should be published additionally elsewhere. A great example is (one of my all-time favourite posts) Ajeya Cotra’s Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. This maybe wouldn’t really fit on arXiv (although it wouldn’t be crazy to put on on arXiv either). But there is a range of other venues that might publish it, from Towards Data Science (on the low effort, low prestige end) to the MIT Technology Review (on the high effort, high prestige end).
- EA & LW Forums Weekly Summary (21 Aug − 27 Aug 22’) by (EA Forum; 30 Aug 2022 1:37 UTC; 144 points)
- We Should Prepare for a Larger Representation of Academia in AI Safety by (13 Aug 2023 18:03 UTC; 90 points)
- EA & LW Forums Weekly Summary (21 Aug − 27 Aug 22′) by (30 Aug 2022 1:42 UTC; 57 points)
- Citability of Lesswrong and the Alignment Forum by (8 Jan 2023 22:12 UTC; 48 points)
- 's comment on Why you should publish your research in academic fashion by (EA Forum; 6 Nov 2023 21:29 UTC; 14 points)
- 's comment on Steering GPT-2-XL by adding an activation vector by (14 May 2023 3:13 UTC; 14 points)
- 's comment on Taking features out of superposition with sparse autoencoders more quickly with informed initialization by (23 Sep 2023 18:08 UTC; 3 points)
A while ago I got most of the way to set up a feature on LW/AIAF that would export LW/AIAF posts to a nicely formatted academic-looking PDF that is linkable. I ended up running into a hurdle somewhat close to the end and shelved the feature, but if there is a lot of demand here, I could probably finish up the work, which would make this process even easier.
Please do this!!
A while ago I made a very quick Python script to pull Markdown from LW, then use pandoc to export to a PDF (because I prefer reading physical papers and Latex formatting). I used it somewhat regularly for ~6 months and found that it was good enough for my purposes. I assume the LW developers could write something much better, but I’ve thrown it into this Github [repo](https://github.com/juesato/lw_pdf_exporter/tree/main) in case it’s of help or interest.
I would especially especially love it if it popped out a .tex file that I could edit, since I’m very likely to be using different language on LW than I would in a fancy academic paper.
Seconding the .tex export, since it’s much more useful than just getting a pdf!
I would love this! I’m currently paying someone ~$200 to port my grokking post to LaTeX, getting a PDF automatically would be great
The arXiv really prefers that you upload in tex. For the author this makes it less likely that your paper will be flagged for moderation etc (I guess). So if it were possible to export to Rex I think that for the purposes of uploading to arXiv this would be substantially better. Of course, I don’t know how much more/less work it is…
I was planning this to be a PDF typeset in TeX (created via Pandoc plus some manual massaging), so additionally exporting the TeX shouldn’t be too hard.
Any update on when this might happen?
I am strongly in favor of our very best content going on arXiv. Both communities should engage more with each other.
As follows are suggestions for posting to arXiv. As a rule of thumb, if the content of a blogpost didn’t take >300 hours of labor to create, then it probably should not go on arXiv. Maintaining a basic quality bar prevents arXiv from being overriden by people who like writing up many of their inchoate thoughts; publication standards are different for LW/AF than for arXiv. Even if a researcher spent many hours on the project, arXiv moderators do not want research that’s below a certain bar. arXiv moderators have reminded some professors that they will likely reject papers at the quality level of a Stanford undergraduate team project (e.g., http://cs231n.stanford.edu/2017/reports.html); consequently labor, topicality, and conforming to formatting standards is not sufficient for arXiv approval. Usually one’s first research project won’t be good enough for arXiv. Furthermore, conceptual/philosophical pieces probably should be primarily posted on arXiv’s .CY section. For more technical deep learning content, do not make the mistake of only putting it on .AI; these should probably go on .LG (machine learning) or .CV (computer vision) or .CL (NLP). arXiv’s .ML section is for more statistical/theoretical machine learning audiences. For content to be approved without complications, it should likely conform to standard (ICLR, ICML, NeurIPS, CVPR, ECCV, ICCV, ACL, EMNLP) formatting. This means automatic blogpost exporting is likely not viable. In trying to diffuse ideas to the broader ML community, we should avoid making the arXiv moderators mad at us.
Strongly agree. Three examples of work I’ve put on Arxiv which originated from the forum, which might be helpful as a touchstone. The first was cited 7 times the first year, and 50 more times since. The latter two were posted last year, and have not been indexed by Google as having been cited yet.
As an example of a technical but fairly conceptual paper, there is the Categorizing Goodhart’s law paper. I pushed for this to be a paper rather than just a post, and I think that the resulting exposure was very worthwhile. Scott wrote the original post, though we had discussed Goodhart’s Law quite a bit in LA, and I had written about it on Ribbonfarm. I think the post took significantly less than 300 hours of specific work, but much more than that in earlier thinking and discussions. The comments and discussion around the post was probably fifty hours, but extending it to cover the items I disagreed with, writing it in Latex, making diagrams, and polishing the paper took about another hundred hours between myself, Scott, and others who helped with editing and proofreading.
As an example of a large project with a final report, we commissioned an edited summary report / compilation of our MTAIR sequence. This was at least a thousand hours of total work on the project, probably closer to 3,000, including all the work on the project and writing. The marginal work over the project and posts was a couple thousand dollars in editing, probably amounting to a few dozen hours of work. (We did not move it to latex, and the diagrams were screenshots rather than being done nicely in Latex.)
As an example of a conceptual paper that we put on .CY, here is a model of why people are working on agent foundations which Issa initially posted on the alignment forum. I pushed for rewiting and posting it on ArXiv. I guesstimate no more than 50 hours of work by Issa for the original post, and perhaps another 100 hours total writing and editing for ArXiv. It gets less attention than more technical work, but was also less work. I think that’s fine, and it’s valuable as a more authoritative reference for the arguments than existed previously.
There’s also a poorly researched post on “dynamic safety envelopes” which I put together for other reasons, was never on the forum, and which I didn’t realized was already superseded by Paul Christiano’s and other’s work on various topics. In retrospect, this should not have been put on ArXiv.
I didn’t log the time I spent on the original blog post, and it’s kinda hard to assign hours to this since most of the reading and thinking for the post happened while working on the modeling aspects of the MTAIR project. If I count just the time I sat down to write the blog post, I would guess maybe less than 20 hours.
As for the “convert the post to paper” part, I did log that time and it came out to 89 hours, so David’s estimate of “perhaps another 100 hours” is fairly accurate.
I probably put in an extra 20-60 hours, so the total is probably closer to 150 - which surprises me. I will add that a lot of the conversion time was dealing with writing more, LaTeX figures and citations, which were all, I think, substantive valuable additions. (Changing to a more scholarly style was not substantively valuable, nor was struggling with latex margins and TikZ for the diagrams, and both took some part of the time.)
As an explanation, because this just took me 5 minutes of search: This is the section “Computers and Society (cs.CY)”
Arxiv posts should be on LessWrong*
har har har
I’m actually starting a weekly series that’s basically “collection of arXiv papers that seem important for alignment”.
Here’s a continual stream of related arXiv papers available through reddit and twitter.
https://www.reddit.com/r/mlsafety/
https://twitter.com/topofmlsafety
Note that arXiv does have some gatekeeping: you must get an “endorsement” before submitting your first paper to any subject area. Details.
Ah, I had forgotten about this. I’m happy to endorse people or help them find endorsers.
Also, arXiv mods not allow publishing what they think is unscientific. Their decision takes weeks and is final and can’t be appealed. They also do not provide any explanations. They previously not allowed AI Safety work and they especially hate independent researchers without PhD and good affiliation. Publishing in second-tier scientific journals is simpler than in arXiv.
But there are other scientific papers archives which have milder moderation policy like Philpapers.
The “unscientific” issue is one I’ve never experienced, despite having posted one paper that I think was poorly researched garbage and which, in retrospect, I think clearly doesn’t meet the standards there. I’ve also been unaffiliated and not had trouble posting things, and IIRC, didn’t mention having a PhD at any point. Even the Goodhart paper, which was not a journal submission, had Scott’s affiliation with MIRI and none on my part, and it was fine. So I’m not sure what would have gotten flagged in that way, but it’s probably stuff which doesn’t belong on the alignment forum either. I honestly think that anything about alignment with, say, over 25 upvotes on the Alignment Forum, or, say, over 100 upvotes on LW would be likely to not get flagged.
Hilariously, this does not work. I converted my Grokking post to a PDF (very crudely—just printing to PDF) and uploaded that, and it was rejected:
I should say formatting is likely a large contributing factor for this outcome. Tom Dietterich, an arXiv moderator, apparently had a positive impression of the content of your grokking analysis. However, research on arXiv will be more likely to go live if it conforms to standard (ICLR, NeurIPS, ICML) formatting and isn’t a blogpost automatically exported into a TeX file.
I agree that formatting is the most likely issue. The content of Neel’s grokking work is clearly suitable for arXiv (just very solid ML work). And the style of presentation of the blog post is already fairly similar to a standard paper (e.g. is has an Introduction section, lists contributions in bullet points, …).
So yeah, I agree that formatting/layout probably will do the trick (including stuff like academic citation style).
I did Print to PDF in Word after formatting my Word document to look like a standard LaTeX-exported document, it had no problem going through! But might depend on the particular moderator.
Ah, sorry to hear. I wouldn’t have predicted this from reading arXiv’s content moderation guidelines.
Some reports are not publicised in order not to speed up timelines. And ELK is a bit rambly—I wonder if it will get subsumed by much better content within 2yr. But I do largely agree.
I also strongly endorse this based on my experience. I was a research consultant who created evidence summaries for decision makers in industry and government. This usually involved searching for published content. Anything that wasn’t indexed by Google Scholar/publication repositories was almost always excluded.
Update 2: The nicely LaTeXed version of my Grokking post was also rejected from Arxiv?! I’ll revisit this at some point in the next few weeks, but I’m going to give up on this for now. I consider this a mark against putting posts on Arxiv being an easy and fairly low effort thing to do (though plausibly still worth the effort).
Dang, that’s disappointing :( Did they give any explanation for why it was rejected?
Nope. Though that one is pretty fair—Arxiv must get a lot of submissions
Could this be accomplished with literally zero effort from the post-writers? The tasks of identifying which posts are arXiv-worthy, formatting for submission, and doing the submission all seem like they could be done by entities other than the author. The only issue might be in associating the arXiv submitter account with the right person.
I think the writer should at least approve of the idea of submitting the post to arXiv.
I suspect the arXiv might not be keen on an account that posts papers by a range of people (not including the account-owner as coauthor). This might lead to heavier moderation/whatever. But I could be very wrong!
It probably could, although I’d argue that even if not, quite often it would be worth the author’s time.
True, but I endorse the “zero-effort” plan because it destroys trivial inconveniences.
Some advice for getting papers accepted on arxiv
As some other comments have pointed out, there is a certain amount of moderation on arXiv. This is a little opaque, so below is an attempt to summarise some things that are likely to make it easier to get your paper accepted. I’m sure the list is very incomplete!
In writing this I don’t want to give the impression that posting things to arXiv is hard; I have currently 28 papers there, have never had a single problem or delay with moderation, and the submission process generally takes me <15 mins these days.
Endorsement. When you first attempt to submit a paper you may need to be endorsed. JanBrauner kindly offered below to help people with endorsements; I might also be able to do the same, but I’ve never posted in the CS part of arXiv, so not sure how effective this will be. However, even better to avoid need for moderation. To this end, use an academic email address if you have one; this is quite likely to already be enough. Also, see below on subject classes (endorsement requirements depend on which subject class(es) you want to post in).
Choosing subject classes. Each paper gets one or more subject classes, like CS.AI; see [https://arxiv.org/category_taxonomy] for a list. Some subject classes attract more junk than others, and the ones that attract more junk are more heavily moderated. In mathematics, it is math.GM (General Mathematics) that attracts most junk, hence is most heavily moderated. I guess most people here are looking at CS.AI, I don’t know what this is like. But one easy thing is to minimise cross-listing (adding additional subject classes for your paper), as then you are moderated by all of them.
Write in (la)tex, submit the tex file. You don’t have to do this, but it is standard and preferred by the arXiv, and I suspect makes it less likely your paper gets flagged for moderation. It is also an easy way to make sure your paper looks like a serious academic paper.
It is possible to submit papers on behalf of third parties. I’ve never done this, and I suspect such papers will be more heavily moderated.
If you have multiple authors, it doesn’t really matter who submits. After the submission is posted you are sent a ‘paper password’ allowing coauthors to ‘claim’ the paper; it is then associated to their arXiv account, orcid etc (orcid is optional, but a really good idea, and free).
Finally, a request: please be nice to the moderators! They are generally unpaid volunteers doing a valuable service to the community (e.g. making sure I don’t have to read nonsense proofs of the Riemann hypothesis every morning). Of course it doesn’t feel good if your paper gets held up, but please try not to take it personally.
I am not sure how useful it is, except for status purposes. If you search for a topic in, say, AI safety, the top hits would be LW and the alignment forum, having the same post on arxiv does not add anything, and a PDF is not nearly is convenient to search or browse as a raw post.
It might be useful to add a quick summary of how arXiv works. I vaguely had the impression that anyone could upload PDFs to it, but some of the comments seem to pretty solidly disagree with that.
Wonderful post, thank you. :)
To build on the benefit you noted here:
There are some areas of work whereby it’s useful to not be implicitly communicating that you affiliate with a somewhat weird group like LW or AF folks but you want the content to be read at face value when you share it with folks who are coming from different subcultures and perspectives. I think it’d be hugely valuable for this collection of people who are sharing things.
This seems solvable and very much worth solving!
It would be worth paying someone to do this in a centralized way:
Reach out to authors
Convert to LaTeX, edit
Publish
If someone is interested in doing this, reach out to me (campos.simeon @gmail.com)
I agree. AI safety literature seems irrelevantly disconnected from the ML literature and I imagine that in large part is caused by simple citation dynamics and non-availability in academic literature. There’s arguments against publishing on arXiv when we talk about info hazards but these seem negligible.