LW Open Source – Getting Started [warning: not up-to-date]
update: since posting this a few years ago, our architecture and practices have changed slightly. I’ve fixed some things but don’t haven’t checked all the pieces here to see if they’re still accurate.
LessWrong 2.0 is open source, but so far we haven’t done a great job at making contributing a good experience. Here’s our first attempt at fixing this, which covers the basics of “how to contribute” and “high level overview of the site architecture.”
tl;dr: Get a local copy of our git repo installed. This post may be out of date but but maybe still helpful for getting oriented.
Getting Started
Assumed Background Knowledge
Currently, this guide assumes that you know your way around a terminal and github, and have worked with javascript, html, sass/css, and ReactJS.
If you know at least some of those things and are excited to contribute, you can probably make decent headway (and if you run into specific obstacles, comment on them on this post or ping us on intercom, and we’ll see if we can update things to be more clear)
Setting up Locally
Go through the instruction on our Github repo.
You can poke around the codebase. Eventually, to make serious contributions you’ll want to read about our site architecture.
What Contributions Are Helpful?
The most reliably helpful thing would be to tackle outstanding issues that have been tagged on github.
In particular, you can filter them by the tag “good first issue.” (Some of these might require some explanation, but I expect I can explain them fairly easily to a new contributor)
Creating Issues
You can create a new issue. If so, please leave it untagged for the time being (so that admins can quickly look for untagged issues, and sort through them)
Bugs – If you run into a bug, the most helpful thing to do is to search for related keywords in the issue tracker. If you can’t find anything relevant, create a new issue. Try to provide as specific information as possible (your browser, exact links to the post or page where you had the issue, information on how to replicate the bug if you can)
Feature Requests – Feature requests will often need to undergo some discussion to get refined into something executable, and may sometimes need to be split into sub-features.
Features tend to cluster into either “things that are pretty straightforward and we almost certainly want to do” and “things that we have weird conceptual philosophical opinions about that may sometimes be hard to explain succinctly.” (An upcoming post will go into some of this).
After you’ve posted a feature, an admin will tag it (If it’s been a week and we haven’t done so, feel free to bug us about it. We’re still adapting to the role of “open source facilitators” so we may drup things a bit)
Understanding the Codebase
LessWrong 2 is built on VulcanJS, a generic framework for online forums. VulcanJS is in turn built on technologies including:
MeteorJS – an underlying framework that you should rarely have to think about while coding for LW.
MongoDB – a NoSQL database. There is no formal schema.
ReactJS – for the front end. I’m hoping React is enough of The Current Hotness that I don’t have to explain it.
GraphQL – our backend API. The main difference between this and a REST API is that you can query multiple resources at once.
Eventually, it’ll be helpful to have a good understanding of each of those technologies (both to develop new features and fix many kinds of bugs). But for now, the most useful things to know are:
Collections – Mongo databases are organized around collections of documents. For example, the Users collection is where the user objects live. Mongo databases do not technically have a rigid schema, but VulcanJS has a pattern for files that determine the intended schema (which is used by the API, forms and permissions systems to determine what database modifications are allowed)
Components – Our React components are organized in a folder structure based loosely on our collections. (i.e. components related to the User collection go in the packages/lesswrong/components/users folder).
Some edge cases just go in a randomly picked folder (such as the RecentDiscussion components, which involve both comments and posts, but live in the comments folder)
There are multiple ways of creating a ReactJS component. Ideally, each component does one (relatively) simple thing and does it well, with smart components and dumb components separated out. In practice, we haven’t done a great job with this. (Scope creep turns what were once simple components into increasingly complex monstrosities that we should really refactor but haven’t gotten around to it).
We use Vulcan’s registerComponent function to add them as children to a central “Components” component. (read more here)
withComponents (Higher Order Components) – VulcanJS makes a lot of use of React’s higher order components. Often you need a component to have access to particular attributes (the currentUser object, or a function that can edit a given document). Vulcan has a series of components that wrap around other components. They have a names like withCurrentUser, withEditMutation, withDocument. Wrapping a component in a withX typically passes extra information in as a prop.
Fragments – GraphQL queries are made using fragments, which describe the fields from a given database object you want to fetch information on. There’s a common failure mode where someone forgets to update a fragment with new fields, and then the site breaks the next time a component attempts to use information from the new field.
A Vulcan project is divided into packages. It comes with several base packages that provide core functionality (users, voting, routing, etc).
The two packages you’ll most likely need to pay attention to are:
packges/example-forum – A sample forum that defines some key collections (posts, comments, and notifications, as well as “categories” [basically tags] which we aren’t currently using).
We avoid editing example-forum if we can, so we can more easily upgrade it when the main Vulcan branch updates. Instead, if a function in example-forum doesn’t suit our needs, we replace in our custom lesswrong package.
However, in some cases it’s impractical to avoid changing example-forum. When we do, we add a nearby comment:// LESSWRONG – [text explaining the change]
So we can easily search for LESSWRONG and find all changes we’ve made.
packages/lesswrong – Our primary codebase.
This includes new collections, components, and additional functions/attributes/extensions for objects in example-forum.
Going Forward
There’s a lot more worth saying – I plan to write a post and/or github documentation that go over all the most important chunks of the codebase to help new developers get started. That’s a fairly big project. But meanwhile it seemed better to get this post up and see what questions people actually had.
I also plan to write a bit more about some our underlying site philosophy, which will be important for people who want to suggest new features or make front-end changes.
Copying the tl;dr one more time:
Get a local copy of our git repo installed
Check out the Good First Issue filter of our github repo Issues page.
Once you’re familiar, check out the Help Wanted filter.
You’ll still want to read this post to get oriented.
There’s a formatting error in this part of the post:
It should be a code block so that the newline is preserved. That makes it easier to notice that you have to clone two repos:
Also, I found this description confusing:
The schema “branchTitle[issueNumber]” suggests that the branch title should be “defaultSettingsFix[425]” instead. It would be better to describe it as “[branchTitle][issueNumber]” or “branchTitle + issueNumber”, or just say “append the issue number to the branch name”.
Thanks. I’ve added a newline for the first one, and just deleted the entire paragraph about the branch naming because we never really stuck to it.
If you’re at least vaguely interested in this (even if it feels a bit intimidating, or you don’t want to commit to anything), could you reply here with a brief note about your level of interest, and anything in particular that’s acting as a barrier-to-entry?
I have the sense it may be worth investing a lot of effort into making the open source system here good, but wanted some sense of how many people might be willing to help.
I am at least vaguely interested. Possible barriers to entry in my case are (1) having other higher-priority calls on my time, (2) the machine I spend most time on being a Windows box rather than Linux or Mac (I have literally no idea whether that is actually an obstacle; I know it was for old-LW), (3) general disorganization and shortage of Round Tuits, and (4) unfamiliarity with some of the specific technologies used.
Of these, #1 and #3 probably aren’t going to go away, #2 may be a non-issue (perhaps someone can comment?), and #4 serves mostly to make #1 and #3 more severe (because it makes getting started feel like a bigger investment of time).
The most effective way (of those actually available to LW developers) to make me[1] more likely to contribute is probably to provide high-level documentation of the system—the sort of thing in the OP here, in fact—so that getting into it is as painless-in-expectation as possible.
[1] Not that making me in particular more likely to contribute should be anyone’s goal (unless maybe my own), but maybe there are others who feel the same way.
On #2, I just tried the Obvious Thing of running the commands listed in the OP on my Windows machine (ordinary Windows command prompt, not MinGW or WSL or anything of the sort). Here is what happened.
The npm install command failed. What I believe to be the first informative line of the error message said Error: Can’t find Python executable “python”, you can set the PYTHON env variable. Indeed I don’t have a PYTHON environment variable set. A bit of googling suggests that on systems where it is set it ought to point to the Python executable. So I said set PATH=C:\ProgramData\Anaconda3\python.exe which is where my Python interpreter lives, and I got the same error except that it said Can’t find Python executable “C:\ProgramData\Anaconda3\python.exe” which, since that file is definitely there, I took to mean that it failed to work in some way.
I tried doing the same from an “Anaconda prompt” (i.e., a command prompt with all environment variables set up suitably) and it failed in pretty much the same way except that it had ”.EXE” instead of ”.exe” at the end of the Python executable name it said it couldn’t find.
Perhaps we need Python 2 rather than Python 3. So I installed Python 2.7.15 (vanilla install from python.org rather than via Anaconda), set PYTHON=C:\Python27\python.exe, and tried again. This time it apparently worked, though there were a bunch of warnings.
Then npm start failed completely, because it tries to run a bash script. Oh well, never mind.
Next try: launch a WSL bash prompt, cd /mnt/c/Users/… and try again from the top. My Ubuntu-inside-Windows doesn’t have any version of Python installed by default, so sudo apt install python-minimal (which uses Python 2 rather than Python 3) before proceeding.
Now running npm install fails in a different and more immediate way: before it even tries to do anything, three error messages, two of which are … not found and the last of which is Syntax error: word unexpected (expecting “in”). I think what’s happening is that my pseudo-Linux shell is trying to run npm from my Windows installation of npm, and then getting confused by DOS-style line endings or something of the kind. (There is quite good evidence for this in the actual error messages.)
Next try: install nodejs and npm inside pseudo-Linux and try again. The result is the same, which doesn’t make any sense to me.
So, current suggestion: step 0 is “Be running Linux or macOS” for now. May also need “make sure you have Python 2 installed”?
I’ll continue trying to get it working later, but right now I’m at work and need to be doing other things :-).
If you try again, I think you can avoid needing bash. See my comment here.
Continuing attempts on WSL under Windows 10. I killed the bash prompt I had, and started a new one. This now seems to find the correct version of npm in preference to my Windows one, and npm install no longer fails instantly.
Unfortunately, it does still fail. After downloading a bunch of things, I get a bunch of errors associated with something called chromedriver. The first error message line says ERR! Linux-4.4.0-17134-Microsoft. Perhaps this means something has tried to figure out what sort of system it’s running on and doesn’t recognize what WSL reports? Possibly-informative later lines of the error message say ERR! code ELIFECYCLE and ERR! errno ENOENT. I can’t find anything in the node_modules directory relating to chromedriver; perhaps when something fails to install npm destroys the evidence? Scrolling back, I see that there’s an earlier sign of trouble: a line of the form > chromedriver@2.38.3 install [path to chromedriver module] (note: that path no longer exists in my filesystem) followed by a line of the form > node install.js followed by a blank line and then sh: 1: node: not found. After that comes a line beginning with “Vulcan” and then a big tree diagram of modules, then some dependency-related warnings, and then the errors I mentioned above.
[EDITED to add:] It turns out that on Ubuntu (which is the particular flavour of WSL I’m using) the nodejs package doesn’t provide a node executable because there’s another Ubuntu package that wants to call its executable node. So you need to install nodejs-legacy which I think just makes there be a symlink. And with that, npm install completes.
Now, of course, npm start fails. Meteor gets installed, we get a proxy started, but then three lots of Unexpected mongo exit code 100. Restarting. and some guesses at what Mongo might be unhappy about. And, actually, one of them looks super-plausible: this is all happening in an ordinary-Windows directory (via /mnt/c/… on WSL) and “MongoDB does not support filesystems like NFS that do not allow file locking.” So, time to put the whole thing inside WSL-land and see if it fares any better, I guess.
When the
npm installoption results in some errors, what I find most helpful is to try to install those things individually (where you can focus on just the errors relevant to a single thing)So my next step if this were my machine would be:
1. look up the version of chromedriver it’s trying to install (in the package.json file)
2. I can see that it’s chromedrive@r^2.34.1
3. Try
npm install chromedriver@^2.34.14. See what errors come up, do a google search for those errors and see if you find anything useful on stack overflow
You can resolve individual failed-to-download packages individually, and once you’ve done that for each package throwing errors,
npm startshould work.Very much appreciate both giving it a try and then putting a fair chunk of effort into debugging.
Elo tried installing yesterday on Linux and didn’t actually get it up and running either, and upon reflection everyone I know who’s gotten it running locally has been using a mac. Linux should hopefully be easier to debug than Windows.
Meanwhile I’ll update this post with the “step 0”. I think once we’ve gotten a couple people who had troublesome installs to work properly I’ll do some more documentation about best-practices-for-install-troubleshooting.
It didn’t work for me on linux either.
npm installgave out a lot of warnings about outdated packages, I don’t know if they were talking about my preexisting environment or about something in the LW2 repo.npm starttook over half an hour before I went away, and when I came back many hours later it seemed to be stuck in a loop of constantly retrying something that was throwing an error.I will also take a moment to grump that
prestart_lesswrong.shinstalls meteor, which I would rather have done myself (and which I wouldn’t have done throughcurl | bash). Still, I understand that having an easy setup makes people more likely to contribute, so I won’t grump too much. I would have appreciated a warning though, or a separatenpm install_depsscript or something.I’m in a similar place except for #2.
Another thing that would make me more likely to contribute would be having an easy-to-setup test database to work with, instead of having to create users, articles etc. manually.
Good to know – we actually used to have the default Vulcan seed database, but it periodically caused issues with our testing environment. But I can add it back as a command that you have the option of running once, and maybe tweak the database parameters to be more LW-relevant (i.e. including an admin user, a sunshine, a trustLevel1 user, a curated post, etc)
I am interested and going to try to set up. Not sure how much coding capacity I have or how much help I can be.
Thanks! I think “get it set up on my local machine” is a pretty achieavable thing for most people and a good first step to at least remove one trivial inconvenience.
I’m interested. it looks like a fun and valuable thing to play with / work on. I also have the time to do so and am on the lookout for possible projects to work on. Barriers:
Not very experienced, so would have a learning curve on the basic things. clearer explanation for beginners or links to good stating places could be helpful, but i also don’t expect this to be too bad to tackle on my own.
I only have windows and the requirements seem to be either mac or linux. I probably would have tried setting it up locally otherwise.
Don’t have much experience with JS or react—so similarly to the first point, links to good starting places could be helpful, but figuring it out on my own probably wouldn’t be too bad.
Not sure how relevant this still is since it’s from 3 years ago, but thought I’d err on the side of showing my interest rather than not.
Definitely appreciate the show of interest. I’ve become less optimistic about open source in the past few years, but I do still think it’s good on the margin to have people fluent with the LW dev setup.
I think I can’t commit in the foreseeable future to doing tutoring or helping that much with the setup. If you were on mac/linux I’d recommend going ahead and trying to install the setup and see how it went. I’m not sure what to recommend for windows. I’ll ask other LW devs and see if they expect that to possibly-go-smoothly, or if it’s overdetermined to be a huge pain.
Now that I have a copy of the code running on my local machine, I was thinking of grabbing an issue to work on. (I can’t promise any commitment level beyond one issue yet.) I’m trying to be thorough reading what docs there are, and I’ve come across the contributing guidelines which says to check out a roadmap (a Trello board) and join a Slack channel before working on anything. The Trello board doesn’t make much sense to me and I’m not sure if either of these instructions are still important to follow, or if I really should just claim an unassigned issue with a ‘good first issue’ or ‘help wanted’ tag.
Ah, that’s out of date. For now, consider this blogpost more authoritative than anything else until I fix some of the older docs (will try to do that soon).
Here are the ‘good first issues’ :
https://github.com/Discordius/Lesswrong2/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22
Also, feel free to ask any questions about any of the issues – I’m happy to spend a bunch of time helping people orient around the issues themselves and the codebase in general.
As far as I know, go claim an issue.
I use Windows and intend to try to get this running on my machine.
Also, fyi, the first link to the GitHub repo in the tl;dr doesn’t point to the right place.
I got further than gjm reported.
I needed to:
Install Node (That link goes to the exact version listed in .nvmrc)
Install Python 2.7.whatever since I only had Python 3 before this.
Install Visual C++ 2015 Tools for Windows Desktop—This was the weirdest one, but after this, npm install works without error.
Install meteor
I learned that
npm startis unnecessary. It runs a bash script that1. checks meteor is installed
2. creates the file settings.json if it doesn’t exist by copying from sample_settings.json
3. runs command
meteor --settings settings.jsonIf you have meteor installed and create settings.json yourself, you can skip npm start and just run
meteor—settings settings.jsonI verified that I was able to successfully build and run the VulcanJS starter example by doing:
cd Vulcan-Starter code repo
run
npm installmanually copy sample_settings.json to settings.json
run
meteor --settings settings.jsonI then had a website running at http://localhost:3000/
Additional info in the VulcanJS docs includes:
This is what the C++ dev tools fixed. I don’t know if something like
npm install -g windows-build-toolwould have fixed this or not. I didn’t read this until afterwards.Also:
But for LessWrong 2 I am stuck at an error that looks like https://pastebin.com/M1vqTMZd
I think I’m stumped for now. I can clearly get the Vulcan Starter project running, just not LessWrong 2.
Ah, there’s actually a line missing from the sample-settings.json file, which I just added. If you
git pullit should fix the “can’t construct client” issue.Adding the missing line fixed it! I have Lesswrong2 running successfully on Windows at http://localhost:3000/ now.
Huge thanks for all your debugging work!
It’s occurring to me we should probably have direct karma rewards for people who pitch in with the codebase. For now I just upvoted all your comments. :)
This might be a good use case for someone to create a Docker image (or some other container) that has a development environment that just works for new users.
Good work on the guide! Are you planning on combining this with the readme or linking to it from there? There’s a bit of redundancy between the two.
Yeah, my hope was for this to replace the readme file (which was sort of out of date, but figuring out exactly how to replace it was a bit more work than I had bandwidth for today and I wanted to get this thing moving)
FYI, this issue that just came up here is pretty much the easiest possible first-issue-to-handle, so anyone who’s looking for a good get-their-feet-wet issue could check it out:
https://github.com/Discordius/Lesswrong2/issues/575
Can you please change/update the links for “git repo” and “Github repo”? One goes through a redirect which may possibly die out, the other points to
tree/develwhile the instructions in readme advise to build from and intomaster.It doesn’t look like users can assign issues to themselves without being invited to be a contributor.
Link.
Ah, whoops. I’m actually not sure if I can change people’s contributor status or change this policy (habryka might have some additional admin powers that I don’t have). But for now, if you’d like to start working on an issue, just comment on it, and then I’ll assign it to you (or maybe we’ll just make do without officially having assignees).