update: since posting this a few years ago, our architecture and practices have changed slightly. I’ve fixed some things but don’t haven’t checked all the pieces here to see if they’re still accurate.
LessWrong 2.0 is open source, but so far we haven’t done a great job at making contributing a good experience. Here’s our first attempt at fixing this, which covers the basics of “how to contribute” and “high level overview of the site architecture.”
tl;dr: Get a local copy of our git repo installed. This post may be out of date but but maybe still helpful for getting oriented.
Getting Started
Assumed Background Knowledge
Currently, this guide assumes that you know your way around a terminal and github, and have worked with javascript, html, sass/css, and ReactJS.
If you know at least some of those things and are excited to contribute, you can probably make decent headway (and if you run into specific obstacles, comment on them on this post or ping us on intercom, and we’ll see if we can update things to be more clear)
In particular, you can filter them by the tag “good first issue.” (Some of these might require some explanation, but I expect I can explain them fairly easily to a new contributor)
Creating Issues
You can create a new issue. If so, please leave it untagged for the time being (so that admins can quickly look for untagged issues, and sort through them)
Bugs – If you run into a bug, the most helpful thing to do is to search for related keywords in the issue tracker. If you can’t find anything relevant, create a new issue. Try to provide as specific information as possible (your browser, exact links to the post or page where you had the issue, information on how to replicate the bug if you can)
Feature Requests – Feature requests will often need to undergo some discussion to get refined into something executable, and may sometimes need to be split into sub-features.
Features tend to cluster into either “things that are pretty straightforward and we almost certainly want to do” and “things that we have weird conceptual philosophical opinions about that may sometimes be hard to explain succinctly.” (An upcoming post will go into some of this).
After you’ve posted a feature, an admin will tag it (If it’s been a week and we haven’t done so, feel free to bug us about it. We’re still adapting to the role of “open source facilitators” so we may drup things a bit)
Understanding the Codebase
LessWrong 2 is built on VulcanJS, a generic framework for online forums. VulcanJS is in turn built on technologies including:
MeteorJS – an underlying framework that you should rarely have to think about while coding for LW.
MongoDB – a NoSQL database. There is no formal schema.
ReactJS – for the front end. I’m hoping React is enough of The Current Hotness that I don’t have to explain it.
GraphQL – our backend API. The main difference between this and a REST API is that you can query multiple resources at once.
Eventually, it’ll be helpful to have a good understanding of each of those technologies (both to develop new features and fix many kinds of bugs). But for now, the most useful things to know are:
Collections – Mongo databases are organized around collections of documents. For example, the Users collection is where the user objects live. Mongo databases do not technically have a rigid schema, but VulcanJS has a pattern for files that determine the intended schema (which is used by the API, forms and permissions systems to determine what database modifications are allowed)
Components – Our React components are organized in a folder structure based loosely on our collections. (i.e. components related to the User collection go in the packages/lesswrong/components/users folder).
Some edge cases just go in a randomly picked folder (such as the RecentDiscussion components, which involve both comments and posts, but live in the comments folder)
There are multiple ways of creating a ReactJS component. Ideally, each component does one (relatively) simple thing and does it well, with smart components and dumb components separated out. In practice, we haven’t done a great job with this. (Scope creep turns what were once simple components into increasingly complex monstrosities that we should really refactor but haven’t gotten around to it).
We use Vulcan’s registerComponent function to add them as children to a central “Components” component. (read more here)
withComponents (Higher Order Components) – VulcanJS makes a lot of use of React’s higher order components. Often you need a component to have access to particular attributes (the currentUser object, or a function that can edit a given document). Vulcan has a series of components that wrap around other components. They have a names like withCurrentUser, withEditMutation, withDocument. Wrapping a component in a withX typically passes extra information in as a prop.
Fragments – GraphQL queries are made using fragments, which describe the fields from a given database object you want to fetch information on. There’s a common failure mode where someone forgets to update a fragment with new fields, and then the site breaks the next time a component attempts to use information from the new field.
A Vulcan project is divided into packages. It comes with several base packages that provide core functionality (users, voting, routing, etc).
The two packages you’ll most likely need to pay attention to are:
packges/example-forum – A sample forum that defines some key collections (posts, comments, and notifications, as well as “categories” [basically tags] which we aren’t currently using).
We avoid editing example-forum if we can, so we can more easily upgrade it when the main Vulcan branch updates. Instead, if a function in example-forum doesn’t suit our needs, we replace in our custom lesswrong package.
However, in some cases it’s impractical to avoid changing example-forum. When we do, we add a nearby comment:
// LESSWRONG – [text explaining the change]
So we can easily search for LESSWRONG and find all changes we’ve made.
packages/lesswrong – Our primary codebase.
This includes new collections, components, and additional functions/attributes/extensions for objects in example-forum.
Going Forward
There’s a lot more worth saying – I plan to write a post and/or github documentation that go over all the most important chunks of the codebase to help new developers get started. That’s a fairly big project. But meanwhile it seemed better to get this post up and see what questions people actually had.
I also plan to write a bit more about some our underlying site philosophy, which will be important for people who want to suggest new features or make front-end changes.
LW Open Source – Getting Started [warning: not up-to-date]
update: since posting this a few years ago, our architecture and practices have changed slightly. I’ve fixed some things but don’t haven’t checked all the pieces here to see if they’re still accurate.
LessWrong 2.0 is open source, but so far we haven’t done a great job at making contributing a good experience. Here’s our first attempt at fixing this, which covers the basics of “how to contribute” and “high level overview of the site architecture.”
tl;dr: Get a local copy of our git repo installed. This post may be out of date but but maybe still helpful for getting oriented.
Getting Started
Assumed Background Knowledge
Currently, this guide assumes that you know your way around a terminal and github, and have worked with javascript, html, sass/css, and ReactJS.
If you know at least some of those things and are excited to contribute, you can probably make decent headway (and if you run into specific obstacles, comment on them on this post or ping us on intercom, and we’ll see if we can update things to be more clear)
Setting up Locally
Go through the instruction on our Github repo.
You can poke around the codebase. Eventually, to make serious contributions you’ll want to read about our site architecture.
What Contributions Are Helpful?
The most reliably helpful thing would be to tackle outstanding issues that have been tagged on github.
In particular, you can filter them by the tag “good first issue.” (Some of these might require some explanation, but I expect I can explain them fairly easily to a new contributor)
Creating Issues
You can create a new issue. If so, please leave it untagged for the time being (so that admins can quickly look for untagged issues, and sort through them)
Bugs – If you run into a bug, the most helpful thing to do is to search for related keywords in the issue tracker. If you can’t find anything relevant, create a new issue. Try to provide as specific information as possible (your browser, exact links to the post or page where you had the issue, information on how to replicate the bug if you can)
Feature Requests – Feature requests will often need to undergo some discussion to get refined into something executable, and may sometimes need to be split into sub-features.
Features tend to cluster into either “things that are pretty straightforward and we almost certainly want to do” and “things that we have weird conceptual philosophical opinions about that may sometimes be hard to explain succinctly.” (An upcoming post will go into some of this).
After you’ve posted a feature, an admin will tag it (If it’s been a week and we haven’t done so, feel free to bug us about it. We’re still adapting to the role of “open source facilitators” so we may drup things a bit)
Understanding the Codebase
LessWrong 2 is built on VulcanJS, a generic framework for online forums. VulcanJS is in turn built on technologies including:
MeteorJS – an underlying framework that you should rarely have to think about while coding for LW.
MongoDB – a NoSQL database. There is no formal schema.
ReactJS – for the front end. I’m hoping React is enough of The Current Hotness that I don’t have to explain it.
GraphQL – our backend API. The main difference between this and a REST API is that you can query multiple resources at once.
Eventually, it’ll be helpful to have a good understanding of each of those technologies (both to develop new features and fix many kinds of bugs). But for now, the most useful things to know are:
Collections – Mongo databases are organized around collections of documents. For example, the Users collection is where the user objects live. Mongo databases do not technically have a rigid schema, but VulcanJS has a pattern for files that determine the intended schema (which is used by the API, forms and permissions systems to determine what database modifications are allowed)
Components – Our React components are organized in a folder structure based loosely on our collections. (i.e. components related to the User collection go in the packages/lesswrong/components/users folder).
Some edge cases just go in a randomly picked folder (such as the RecentDiscussion components, which involve both comments and posts, but live in the comments folder)
There are multiple ways of creating a ReactJS component. Ideally, each component does one (relatively) simple thing and does it well, with smart components and dumb components separated out. In practice, we haven’t done a great job with this. (Scope creep turns what were once simple components into increasingly complex monstrosities that we should really refactor but haven’t gotten around to it).
We use Vulcan’s registerComponent function to add them as children to a central “Components” component. (read more here)
withComponents (Higher Order Components) – VulcanJS makes a lot of use of React’s higher order components. Often you need a component to have access to particular attributes (the currentUser object, or a function that can edit a given document). Vulcan has a series of components that wrap around other components. They have a names like withCurrentUser, withEditMutation, withDocument. Wrapping a component in a withX typically passes extra information in as a prop.
Fragments – GraphQL queries are made using fragments, which describe the fields from a given database object you want to fetch information on. There’s a common failure mode where someone forgets to update a fragment with new fields, and then the site breaks the next time a component attempts to use information from the new field.
A Vulcan project is divided into packages. It comes with several base packages that provide core functionality (users, voting, routing, etc).
The two packages you’ll most likely need to pay attention to are:
packges/example-forum – A sample forum that defines some key collections (posts, comments, and notifications, as well as “categories” [basically tags] which we aren’t currently using).
We avoid editing example-forum if we can, so we can more easily upgrade it when the main Vulcan branch updates. Instead, if a function in example-forum doesn’t suit our needs, we replace in our custom lesswrong package.
However, in some cases it’s impractical to avoid changing example-forum. When we do, we add a nearby comment:
// LESSWRONG – [text explaining the change]So we can easily search for LESSWRONG and find all changes we’ve made.
packages/lesswrong – Our primary codebase.
This includes new collections, components, and additional functions/attributes/extensions for objects in example-forum.
Going Forward
There’s a lot more worth saying – I plan to write a post and/or github documentation that go over all the most important chunks of the codebase to help new developers get started. That’s a fairly big project. But meanwhile it seemed better to get this post up and see what questions people actually had.
I also plan to write a bit more about some our underlying site philosophy, which will be important for people who want to suggest new features or make front-end changes.
Copying the tl;dr one more time:
Get a local copy of our git repo installed
Check out the Good First Issue filter of our github repo Issues page.
Once you’re familiar, check out the Help Wanted filter.
You’ll still want to read this post to get oriented.