I think a good estimate is close to $10k. Expect to pay about $100/hr for developer time, and something like 100 hours of work to get from where we are to where we want to be doesn’t seem like a crazy estimate. Historically, the trouble has been finding people willing to do the work, not the money to fund people willing to do the work.
If you can find volunteers who want to do this, we would love code contributions, and you can point them towards here to see what needs to be worked on.
I think you are underestimating this, and a better estimate is “$100k or more”. With an emphasis on the “or more” part.
Historically, the trouble has been finding people willing to do the work, not the money to fund people willing to do the work.
Having “trouble to find people willing to do the work” usually means you are not paying enough to solve the problem. Market price, by definition, is a price at which you can actually buy a product or service, not a price that seems like it should be enough but you just can’t find anyone able and/or willing to accept the deal.
The problem with volunteers is that LW codebase needs too much highly specialized knowledge. Python and Ruby just to get a chance, and then study the code which was optimized for perfomance and backwards compatibility, at the expense of legibility and extensibility. (Database-in-the-database antipattern; values precomputed and cached everywhere.) Most of the professional programmers are simply unable to contribute, without spending a lot of time studying something they will never use again. For a person who has the necessary skills, $10k is about their monthly salary (if you include taxes), and one month feels like too short time to understand the mess of the Reddit code, and implement everything that needs to be done. And the next time, if you need another upgrade, and the same person isn’t available, you need another person to spend the same time to understand the Reddit code.
I believe in long term it would be better to rewrite the code from scratch, but that’s definitely going to take more than one month.
At one point I was planning on making a contribution. It was difficult just getting the code setup and there was very little documentation on the big picture of how everything was supposed to work. It is also very frustrating to run in a development mode. For example, on Mac you have to run it from within a disk image, the VM didn’t work, and setting up new user accounts for testing purposes was a huge pain.
I started trying to understand the code after it was set up, and it is an extremely confusing mess of concepts with virtually no comments, and I am fluent in web development with Python. After 4-6 hours I was making progress on understanding what I needed to make the change I was working on, but I wasn’t there yet. I realized that making the first trivial contribution would probably take another 10-15 hours and stopped. The specific feature I was going to implement was an admin view link that would show the usernames of people who had upvoted / downvoted a comment.
The issues list on GitHub represents at least several hundred hours of work. I think 3 or 4 contributors could probably do a lot of damage in a couple months of free time, if it weren’t quite so unenjoyable. $10K is definitely a huge underestimate for paying an outsider. I do think that a lot of valuable low-hanging fruit, like stopping karma abuses and providing better admin tools, could be done for $10-20K though.
The specific feature I was going to implement was an admin view link that would show the usernames of people who had upvoted / downvoted a comment.
Thanks for trying to work on that one!
setting up new user accounts for testing purposes was a huge pain.
This seems like the sort of thing that we should be able to include with whatever makes the admin account that’s already there; I was watching someone running a test yesterday and while I showed them the way to award accounts karma, I didn’t know of a way to force the karma cache to invalidate, and so they had to wait ~15 minutes to be able to actually make a post with their new test account.
These sorts of usability improvements—a pull request that just adds comments for a section of code you spent a few hours understanding, an improvement to the setup script that makes the dev environment better, are sorely needed and greatly appreciated. In particular, don’t feel at all bad about changing the goal from “I’m going to close out issue X” to “I’m going to make it not as painful to have test accounts,” since those sorts of improvements will lead to probably more than one issue getting closed out.
Maybe it would be easier to make contributions that rely on the code as little as possible—scripts running on separate pages, that woud (1) verify that the person running them is a moderator, and (2) connect to the LW database (these two parts would be common for all such scripts, so have them as two functions in a shared library) -- and then have a separate simple user interface for doing whatever needs to be done.
For example, make a script called “expose_downvotes” that displays a text field where the moderator can copy the comment permalink, and after clicking “OK” a list of usernames who downvoted the specific comment is displayed (preferably with hyperlinks to their user profiles). For the user’s convenience, the comment id is automatically extracted from the permalink.
Then the moderator would simply open this script in a second browser tab, copy link location from the “Permalink” icon at the bottom of a comment, click “OK”, done.
Compared with the solutions integrated into LW web page, this solutions is only slightly more complicated for the moderator, but probably much more simple for the developer to write. Most likely, the moderator will have the page bookmarked, so it’s just “open bookmark in a new tab, switch to old tab, right-click on the comment icon, copy URL, switch to new tab, click on the text field, Ctrl+V, click OK”. Still hundred times more simple (and thousand times faster!) than calling tech support, even assuming their full cooperation.
Each such script could be on a separate page. And they could all be linked together by having another function in the shared library which adds a header containing hyperlinks to all such scripts.
Having “trouble to find people willing to do the work” usually means you are not paying enough to solve the problem.
I had difficulties finding people without mentioning a price; I’m pretty sure the defect was in where and how I was looking for people.
I also agree that it makes more sense to have a small number of programmers make extensive changes, rather than having a large number of people become familiar with how to deal with LW’s code.
I believe in long term it would be better to rewrite the code from scratch, but that’s definitely going to take more than one month.
I will point out there’s no strong opposition to replacing the current LW codebase with something different, so long as we can transfer over all the old posts without breaking any links. The main reason we haven’t been approaching it that way is that it’s harder to make small moves and test their results; either you switch over, or you don’t, and no potential replacement was obviously superior.
I’m new and came here from Sarah Constantin’s blog. I’d like to build a new infrastructure for LW, from scratch. I’m in a somewhat unique position to do so because I’m (1) currently searching for an open source project to do, and (2) taking a few months off before starting my next job, granting the bandwidth to contribute significantly to this project. As it stands right now, I can commit to working full time on this project for the next three months. At that point, I will continue to work on the project part time and it will be robust enough to be used in an alpha or beta state, and attract devs to contribute to further development.
Here is how I envision the basic architecture of this project:
A server that manages all business logic (i.e. posting, moderation, analytics) and interfaces with the frontend (2) and database (3).
A standalone, modular frontend (probably built with React, maybe reusing components provided by Telescope) that is modern, beautiful, and easily extensible/composable from a dev perspective.
A database, possibly NoSql given the nature of the data that needs to be stored (posts, comments, etc). The first concern is security, all others predicated on that.
I will kickstart all three parts and bring them to a good place. After this threshold, I will need help with the frontend—this is not my forte and will be better executed by someone passionate about it.
I’m not asking for any compensation for my work. My incentive is to create a project that is actually immediately useful to someone; open-sourcing it and extending that usability is also nice. I also sympathize with the LW community and the goals laid out in this post.
I considered another approach: reverse-engineer HackerNews and use that as the foundation to be adapted to LW’s unique needs. If this approach would be of greater utility to LW, I’d be happy to take it.
I considered another approach: reverse-engineer HackerNews and use that as the foundation to be adapted to LW’s unique needs
Currently HackerNews and LW both run on the Reddit code base. On of the problems is that Reddit didn’t design their software to be easily adopted to new projects. That means it’s not easily possible to update the code with new versions.
A database, possibly NoSql given the nature of the data that needs to be stored (posts, comments, etc).
I see various people volunteering for different roles. I’d be interested in providing design research and user experience support, which would probably only be needed intermittently if we have someone acting as a product manager. It might be nice to have someone in a light-weight graphic design role as well, and that can be freelance.
Like ananda, I’m happy to do this as an open-contribution project rather than paid. I’ll reach out to Vaniver via email.
Well, if someone would be willing me to pay for one year of full-time work, I would be happy to rewrite the LW code from scratch. Maybe one year is an overestimate, but maybe not—there is this thing known as planning fallacy. That would cost somewhat less than $100k. Let’s say $100k, and that included a reserves for occassionally paying someone else to help me with some specific thing, if needed.
I am not saying that paying me for this job is a rational thing to do; let’s just take this as an approximate estimate of the upper bound. (The lower bound is hoping that one day someone will appear and do it for free. Probably also not a rational thing to do.)
Maybe it was a mistake that I didn’t mention this option sooner… but hearing all the talk about “some volunteers doing it for free in their free time” made me believe that this offer would be seen as exaggerated. (Maybe I was wrong. Sorry, can’t change the past.)
I certainly couldn’t do this in my free time. And trying to fix the existing code would probably take just as much time, the difference being that at the end, instead of new easily maintainable and extensible code, we would have the same old code with a few patches.
And there is also a risk that I am overestimating my abilities here. I never did a project of this scale alone. I mean, I feel quite confident that I could do it in a given time frame, but maybe there would be problems with performance, or some kind of black swan.
I will point out there’s no strong opposition to replacing the current LW codebase with something different, so long as we can transfer over all the old posts without breaking any links.
I would probably try to solve it as a separate step. First, make the new website, as good as possible. Second, import the old content, and redirect the links. Only worry about the import when the new site works as expected.
Or maybe don’t even import the old stuff, and keep the old website frozen. Just static pages, without ability to edit anything. All we lose is the ability to vote or comment on a years-old content. At the moment of transition, open officially the new website, block the ability to post new articles on the old one, but still allow people to post comments on the old one for the following three months. At the end, all old links will work, read-only.
How is the LW codebase so awful? What makes it so much more complicated than just a typical blog, + karma? I feel like I must be missing something.
From a UI perspective it is text boxes and buttons. The data structure that you need to track doesn’t SEEM too complicated (Users have names, karma totals, passwords and roles? What am I not taking into account?
Age, mostly. My understanding is Reddit was one of the first of its kind, and so when building it they didn’t have a good sense of what they were actually making. One of the benefits of switching to something new is not just that it’s using technology people are more likely to be using in their day jobs, but also that the data arrangement is more aligned with how the data is actually used and thought about.
It’s a modified copy of an early Reddit codebase. Besides it has, um, founder effects X-/ -- for example the backend SQL database is used just as an engine behind a handcrafted key-value store...
Historically, the answers have been things like a desire to keep it in the community (given the number of software devs floating around), the hope that volunteer effort would come through, and me not having much experience with sites like those and thus relatively low affordance for that option. But I think if we pay for another major wave of changes, we’ll hire a freelancer through one of those sites.
(Right now we’re discussing how much we’re willing to pay for various changes that could be made, and once I have that list I think it’ll be easy to contact freelancers, see if they’re cheap enough, and then get done the things that make sense to do.)
[edit] I missed one—until I started doing some coordination work, there wasn’t shared knowledge of what sort of changes should actually be bought. The people who felt like they had the authority to design changes didn’t feel like they had the authority to spend money, but the people who felt like they had the authority to spend money didn’t feel like they had the authority to design changes, and both of them had more important things to be working on.
The people who felt like they had the authority to design changes didn’t feel like they had the authority to spend money, but the people who felt like they had the authority to spend money didn’t feel like they had the authority to design changes, and both of them had more important things to be working on.
This sort of leadership vacuum seems to be a common problem in the LW community. Feels to me like people can err more on the side of assuming they have the authority to do things.
I can code in python, but I have no web dev experience—I could work out what algorithms are needed, but I’m not sure I would know how to implement them, at least not off the bat.
Still, I’d be willing to work on it for less then $100 per hour.
If you’re working for $x an hour, do you think you would take fewer that 100/x times as long as someone who is experienced at web dev?
Fair pay would be $x an hour given that it takes me 100/x times as long as someone who is experienced at web dev. However in reality estimates of how long the work will take seem to vary wildly—for instance you and Viliam disagree by an order of magnitude.
The more efficient system might be for me to work with someone who does have some web dev experience, if there is someone else working on this.
I think a good estimate is close to $10k. Expect to pay about $100/hr for developer time, and something like 100 hours of work to get from where we are to where we want to be doesn’t seem like a crazy estimate. Historically, the trouble has been finding people willing to do the work, not the money to fund people willing to do the work.
If you can find volunteers who want to do this, we would love code contributions, and you can point them towards here to see what needs to be worked on.
I think you are underestimating this, and a better estimate is “$100k or more”. With an emphasis on the “or more” part.
Having “trouble to find people willing to do the work” usually means you are not paying enough to solve the problem. Market price, by definition, is a price at which you can actually buy a product or service, not a price that seems like it should be enough but you just can’t find anyone able and/or willing to accept the deal.
The problem with volunteers is that LW codebase needs too much highly specialized knowledge. Python and Ruby just to get a chance, and then study the code which was optimized for perfomance and backwards compatibility, at the expense of legibility and extensibility. (Database-in-the-database antipattern; values precomputed and cached everywhere.) Most of the professional programmers are simply unable to contribute, without spending a lot of time studying something they will never use again. For a person who has the necessary skills, $10k is about their monthly salary (if you include taxes), and one month feels like too short time to understand the mess of the Reddit code, and implement everything that needs to be done. And the next time, if you need another upgrade, and the same person isn’t available, you need another person to spend the same time to understand the Reddit code.
I believe in long term it would be better to rewrite the code from scratch, but that’s definitely going to take more than one month.
At one point I was planning on making a contribution. It was difficult just getting the code setup and there was very little documentation on the big picture of how everything was supposed to work. It is also very frustrating to run in a development mode. For example, on Mac you have to run it from within a disk image, the VM didn’t work, and setting up new user accounts for testing purposes was a huge pain.
I started trying to understand the code after it was set up, and it is an extremely confusing mess of concepts with virtually no comments, and I am fluent in web development with Python. After 4-6 hours I was making progress on understanding what I needed to make the change I was working on, but I wasn’t there yet. I realized that making the first trivial contribution would probably take another 10-15 hours and stopped. The specific feature I was going to implement was an admin view link that would show the usernames of people who had upvoted / downvoted a comment.
The issues list on GitHub represents at least several hundred hours of work. I think 3 or 4 contributors could probably do a lot of damage in a couple months of free time, if it weren’t quite so unenjoyable. $10K is definitely a huge underestimate for paying an outsider. I do think that a lot of valuable low-hanging fruit, like stopping karma abuses and providing better admin tools, could be done for $10-20K though.
Thanks for trying to work on that one!
This seems like the sort of thing that we should be able to include with whatever makes the admin account that’s already there; I was watching someone running a test yesterday and while I showed them the way to award accounts karma, I didn’t know of a way to force the karma cache to invalidate, and so they had to wait ~15 minutes to be able to actually make a post with their new test account.
These sorts of usability improvements—a pull request that just adds comments for a section of code you spent a few hours understanding, an improvement to the setup script that makes the dev environment better, are sorely needed and greatly appreciated. In particular, don’t feel at all bad about changing the goal from “I’m going to close out issue X” to “I’m going to make it not as painful to have test accounts,” since those sorts of improvements will lead to probably more than one issue getting closed out.
Maybe it would be easier to make contributions that rely on the code as little as possible—scripts running on separate pages, that woud (1) verify that the person running them is a moderator, and (2) connect to the LW database (these two parts would be common for all such scripts, so have them as two functions in a shared library) -- and then have a separate simple user interface for doing whatever needs to be done.
For example, make a script called “expose_downvotes” that displays a text field where the moderator can copy the comment permalink, and after clicking “OK” a list of usernames who downvoted the specific comment is displayed (preferably with hyperlinks to their user profiles). For the user’s convenience, the comment id is automatically extracted from the permalink.
Then the moderator would simply open this script in a second browser tab, copy link location from the “Permalink” icon at the bottom of a comment, click “OK”, done.
Compared with the solutions integrated into LW web page, this solutions is only slightly more complicated for the moderator, but probably much more simple for the developer to write. Most likely, the moderator will have the page bookmarked, so it’s just “open bookmark in a new tab, switch to old tab, right-click on the comment icon, copy URL, switch to new tab, click on the text field, Ctrl+V, click OK”. Still hundred times more simple (and thousand times faster!) than calling tech support, even assuming their full cooperation.
Each such script could be on a separate page. And they could all be linked together by having another function in the shared library which adds a header containing hyperlinks to all such scripts.
I had difficulties finding people without mentioning a price; I’m pretty sure the defect was in where and how I was looking for people.
I also agree that it makes more sense to have a small number of programmers make extensive changes, rather than having a large number of people become familiar with how to deal with LW’s code.
I will point out there’s no strong opposition to replacing the current LW codebase with something different, so long as we can transfer over all the old posts without breaking any links. The main reason we haven’t been approaching it that way is that it’s harder to make small moves and test their results; either you switch over, or you don’t, and no potential replacement was obviously superior.
I’m new and came here from Sarah Constantin’s blog. I’d like to build a new infrastructure for LW, from scratch. I’m in a somewhat unique position to do so because I’m (1) currently searching for an open source project to do, and (2) taking a few months off before starting my next job, granting the bandwidth to contribute significantly to this project. As it stands right now, I can commit to working full time on this project for the next three months. At that point, I will continue to work on the project part time and it will be robust enough to be used in an alpha or beta state, and attract devs to contribute to further development.
Here is how I envision the basic architecture of this project:
A server that manages all business logic (i.e. posting, moderation, analytics) and interfaces with the frontend (2) and database (3).
A standalone, modular frontend (probably built with React, maybe reusing components provided by Telescope) that is modern, beautiful, and easily extensible/composable from a dev perspective.
A database, possibly NoSql given the nature of the data that needs to be stored (posts, comments, etc). The first concern is security, all others predicated on that.
I will kickstart all three parts and bring them to a good place. After this threshold, I will need help with the frontend—this is not my forte and will be better executed by someone passionate about it.
I’m not asking for any compensation for my work. My incentive is to create a project that is actually immediately useful to someone; open-sourcing it and extending that usability is also nice. I also sympathize with the LW community and the goals laid out in this post.
I considered another approach: reverse-engineer HackerNews and use that as the foundation to be adapted to LW’s unique needs. If this approach would be of greater utility to LW, I’d be happy to take it.
Thanks for the offer! Maybe we should talk by email? (this username @ gmail.com)
If you don’t get a proper response, it may be worthwhile to make this into its own post, if you have the karma. (Open thread is another option.)
Currently HackerNews and LW both run on the Reddit code base. On of the problems is that Reddit didn’t design their software to be easily adopted to new projects. That means it’s not easily possible to update the code with new versions.
A lot of the data will be votes.
Nitpick: Hackernews isn’t reddit derived. It is some written in arc. And not open source.
I see various people volunteering for different roles. I’d be interested in providing design research and user experience support, which would probably only be needed intermittently if we have someone acting as a product manager. It might be nice to have someone in a light-weight graphic design role as well, and that can be freelance.
Like ananda, I’m happy to do this as an open-contribution project rather than paid. I’ll reach out to Vaniver via email.
I have some front-end experience and would love to help you(I’m a student). Email me at my username @gmail.com
Well, if someone would be willing me to pay for one year of full-time work, I would be happy to rewrite the LW code from scratch. Maybe one year is an overestimate, but maybe not—there is this thing known as planning fallacy. That would cost somewhat less than $100k. Let’s say $100k, and that included a reserves for occassionally paying someone else to help me with some specific thing, if needed.
I am not saying that paying me for this job is a rational thing to do; let’s just take this as an approximate estimate of the upper bound. (The lower bound is hoping that one day someone will appear and do it for free. Probably also not a rational thing to do.)
Maybe it was a mistake that I didn’t mention this option sooner… but hearing all the talk about “some volunteers doing it for free in their free time” made me believe that this offer would be seen as exaggerated. (Maybe I was wrong. Sorry, can’t change the past.)
I certainly couldn’t do this in my free time. And trying to fix the existing code would probably take just as much time, the difference being that at the end, instead of new easily maintainable and extensible code, we would have the same old code with a few patches.
And there is also a risk that I am overestimating my abilities here. I never did a project of this scale alone. I mean, I feel quite confident that I could do it in a given time frame, but maybe there would be problems with performance, or some kind of black swan.
I would probably try to solve it as a separate step. First, make the new website, as good as possible. Second, import the old content, and redirect the links. Only worry about the import when the new site works as expected.
Or maybe don’t even import the old stuff, and keep the old website frozen. Just static pages, without ability to edit anything. All we lose is the ability to vote or comment on a years-old content. At the moment of transition, open officially the new website, block the ability to post new articles on the old one, but still allow people to post comments on the old one for the following three months. At the end, all old links will work, read-only.
Not trolling here, genuine question.
How is the LW codebase so awful? What makes it so much more complicated than just a typical blog, + karma? I feel like I must be missing something.
From a UI perspective it is text boxes and buttons. The data structure that you need to track doesn’t SEEM too complicated (Users have names, karma totals, passwords and roles? What am I not taking into account?
Age, mostly. My understanding is Reddit was one of the first of its kind, and so when building it they didn’t have a good sense of what they were actually making. One of the benefits of switching to something new is not just that it’s using technology people are more likely to be using in their day jobs, but also that the data arrangement is more aligned with how the data is actually used and thought about.
It’s also tied to some pretty old versions of Python and various libraries, and people often need help just getting the development server going.
It’s a modified copy of an early Reddit codebase. Besides it has, um, founder effects X-/ -- for example the backend SQL database is used just as an engine behind a handcrafted key-value store...
If the money is there, why not just pay a freelancer via Gigster or Toptal?
Historically, the answers have been things like a desire to keep it in the community (given the number of software devs floating around), the hope that volunteer effort would come through, and me not having much experience with sites like those and thus relatively low affordance for that option. But I think if we pay for another major wave of changes, we’ll hire a freelancer through one of those sites.
(Right now we’re discussing how much we’re willing to pay for various changes that could be made, and once I have that list I think it’ll be easy to contact freelancers, see if they’re cheap enough, and then get done the things that make sense to do.)
[edit] I missed one—until I started doing some coordination work, there wasn’t shared knowledge of what sort of changes should actually be bought. The people who felt like they had the authority to design changes didn’t feel like they had the authority to spend money, but the people who felt like they had the authority to spend money didn’t feel like they had the authority to design changes, and both of them had more important things to be working on.
This sort of leadership vacuum seems to be a common problem in the LW community. Feels to me like people can err more on the side of assuming they have the authority to do things.
Yeah, a good default is the UNODIR pattern (“I will do X at Y time unless otherwise directed”)
I can code in python, but I have no web dev experience—I could work out what algorithms are needed, but I’m not sure I would know how to implement them, at least not off the bat.
Still, I’d be willing to work on it for less then $100 per hour.
Thanks for the offer!
If you’re working for $x an hour, do you think you would take fewer that 100/x times as long as someone who is experienced at web dev?
Fair pay would be $x an hour given that it takes me 100/x times as long as someone who is experienced at web dev. However in reality estimates of how long the work will take seem to vary wildly—for instance you and Viliam disagree by an order of magnitude.
The more efficient system might be for me to work with someone who does have some web dev experience, if there is someone else working on this.