[edit: I might add a bullet-point summary to the bottom of this post without justifications after I’ve had a chance to see the comments and which objections people actually raise]
I assumed you didn’t want people to raise technical objections to this post, and let you present your argument first. But if you want them now, here are some objections that gjm didn’t mention:
Our goal is to make something better than the existing LW software / UX. But we must also allow free linking in, to posts and comments, from ordinary blogs and other sites. These links will be the natural gateway for users new to the community, and lurkers. They must also have a UX at least as good as today’s LW, exposing the features of the new / non-web solution, or else this whole endeavor will be a regression and doomed to fail.
People won’t like a plaintext-oriented interface; they want rich text, inline images and tables, which in practice (in the world of NNTP) translates to HTML. But HTML is far too rich (and unsuitable for human editing). We need an equivalent to comments’ markdown support (or something better that would also be usable for posts). With NNTP, this would be client dependent, so at best it would vary by poster and at worst it would simply not be supported (or not out of the box).
Editing of posts and comments is a crucial feature which NNTP doesn’t provide.
Other features we need or are used to: RSS feed; tagging; user management and direct messaging without the trivial inconvenience of creating a new pseudonymous email account on a different site; server-based state (e.g. ‘unread’ message state in user inbox) for those with multiple client devices; …..
Any proposal that isn’t for a gradual change to the existing site will need to be run on a new site. The old lesswrong.com won’t be shut down until the new one has clearly succeeded. So you’ll need to directly compete with lesswrong.com to convince users to switch. You can make an LW → NNTP gateway, but not an NNTP → LW gateway; that is, lesswrong.com can’t automatically publish content (comments) posted via NNTP (or any other protocol, really). So during this period of competition, even if users cross-post, discussion threads will be separate. The new software will have to be clearly superior to convince LW users to switch, let alone diaspora blog authors.
I didn’t, but I was assuming people would anyway. I was actually hoping for higher-level objections like the problems I listed in the post, but, well. I’ll answer you anyway and maybe edit the post later. Most of these fall under 3.2 and 3.3 in the outline.
is actually the only part of the problem for which no off-the-shelf solution exists. The short version is that the site UX, instead of talking to a database directly on the backend, would talk to NNTP on the backend. Links to arbitrary posts (as a non-exhaustive example) could be as simple as https://newlesswrong/message-id. Top level posts could support some subject-generated shorthand, perhaps.
I actually do want a plaintext-oriented interface, but I know I’m in the minority. You’ve expressed the solution to the markup problem yourself, though: Markdown is already the effectively-default format on Usenet and all extant NNTP clients. In fact it’s the rise of lightweight markup in general and Markdown in particular that convinced me this could ever be more than a pipe dream. There are more powerful forms of lightweight markup that could be used; the key is that the input format must be readable as plaintext for interoperability between the web and native clients to be possible.
I believe that cancels and supersedes can be kludged to support something that looks like editing even if it isn’t. I may be wrong about this; in particular I’m not sure how supersedes interact with reply chains, because they’re unusable for that purpose on Usenet proper.
User management has an established solution, and DMing can be implemented as a LW mailbox (that only takes local messages) or a forwarding address, or both. RSS is probably easy. Server based state is easy for the ‘default’ website (really an in-browser client, covered in 1.1 and 1.7) but may be hard for native clients (but anyone using a native client presumably knows what they’re getting into). Tagging may be hard, I’m not sure. Karma is definitely hard, but may be unnecessary.
The short version of this is that, designed correctly, any author or site can adopt the network without being any worse off than they currently are; that is, cooperate-defect leaves you no worse off than defect-defect. The long version is...pretty much all of section 2, actually.
(ETA: I am answering this in moderate detail not to encourage technical back-and-forth but to demonstrate that I have thought this through)
I think this reduces to problem 5. Namely, you can’t replace lesswrong.com with a completely new NNTP-backed site as your opening move.
Ordinary Markdown is insufficient because it doesn’t do inline images and tables, which are sometimes important for posts. There are more powerful alternatives, and some markdown supersets. But I’ll bet they’re not supported by NNTP clients. At best you’ll find one client (or client plugin) that supports what you want. But forcing everyone to use the same client negates the point of NNTP, and isn’t plausible anyway because we need clients for different platforms including mobile.
This definitely requires proof. In addition to the issue with reply threads, NNTP clients expect to be able to cache articles. Propagation of supersede/replace is at best delayed and at worst out-of-order, and will vary between clients. This is IMO a critical issue, a major LW feature.
I’m surprised that you say “anyone using a native client presumably knows what they’re getting into”. You’re proposing NNTP as a superior solution, but also saying anyone who actually chooses to use NNTP will have a hard time and/or fewer features. And I think this applies to other, more important features and not just server based state.
Adoption, for site owners, will at minimum require a significant time/money investment, and at least trivial inconveniences to web-based readers due to unavoidable minor undesirable UX differences. So moving to the new platform needs to make people better off, not just no worse than before. The same applies to posters/readers: they won’t switch over without a compelling reason, nor should we expect them to.
I completely forgot about karma, but it’s so important that I’m promoting it to a new item. I think karma is pretty good, and some alternatives may be better, but nothing at all is much worse.
Meta: replying in a list-based format is inconvenient and I tentatively suggest making a separate reply for each significant list item.
Agreed that it’s inconvenient. Rather than separate it I’ll cut it down. 2, 4, and 6 all collapse to “client feature sets differ”; this is a meta-feature, not a meta-bug. The downside of client control is that not everybody sees the same thing. The upside of client control is client competition, which has similar benefits to market competition.
Solving adoption is the point of section 2 and is too long to describe here. Note that, as mentioned, I do not actually expect this to be adopted. The world isn’t that kind.
3 is a legitimate problem and will be addressed, but it’s the sort of thing where I need to spin up INN and see how it actually behaves when presented with edit-style supersedes. If there is a weak point in my “this is possible” argument, this is it.
2, 4, and 6 all collapse to “client feature sets differ”; this is a meta-feature, not a meta-bug.
That misses my point. Some features, like voting, can’t be implemented as clientside features, because clients would need to communicate about them (and establish consensus).
A subset of HTML is still unsuited to human editing. Even more so than full HTML, because it doesn’t have the justification of being a complex and extendable syntax. A superset of markdown would be much more usable, for people writing plaintext, than a subset of HTML. Especially if, as now, the majority of posts and comments require more or less only regular markdown and no superset features like tables.
Asciidoc might be an alternative when more power is needed. I haven’t used it, but ESR once said that it does markdown’s job better than Markdown itself.
HTML is wholly unsuited to human editing or even reading. I blame it for ruining email. Well, that and top-posting.
A subset of HTML is still unsuited to human editing.
People from the 90s would disagree and rich text editors can output whatever. Of course markdown is better for specifying minor formatting while you write the content—that’s what it was explicitly designed for. However the advantage of HMTL is ubiquity.
the majority of posts and comments require more or less only regular markdown
The majority of posts and comments use only links, bold/italics, and an occasional bulleted list. Inline images are culturally disapproved of and tables are rare. At this level pretty much anything would work.
I don’t get your point here. HTML’s ubiquity is important for display, not for editing. Markdown is converted to HTML for display. As a user I prefer writing in markdown to writing in HTML. Don’t you?
At this level pretty much anything would work.
That doesn’t mean anything would be equally as comfortable as anything else.
We are talking about the acceptable format for messages as they are processed and stored by the system, right? Ease of input is a separate issue and your editor can and should allow you to write in whatever way you find most comfortable.
The format for server-side processing and storage should be the input format unless there is specific cause not to use it (3.2). Conversion to display formats should be done client-side and as late as possible. HTML, as Dan says, is a display format.
(this distinction exists even for server-side clients, e.g. web clients)
The format for server-side processing and storage should be the input format
When you say “input” here you mean “what the client sends to the server”. When DanArmak is talking about input, he is talking about the user experience, ease of writing and editing. These are obviously not the same thing.
If all users input in the same format, then it should be the storage format too. Rendering to HTML can be done when it’s actually needed. (Plus or minus caching/prerendering for performance.)
If users can choose to input in different formats, and we can’t convert between these formats (e.g. from HTML to markdown), then I think it would be easiest to just store whatever the user originally input. The main reason for original-format storage is editing, and users normally edit only their own content, so they shouldn’t mind the format it’s in.
If I write in markdown, but my editor has to send HTML to the server, then it has to implement an HTML-to-markdown conversion for editing, which raises all kinds of issues (like supporting the HTML output of an old version of the editor, never mind of different editors) and trying to solve them just doesn’t seem worth the bother. What does adopting HTML as a storage format get you?
I assumed you didn’t want people to raise technical objections to this post, and let you present your argument first. But if you want them now, here are some objections that gjm didn’t mention:
Our goal is to make something better than the existing LW software / UX. But we must also allow free linking in, to posts and comments, from ordinary blogs and other sites. These links will be the natural gateway for users new to the community, and lurkers. They must also have a UX at least as good as today’s LW, exposing the features of the new / non-web solution, or else this whole endeavor will be a regression and doomed to fail.
People won’t like a plaintext-oriented interface; they want rich text, inline images and tables, which in practice (in the world of NNTP) translates to HTML. But HTML is far too rich (and unsuitable for human editing). We need an equivalent to comments’ markdown support (or something better that would also be usable for posts). With NNTP, this would be client dependent, so at best it would vary by poster and at worst it would simply not be supported (or not out of the box).
Editing of posts and comments is a crucial feature which NNTP doesn’t provide.
Other features we need or are used to: RSS feed; tagging; user management and direct messaging without the trivial inconvenience of creating a new pseudonymous email account on a different site; server-based state (e.g. ‘unread’ message state in user inbox) for those with multiple client devices; …..
Any proposal that isn’t for a gradual change to the existing site will need to be run on a new site. The old lesswrong.com won’t be shut down until the new one has clearly succeeded. So you’ll need to directly compete with lesswrong.com to convince users to switch. You can make an LW → NNTP gateway, but not an NNTP → LW gateway; that is, lesswrong.com can’t automatically publish content (comments) posted via NNTP (or any other protocol, really). So during this period of competition, even if users cross-post, discussion threads will be separate. The new software will have to be clearly superior to convince LW users to switch, let alone diaspora blog authors.
I didn’t, but I was assuming people would anyway. I was actually hoping for higher-level objections like the problems I listed in the post, but, well. I’ll answer you anyway and maybe edit the post later. Most of these fall under 3.2 and 3.3 in the outline.
is actually the only part of the problem for which no off-the-shelf solution exists. The short version is that the site UX, instead of talking to a database directly on the backend, would talk to NNTP on the backend. Links to arbitrary posts (as a non-exhaustive example) could be as simple as https://newlesswrong/message-id. Top level posts could support some subject-generated shorthand, perhaps.
I actually do want a plaintext-oriented interface, but I know I’m in the minority. You’ve expressed the solution to the markup problem yourself, though: Markdown is already the effectively-default format on Usenet and all extant NNTP clients. In fact it’s the rise of lightweight markup in general and Markdown in particular that convinced me this could ever be more than a pipe dream. There are more powerful forms of lightweight markup that could be used; the key is that the input format must be readable as plaintext for interoperability between the web and native clients to be possible.
I believe that cancels and supersedes can be kludged to support something that looks like editing even if it isn’t. I may be wrong about this; in particular I’m not sure how supersedes interact with reply chains, because they’re unusable for that purpose on Usenet proper.
User management has an established solution, and DMing can be implemented as a LW mailbox (that only takes local messages) or a forwarding address, or both. RSS is probably easy. Server based state is easy for the ‘default’ website (really an in-browser client, covered in 1.1 and 1.7) but may be hard for native clients (but anyone using a native client presumably knows what they’re getting into). Tagging may be hard, I’m not sure. Karma is definitely hard, but may be unnecessary.
The short version of this is that, designed correctly, any author or site can adopt the network without being any worse off than they currently are; that is, cooperate-defect leaves you no worse off than defect-defect. The long version is...pretty much all of section 2, actually.
(ETA: I am answering this in moderate detail not to encourage technical back-and-forth but to demonstrate that I have thought this through)
I think this reduces to problem 5. Namely, you can’t replace lesswrong.com with a completely new NNTP-backed site as your opening move.
Ordinary Markdown is insufficient because it doesn’t do inline images and tables, which are sometimes important for posts. There are more powerful alternatives, and some markdown supersets. But I’ll bet they’re not supported by NNTP clients. At best you’ll find one client (or client plugin) that supports what you want. But forcing everyone to use the same client negates the point of NNTP, and isn’t plausible anyway because we need clients for different platforms including mobile.
This definitely requires proof. In addition to the issue with reply threads, NNTP clients expect to be able to cache articles. Propagation of supersede/replace is at best delayed and at worst out-of-order, and will vary between clients. This is IMO a critical issue, a major LW feature.
I’m surprised that you say “anyone using a native client presumably knows what they’re getting into”. You’re proposing NNTP as a superior solution, but also saying anyone who actually chooses to use NNTP will have a hard time and/or fewer features. And I think this applies to other, more important features and not just server based state.
Adoption, for site owners, will at minimum require a significant time/money investment, and at least trivial inconveniences to web-based readers due to unavoidable minor undesirable UX differences. So moving to the new platform needs to make people better off, not just no worse than before. The same applies to posters/readers: they won’t switch over without a compelling reason, nor should we expect them to.
I completely forgot about karma, but it’s so important that I’m promoting it to a new item. I think karma is pretty good, and some alternatives may be better, but nothing at all is much worse.
Meta: replying in a list-based format is inconvenient and I tentatively suggest making a separate reply for each significant list item.
Agreed that it’s inconvenient. Rather than separate it I’ll cut it down. 2, 4, and 6 all collapse to “client feature sets differ”; this is a meta-feature, not a meta-bug. The downside of client control is that not everybody sees the same thing. The upside of client control is client competition, which has similar benefits to market competition.
Solving adoption is the point of section 2 and is too long to describe here. Note that, as mentioned, I do not actually expect this to be adopted. The world isn’t that kind.
3 is a legitimate problem and will be addressed, but it’s the sort of thing where I need to spin up INN and see how it actually behaves when presented with edit-style supersedes. If there is a weak point in my “this is possible” argument, this is it.
That misses my point. Some features, like voting, can’t be implemented as clientside features, because clients would need to communicate about them (and establish consensus).
Contemporary HTML. On the other hand the original HTML (of the early 90s vintage) is a simple page description language designed to be hand-coded.
A well-defined limited subset of HTML would probably be easier to implement than some superset of markdown.
A subset of HTML is still unsuited to human editing. Even more so than full HTML, because it doesn’t have the justification of being a complex and extendable syntax. A superset of markdown would be much more usable, for people writing plaintext, than a subset of HTML. Especially if, as now, the majority of posts and comments require more or less only regular markdown and no superset features like tables.
Asciidoc might be an alternative when more power is needed. I haven’t used it, but ESR once said that it does markdown’s job better than Markdown itself.
HTML is wholly unsuited to human editing or even reading. I blame it for ruining email. Well, that and top-posting.
People from the 90s would disagree and rich text editors can output whatever. Of course markdown is better for specifying minor formatting while you write the content—that’s what it was explicitly designed for. However the advantage of HMTL is ubiquity.
The majority of posts and comments use only links, bold/italics, and an occasional bulleted list. Inline images are culturally disapproved of and tables are rare. At this level pretty much anything would work.
I don’t get your point here. HTML’s ubiquity is important for display, not for editing. Markdown is converted to HTML for display. As a user I prefer writing in markdown to writing in HTML. Don’t you?
That doesn’t mean anything would be equally as comfortable as anything else.
We are talking about the acceptable format for messages as they are processed and stored by the system, right? Ease of input is a separate issue and your editor can and should allow you to write in whatever way you find most comfortable.
The format for server-side processing and storage should be the input format unless there is specific cause not to use it (3.2). Conversion to display formats should be done client-side and as late as possible. HTML, as Dan says, is a display format.
(this distinction exists even for server-side clients, e.g. web clients)
When you say “input” here you mean “what the client sends to the server”. When DanArmak is talking about input, he is talking about the user experience, ease of writing and editing. These are obviously not the same thing.
It is, now. When designed, it wasn’t.
If all users input in the same format, then it should be the storage format too. Rendering to HTML can be done when it’s actually needed. (Plus or minus caching/prerendering for performance.)
If users can choose to input in different formats, and we can’t convert between these formats (e.g. from HTML to markdown), then I think it would be easiest to just store whatever the user originally input. The main reason for original-format storage is editing, and users normally edit only their own content, so they shouldn’t mind the format it’s in.
If I write in markdown, but my editor has to send HTML to the server, then it has to implement an HTML-to-markdown conversion for editing, which raises all kinds of issues (like supporting the HTML output of an old version of the editor, never mind of different editors) and trying to solve them just doesn’t seem worth the bother. What does adopting HTML as a storage format get you?