In many research areas, ideas are common, and it isn’t clear which ideas are most important. The most useful contributions come from someone taking an idea and demonstrating that it is viable and important, which often requires a lot of solitary work that can’t be done in the typical amount of time it takes to write a comment or post.
Agreed. My recommendations aren’t meant to be universally applicable. (ETA: Also, one could marginally increase one’s forum participation in order to capture some of the benefits, and not necessarily go all the way to adopting it as one’s primary research strategy.)
FP often leads to long, winding discussions that may end with two researchers agreeing, but the resulting transcript is not great for future readers.
There’s nothing that explicitly prevents people from distilling such discussions into subsequent posts or papers. If people aren’t doing that, or are doing that less than they should, that could potentially be solved as a problem that’s separate from “should more people be doing FP or traditional research?”
Also, it’s not clear to me that traditional research produces more clear distillations of how disagreements get resolved. It seems like most such discussions don’t make for publishable papers and therefore most disagreements between “traditional researchers” just don’t get resolved in a way that leaves a public record (or at all).
There’s nothing that explicitly prevents people from distilling such discussions into subsequent posts or papers. If people aren’t doing that, or are doing that less than they should, that could potentially be solved as a problem that’s separate from “should more people be doing FP or traditional research?”
FYI this is something the LW team thinks about a bunch and I expect us to have made some serious effort towards incentivizing and simplifying this process in the coming year.
This is actually a major motivation for the wiki/tagging system we are building. Also, you might have noticed all the edited transcripts we’ve been publishing, and the debates we’ve started organizing, which is also part of this. I‘be experimented a lot over the last year with UI for potentially directly distilling comment threads, but all of them ended up too clunky and messy to ever make me excited about them, though I still have some things I might want to give a shot, but overall I am currently thinking of tackling this problem in a slightly more indirect way.
Hm, I percieved Raemon to be referring more specifically to turning forum discussions into posts, or otherwise tidying them up. I think that’s importantly different to transcribing a talk (since a talk isn’t a discussion), or a debate (since you only have a short period of time to think about your response to the other person). I guess it’s possible that the tagging system helps with this, but it’s not obvious to me how it would. That being said, I do agree that more broadly LW has moved towards more synthesis and intertemporal discussions.
I’d add “The LessWrong 2018 Review” to the list of things that are “sort of exploring the same direction”. I agree my particular prediction about mechanical tools for distilling comments didn’t materialize, but we did definitely allocate tons of effort towards distillation as a whole.
Yeah, and I experimented a bunch with that (directly turning forum discussions into posts) and mostly felt like it didn’t really work that well. I mostly updated that there needs to be a larger synthesis step, though I still have some guesses for more direct things that could work. Ben spent some hours distilling the discussion and comments on a bunch of posts, which we should get around to posting (I just realized we never published them).
Re tagging: In general the tagging system that we are building has a lot in common with being a wiki (collaboratively editable descriptions, providing canonical definitions and references, and providing good summaries of existing content), and I expect it to grow into being more of a wiki over time (the tagging use-case was a specific narrow use-case that seemed easy to get traction on, but the mid-term goal is to do a lot more wiki-like stuff). And I think from that perspective it’s more clear how it helps with distillation.
There’s nothing that explicitly prevents people from distilling such discussions into subsequent posts or papers. If people aren’t doing that, or are doing that less than they should, that could potentially be solved as a problem that’s separate from “should more people be doing FP or traditional research?”
Doing these types of summarize feels like a good place to start out if you are new to doing FP. It is a fairly straight-forward task, but provides a lot of value, and helps you grow skills and reputation that will help you when you do more independent work later.
It might be useful for more experienced researchers/posters to explicitly point out when they are leaving this kind of value on the table. (“This was an interesting conversation, it contains a few valuable insights, and if I didn’t have more pressing things to work on, I would have liked to distill it to make it more clear. If someone feels like doing that, I will happily comment on the draft and signal boost the post.”)
There’s nothing that explicitly prevents people from distilling such discussions into subsequent posts or papers. If people aren’t doing that, or are doing that less than they should, that could potentially be solved as a problem that’s separate from “should more people be doing FP or traditional research?”
Agreed. I’m mostly saying that empirically people don’t do that, but yes there could be other solutions to the problem, it need not be inherent to FP.
Also, it’s not clear to me that traditional research produces more clear distillations of how disagreements get resolved.
I agree you don’t see how the disagreement gets resolved, but you usually can see the answer to the question that prompted the disagreement, because the resolution itself can be turned into a paper. This is assuming that the resolution came via new evidence. I agree that if a disagreement is resolved via simply talking through the arguments, then it doesn’t turn into a paper, but this seems pretty rare (at least in CS).
Agreed. My recommendations aren’t meant to be universally applicable. (ETA: Also, one could marginally increase one’s forum participation in order to capture some of the benefits, and not necessarily go all the way to adopting it as one’s primary research strategy.)
There’s nothing that explicitly prevents people from distilling such discussions into subsequent posts or papers. If people aren’t doing that, or are doing that less than they should, that could potentially be solved as a problem that’s separate from “should more people be doing FP or traditional research?”
Also, it’s not clear to me that traditional research produces more clear distillations of how disagreements get resolved. It seems like most such discussions don’t make for publishable papers and therefore most disagreements between “traditional researchers” just don’t get resolved in a way that leaves a public record (or at all).
FYI this is something the LW team thinks about a bunch and I expect us to have made some serious effort towards incentivizing and simplifying this process in the coming year.
Just got a calendar reminder to check if this happened—my impression is that any such efforts haven’t really materialised on the site.
This is actually a major motivation for the wiki/tagging system we are building. Also, you might have noticed all the edited transcripts we’ve been publishing, and the debates we’ve started organizing, which is also part of this. I‘be experimented a lot over the last year with UI for potentially directly distilling comment threads, but all of them ended up too clunky and messy to ever make me excited about them, though I still have some things I might want to give a shot, but overall I am currently thinking of tackling this problem in a slightly more indirect way.
Hm, I percieved Raemon to be referring more specifically to turning forum discussions into posts, or otherwise tidying them up. I think that’s importantly different to transcribing a talk (since a talk isn’t a discussion), or a debate (since you only have a short period of time to think about your response to the other person). I guess it’s possible that the tagging system helps with this, but it’s not obvious to me how it would. That being said, I do agree that more broadly LW has moved towards more synthesis and intertemporal discussions.
I’d add “The LessWrong 2018 Review” to the list of things that are “sort of exploring the same direction”. I agree my particular prediction about mechanical tools for distilling comments didn’t materialize, but we did definitely allocate tons of effort towards distillation as a whole.
Yeah, and I experimented a bunch with that (directly turning forum discussions into posts) and mostly felt like it didn’t really work that well. I mostly updated that there needs to be a larger synthesis step, though I still have some guesses for more direct things that could work. Ben spent some hours distilling the discussion and comments on a bunch of posts, which we should get around to posting (I just realized we never published them).
Re tagging: In general the tagging system that we are building has a lot in common with being a wiki (collaboratively editable descriptions, providing canonical definitions and references, and providing good summaries of existing content), and I expect it to grow into being more of a wiki over time (the tagging use-case was a specific narrow use-case that seemed easy to get traction on, but the mid-term goal is to do a lot more wiki-like stuff). And I think from that perspective it’s more clear how it helps with distillation.
I went and published such a distillation (which attempts to summarise the post What Failure Looks Like and distill its comments).
Doing these types of summarize feels like a good place to start out if you are new to doing FP. It is a fairly straight-forward task, but provides a lot of value, and helps you grow skills and reputation that will help you when you do more independent work later.
It might be useful for more experienced researchers/posters to explicitly point out when they are leaving this kind of value on the table. (“This was an interesting conversation, it contains a few valuable insights, and if I didn’t have more pressing things to work on, I would have liked to distill it to make it more clear. If someone feels like doing that, I will happily comment on the draft and signal boost the post.”)
Agreed. I’m mostly saying that empirically people don’t do that, but yes there could be other solutions to the problem, it need not be inherent to FP.
I agree you don’t see how the disagreement gets resolved, but you usually can see the answer to the question that prompted the disagreement, because the resolution itself can be turned into a paper. This is assuming that the resolution came via new evidence. I agree that if a disagreement is resolved via simply talking through the arguments, then it doesn’t turn into a paper, but this seems pretty rare (at least in CS).