One fairly strong belief of mine is that Less Wrong’s epistemic standards are not high enough to make solid intellectual progress here. So far my best effort to make that argument has been in the comment thread starting here. Looking back at that thread, I just noticed that a couple of those comments have been downvoted to negative karma. I don’t think any of my comments have ever hit negative karma before; I find it particularly sad that the one time it happens is when I’m trying to explain why I think this community is failing at its key goal of cultivating better epistemics.
There’s all sorts of arguments to be made here, which I don’t have time to lay out in detail. But just step back for a moment. Tens or hundreds of thousands of academics are trying to figure out how the world works, spending their careers putting immense effort into reading and producing and reviewing papers. Even then, there’s a massive replication crisis. And we’re trying to produce reliable answers to much harder questions by, what, writing better blog posts, and hoping that a few of the best ideas stick? This is not what a desperate effort to find the truth looks like.
And we’re trying to produce reliable answers to much harder questions by, what, writing better blog posts, and hoping that a few of the best ideas stick? This is not what a desperate effort to find the truth looks like.
It seems to me that maybe this is what a certain stage in the desperate effort to find the truth looks like?
Like, the early stages of intellectual progress look a lot like thinking about different ideas and seeing which ones stand up robustly to scrutiny. Then the best ones can be tested more rigorously and their edges refined through experimentation.
It seems to me like there needs to be some point in the desparate search for truth in which you’re allowing for half-formed thoughts and unrefined hypotheses, or else you simply never get to a place where the hypotheses you’re creating even brush up against the truth.
In the half-formed thoughts stage, I’d expect to see a lot of literature reviews, agendas laying out problems, and attempts to identify and question fundamental assumptions. I expect that (not blog-post-sized speculation) to be the hard part of the early stages of intellectual progress, and I don’t see it right now.
Perhaps we can split this into technical AI safety and everything else. Above I’m mostly speaking about “everything else” that Less Wrong wants to solve. Since AI safety is now a substantial enough field that its problems need to be solved in more systemic ways.
In the half-formed thoughts stage, I’d expect to see a lot of literature reviews, agendas laying out problems, and attempts to identify and question fundamental assumptions. I expect that (not blog-post-sized speculation) to be the hard part of the early stages of intellectual progress, and I don’t see it right now.
I would expect that later in the process. Agendas laying out problems and fundamental assumptions don’t spring from nowhere (at least for me), they come from conversations where I’m trying to articulate some intuition, and I recognize some underlying pattern. The pattern and structure doesn’t emerge spontaneously, it comes from trying to pick around the edges of a thing, get thoughts across, explain my intuitions and see where they break.
I think it’s fair to say that crystallizing these patterns into a formal theory is a “hard part”, but the foundation for making it easy is laid out in the floundering and flailing that came before.
Ironically, some people already feel threatened by the high standards here. Setting them higher probably wouldn’t result in more good content. It would result in less mediocre content, but probably also less good content, as the authors who sometimes write a mediocre article and sometimes a good one, would get discouraged and give up.
Ben Pace gives a few examples of great content in the next comment. It would be better to easier separate the good content from the rest, but that’s what the reviews are for. Well, only one review so far, if I remember correctly. I would love to see reviews of pre-2018 content (maybe multiple years in one review, if they were less productive). Then I would love to see the winning content get the same treatment as the Sequences—edit them and arrange them into a book, and make it “required reading” for the community (available as a free PDF).
The top posts in the 2018 Review are filled with fascinating and well-explained ideas. Many of the new ideas are not settled science, but they’re quite original and substantive, or excellent distillations of settled science, and are often the best piece of writing on the internet about their topics.
You’re wrong about LW epistemic standards not being high enough to make solid intellectual progress, we already have. On AI alone (which I am using in large part because there’s vaguely more consensus around it than around rationality), I think you wouldn’t have seen almost any of the public write-ups (like Embedded Agency and Zhukeepa’s Paul FAQ) without LessWrong, and I think a lot of them are brilliant.
I’m not saying we can’t do far better, or that we’re sufficiently good. Many of the examples of success so far are “Things that were in people’s heads but didn’t have a natural audience to share them with”. There’s not a lot of collaboration at present, which is why I’m very keen to build the new LessWrong Docs that allows for better draft sharing and inline comments and more. We’re working on the tools for editing tags, things like edit histories and so on, that will allow us to build a functioning wiki system to have canonical writeups and explanation that people add to and refine. I want future iterations of the LW Review to have more allowance for incorporating feedback from reviewers. There’s lots of work to do, and we’re just getting started. But I disagree the direction isn’t “a desperate effort to find the truth”. That’s what I’m here for.
Even in the last month or two, how do you look at things like this and this and this and this and not think that they’re likely the best publicly available pieces of writing in the world about their subjects? Wrt rationality, I expect things like this and this and this and this will probably go down as historically important LW posts that helped us understand the world, and make a strong showing in the 2020 LW Review.
As mentioned in my reply to Ruby, this is not a critique of the LW team, but of the LW mentality. And I should have phrased my point more carefully—“epistemic standards are too low to make any progress” is clearly too strong a claim, it’s more like “epistemic standards are low enough that they’re an important bottleneck to progress”. But I do think there’s a substantive disagreement here. Perhaps the best way to spell it out is to look at the posts you linked and see why I’m less excited about them than you are.
Of the top posts in the 2018 review, and the ones you linked (excluding AI), I’d categorise them as follows:
Interesting speculation about psychology and society, where I have no way of knowing if it’s true:
Local Validity as a Key to Sanity and Civilization
The Loudest Alarm Is Probably False
Anti-social punishment (which is, unlike the others, at least based on one (1) study).
Babble
Intelligent social web
Unrolling social metacognition
Simulacra levels
Can you keep this secret?
Same as above but it’s by Scott so it’s a bit more rigorous and much more compelling:
Is Science Slowing Down?
The tails coming apart as a metaphor for life
Useful rationality content:
Toolbox-thinking and law-thinking
A sketch of good communication
Varieties of argumentative experience
Review of basic content from other fields. This seems useful for informing people on LW, but not actually indicative of intellectual progress unless we can build on them to write similar posts on things that *aren’t* basic content in other fields:
Voting theory primer
Prediction markets: when do they work
Costly coordination mechanism of common knowledge (Note: I originally said I hadn’t seen many examples of people building on these ideas, but at least for this post there seems to be a lot.)
Six economics misconceptions
Swiss political system
It’s pretty striking to me how much the original sequences drew on the best academic knowledge, and how little most of the things above draw on the best academic knowledge. And there’s nothing even close to the thoroughness of Luke’s literature reviews.
The three things I’d like to see more of are:
1. The move of saying “Ah, this is interesting speculation about a complex topic. It seems compelling, but I don’t have good ways of verifying it; I’ll treat it like a plausible hypothesis which could be explored more by further work.” (I interpret the thread I originally linked as me urging Wei to do this).
2. Actually doing that follow-up work. If it’s an empirical hypothesis, investigating empirically. If it’s a psychological hypothesis, does it apply to anyone who’s not you? If it’s more of a philosophical hypothesis, can you identify the underlying assumptions and the ways it might be wrong? In all cases, how does it fit into existing thought? (That’ll probably take much more than a single blog post).
3. Insofar as many of these scattered plausible insights are actually related in deep ways, trying to combine them so that the next generation of LW readers doesn’t have to separately learn about each of them, but can rather download a unified generative framework.
(Thanks for laying out your position in this level of depth. Sorry for how long this comment turned out. I guess I wanted to back up a bunch of my agreement with words. It’s a comment for the sake of everyone else, not just you.)
I think there’s something to what you’re saying, that the mentality itself could be better. The Sequences have been criticized because Eliezer didn’t cite previous thinkers all that much, but at least as far as the science goes, as you said, he was drawing on academic knowledge. I also think we’ve lost something precious with the absence of epic topic reviews by the likes of Luke. Kaj Sotala still brings in heavily from outside knowledge, John Wentworth did a great review on Biological Circuits, and we get SSC crossposts that have that, but otherwise posts aren’t heavily referencing or building upon outside stuff. I concede that I would like to see a lot more of that.
I think Kaj was rightly disappointed that he didn’t get more engagement with his post whose gist was “this is what the science really says about S1 & S2, one of your most cherished concepts, LW community”.
I wouldn’t say the typical approach is strictly bad, there’s value in thinking freshly for oneself or that failure to reference previous material shouldn’t be a crime or makes a text unworthy, but yeah, it’d be pretty cool if after Alkjash laid out Babble & Prune (which intuitively feels so correct), someone had dug through what empirical science we have to see whether the picture lines up. Or heck, actually gone and done some kind of experiment. I bet it would turn up something interesting.
And I think what you’re saying is that the issue isn’t just that people aren’t following up with scholarship and empiricism on new ideas and models, but that they’re actually forgetting that these are the next steps. Instead, they’re overconfident in our homegrown models, as though LessWrong were the one place able to come up with good ideas. (Sorry, some of this might be my own words.)
The category I’d label a lot of LessWrong posts with is “engaging articulation of a point which is intuitive in hindsight” / “creation of common vocabulary around such points”. That’s pretty valuable, but I do think solving the hardest problems will take more.
-----
You use the word “reliably” in a few places. It feels like it’s doing some work in your statements, and I’m not entirely sure what you mean or why it’s important.
-----
A model which is interesting but maybe not of obvious connection. I was speaking to a respected rationalist thinker this week and they classified potential writing on LessWrong into three categories:
Writing stuff to help oneself figure things out. Like a diary, but publicly shared.
People exchanging “letters” as they attempt to figure things out. Like old school academic journals.
Someone having something mostly figured out but with a large inferential distance to bridge. They write a large collection of posts trying to cover that distance. One example is The Sequences, and more recent examples are from John Wentworth and Kaj Sotala
I mention this because I recall you (alongside the rationalist thinker) complaining about the lack of people “presenting their worldviews on LessWrong”.
The kinds of epistemic norms I think you’re advocating for feel like a natural fit for 2nd kind of writing, but it’s less clear to me how they should apply to people presenting world views. Maybe it’s not more complicated than it’s fine to present your worldview without a tonne of evidence, but people shouldn’t forget that the evidence hasn’t been presented and it feeling intuitively correct isn’t enough.
-----
There’s something in here about Epistemic Modesty, something, something. Some part of me reads you as calling for more of that, which I’m wary of, but I don’t currently have more to say than flagging it as maybe a relevant variable in any disagreements here.
We probably do disagree about the value of academic sources, or what it takes to get value from them. Hmm. Maybe it’s something like there’s something to be said for thinking about models and assessing their plausibility yourself rather than relying on likely very flawed empirical studies.
Maybe I’m in favor of large careful reviews of what science knows but less in favor of trying to find sources for each idea or model that gets raised. I’m not sure.
-----
I can’t recall whether I’ve written publicly much about this, but a model I’ve had for a year or more is that for LW to make intellectual progress, we need to become a “community of practice”, not just a “community of interest”. Martial arts vs literal stamp collecting. (Streetfighting might be better still due to actual testing real fighting ability.) It’s great that many people find LessWrong a guilty pleasure they feel less guilty about than Facebook, but for us to make progress, people need to see LessWrong as a place where one of things you do is show up and do Serious Work, some of which is relatively hard and boring, like writing and reading lit reviews.
I suspect that a cap on the epistemic standards people hold stuff to is downstream of the level of effort people are calibrated on applying. But maybe it goes in other direction, so I don’t know.
Probably the 2018 Review is biased towards the posts which are most widely read, i.e., those easiest and most enjoyable to read, rather than solely rewarding those with the best contributions. Not overwhelmingly, but enough. Maybe same for karma. I’m not sure how to relate to that.
-----
3. Insofar as many of these scattered plausible insights are actually related in deep ways, trying to combine them so that the next generation of LW readers doesn’t have to separately learn about each of them, but can rather download a unified generative framework.
This sounds partially like distillation work plus extra integration. And sounds pretty good to me too.
-----
I still remember my feeling of disillusionment in the LessWrong community relative soon after I joined in late 2012. I realized that the bulk of members didn’t seem serious about advancing the Art. I never heard people discussing new results from cognitive science and how to apply them, even though that’s what Sequences were in large part and the Sequences hardly claimed to be complete! I guess I do relate somewhat to your “desperate effort” comment, though we’ve got some people trying pretty hard that I wouldn’t want to short change.
We do good stuff, but more is possible. I appreciate the reminder. I hope we succeed at pushing the culture and mentality in directions you like.
This is only tangentially relevant, but adding it here as some of you might find it interesting:
Venkatesh Rao has an excellent Twitter thread on why most independent research only reaches this kind of initial exploratory level (he tried it for a bit before moving to consulting). It’s pretty pessimistic, but there is a somewhat more optimistic follow-up thread on potential new funding models. Key point is that the later stages are just really effortful and time-consuming, in a way that keeps out a lot of people trying to do this as a side project alongside a separate main job (which I think is the case for a lot of LW contributors?)
Quote from that thread:
Research =
a) long time between having an idea and having something to show for it that even the most sympathetic fellow crackpot would appreciate (not even pay for, just get)
b) a >10:1 ratio of background invisible thinking in notes, dead-ends, eliminating options etc
With a blogpost, it’s like a week of effort at most from idea to mvp, and at most a 3:1 ratio of invisible to visible. That’s sustainable as a hobby/side thing.
To do research-grade thinking you basically have to be independently wealthy and accept 90% deadweight losses
Also just wanted to say good luck! I’m a relative outsider here with pretty different interests to LW core topics but I do appreciate people trying to do serious work outside academia, have been trying to do this myself, and have thought a fair bit about what’s currently missing (I wrote that in a kind of jokey style but I’m serious about the topic).
Thanks, these links seem great! I think this is a good (if slightly harsh) way of making a similar point to mine:
“I find that autodidacts who haven’t experienced institutional R&D environments have a self-congratulatory low threshold for what they count as research. It’s a bit like vanity publishing or fan fiction. This mismatch doesn’t exist as much in indie art, consulting, game dev etc”
Also, I liked your blog post! More generally, I strongly encourage bloggers to have a “best of” page, or something that directs people to good posts. I’d be keen to read more of your posts but have no idea where to start.
Thanks! I have been meaning to add a ‘start here’ page for a while, so that’s good to have the extra push :) Seems particularly worthwhile in my case because a) there’s no one clear theme and b) I’ve been trying a lot of low-quality experimental posts this year bc pandemic trashed motivation, so recent posts are not really reflective of my normal output.
For now some of my better posts in the last couple of years might be Cognitive decoupling and banana phones (tracing back the original precursor of Stanovich’s idea), The middle distance (a writeup of a useful and somewhat obscure idea from Brian Cantwell Smith’s On the Origin of Objects), and the negative probability post and its followup.
Quoting your reply to Ruby below, I agree I’d like LessWrong to be much better at “being able to reliably produce and build on good ideas”.
The reliability and focus feels most lacking to me on the building side, rather than the production, which I think we’re doing quite well at. I think we’ve successfully formed a publishing platform that provides and audience who are intensely interested in good ideas around rationality, AI, and related subjects, and a lot of very generative and thoughtful people are writing down their ideas here.
We’re low on the ability to connect people up to do more extensive work on these ideas – most good hypotheses and arguments don’t get a great deal of follow up or further discussion.
Here are some subjects where I think there’s been various people sharing substantive perspectives, but I think there’s also a lot of space for more ‘details’ to get fleshed out and subquestions to be cleanly answered:
The above isn’t complete, it’s just some of the ones that come to mind as having lots of people sharing perspectives. And the list of people definitely isn’t complete.
Here examples of things that I’d like to see more of, that feel more like doing the legwork to actually dive into the details:
Eli Tyre and Bucky replicating Scott’s birth-order hypothesis
Katja and the other fine people at AI Impacts doing long-term research on a question (discontinuous progress) with lots of historical datapoints
Jameson writing up his whole research question in great detail and very well, and then an excellent commenter turning up and answering it
Zhukeepa writing up an explanation of Paul’s research, allowing many more to understand it, and allowing Eliezer to write a response
Scott writing Goodhart Taxonomy, and the commenters banding together to find a set of four similar examples to add to the post
Val writing some interesting things about insight meditation, prompting Kaj to write a non-mysterious explanation
In the LW Review when Bucky checked out the paper Zvi analysed and argued it did not support the conclusions Zvi reached (this changed my opinion of Zvi’s post from ‘true’ to ‘false’)
The discussion around covid and EMH prompting Richard Meadows to write down a lot of the crucial and core arguments around the EMH
The above is also not mentioning lots of times when the person generating the idea does a lot of the legwork, like Scott or Jameson or Sarah or someone.
I see a lot of (very high quality) raw energy here that wants shaping and directing, with the use of lots of tools for coordination (e.g. better collaboration tools).
The epistemic standards being low is one way of putting it, but it doesn’t resonate with me much and kinda feels misleading. I think our epistemic standards are way higher than the communities you mention (historians, people interested in progress studies). Bryan Caplan said he knows of no group whose beliefs are more likely to be right in general than the rationalists, this seems often accurate to me. I think we do a lot of exploration and generation and evaluation, just not in a very coordinated manner, and so could make progress at like 10x–100x the rate if we collaborated better, and I think we can get there without too much work.
“I see a lot of (very high quality) raw energy here that wants shaping and directing, with the use of lots of tools for coordination (e.g. better collaboration tools).”
Yepp, I agree with this. I guess our main disagreement is whether the “low epistemic standards” framing is a useful way to shape that energy. I think it is because it’ll push people towards realising how little evidence they actually have for many plausible-seeming hypotheses on this website. One proven claim is worth a dozen compelling hypotheses, but LW to a first approximation only produces the latter.
When you say “there’s also a lot of space for more ‘details’ to get fleshed out and subquestions to be cleanly answered”, I find myself expecting that this will involve people who believe the hypothesis continuing to build their castle in the sky, not analysis about why it might be wrong and why it’s not.
That being said, LW is very good at producing “fake frameworks”. So I don’t want to discourage this too much. I’m just arguing that this is a different thing from building robust knowledge about the world.
One proven claim is worth a dozen compelling hypotheses
I will continue to be contrary and say I’m not sure I agree with this.
For one, I think in many domains new ideas are really hard to come by, as opposed to making minor progress in the existing paradigms. Fundamental theories in physics, a bunch of general insights about intelligence (in neuroscience and AI), etc.
And secondly, I am reminded of what Lukeprog wrote in his moral consciousness report, that he wished the various different philosophies-of-consciousness would stop debating each other, go away for a few decades, then come back with falsifiable predictions. I sometimes take this stance regarding many disagreements of import, such as the basic science vs engineering approaches to AI alignment. It’s not obvious to me that the correct next move is for e.g. Eliezer and Paul to debate for 1000 hours, but instead to go away and work on their ideas for a decade then come back with lots of fleshed out details and results that can be more meaningfully debated.
I feel similarly about simulacra levels, Embedded Agency, and a bunch of IFS stuff. I would like to see more experimentation and literature reviews where they make sense, but I also feel like these are implicitly making substantive and interesting claims about the world, and I’d just be interested in getting a better sense of what claims they’re making, and have them fleshed out + operationalized more. That would be a lot of progress to me, and I think each of them is seeing that sort of work (with Zvi, Abram, and Kaj respectively leading the charges on LW, alongside many others).
I think I’m concretely worried that some of those models / paradigms (and some other ones on LW) don’t seem pointed in a direction that leads obviously to “make falsifiable predictions.”
And I can imagine worlds where “make falsifiable predictions” isn’t the right next step, you need to play around with it more and get it fleshed out in your head before you can do that. But there is at least some writing on LW that feels to me like it leaps from “come up with an interesting idea” to “try to persuade people it’s correct” without enough checking.
(In the case of IFS, I think Kaj’s sequence is doing a great job of laying it out in a concrete way where it can then be meaningfully disagreed with. But the other people who’ve been playing around with IFS didn’t really seem interested in that, and I feel like we got lucky that Kaj had the time and interest to do so.)
I feel like this comment isn’t critiquing a position I actually hold. For example, I don’t believe that “the correct next move is for e.g. Eliezer and Paul to debate for 1000 hours”. I am happy for people to work towards building evidence for their hypotheses in many ways, including fleshing out details, engaging with existing literature, experimentation, and operationalisation.
Perhaps this makes “proven claim” a misleading phrase to use. Perhaps more accurate to say: “one fully fleshed out theory is more valuable than a dozen intuitively compelling ideas”. But having said that, I doubt that it’s possible to fully flesh out a theory like simulacra levels without engaging with a bunch of academic literature and then making predictions.
Yepp, I agree with this. I guess our main disagreement is whether the “low epistemic standards” framing is a useful way to shape that energy. I think it is because it’ll push people towards realising how little evidence they actually have for many plausible-seeming hypotheses on this website.
A housemate of mine said to me they think LW has a lot of breadth, but could benefit from more depth.
I think in general when we do intellectual work we have excellent epistemic standards, capable of listening to all sorts of evidence that other communities and fields would throw out, and listening to subtler evidence than most scientists (“faster than science”), but that our level of coordination and depth is often low. “LessWrongers should collaborate more and go into more depth in fleshing out their ideas” sounds more true to me than “LessWrongers have very low epistemic standards”.
In general when we do intellectual work we have excellent epistemic standards, capable of listening to all sorts of evidence that other communities and fields would throw out, and listening to subtler evidence than most scientists (“faster than science”)
“Being more openminded about what evidence to listen to” seems like a way in which we have lower epistemic standards than scientists, and also that’s beneficial. It doesn’t rebut my claim that there are some ways in which we have lower epistemic standards than many academic communities, and that’s harmful.
In particular, the relevant question for me is: why doesn’t LW have more depth? Sure, more depth requires more work, but on the timeframe of several years, and hundreds or thousands of contributors, it seems viable. And I’m proposing, as a hypothesis, that LW doesn’t have enough depth because people don’t care enough about depth—they’re willing to accept ideas even before they’ve been explored in depth. If this explanation is correct, then it seems accurate to call it a problem with our epistemic standards—specifically, the standard of requiring (and rewarding) deep investigation and scholarship.
LW doesn’t have enough depth because people don’t care enough about depth—they’re willing to accept ideas even before they’ve been explored in depth. If this explanation is correct, then it seems accurate to call it a problem with our epistemic standards—specifically, the standard of requiring (and rewarding) deep investigation and scholarship.
Your solution to the “willingness to accept ideas even before they’ve been explored in depth” problem is to explore ideas in more depth. But another solution is to accept fewer ideas, or hold them much more provisionally.
I suspect trying to browbeat people to explore ideas in more depth works against the grain of an online forum as an institution. Browbeating works in academia because your career is at stake, but in an online forum, it just hurts intrinsic motivation and cuts down on forum use (the forum runs on what Clay Shirky called “cognitive surplus”, essentially a term for peoples’ spare time and motivation). I’d say one big problem with LW 1.0 that LW 2.0 had to solve before flourishing was people felt too browbeaten to post much of anything.
If we accept fewer ideas / hold them much more provisionally, but provide a clear path to having an idea be widely held as true, that creates an incentive for people to try & jump through hoops—and this incentive is a positive one, not a punishment-driven browbeating incentive.
Maybe part of the issue is that on LW, peer review generally happens in the comments after you publish, not before. So there’s no publication carrot to offer in exchange for overcoming the objections of peer reviewers.
“If we accept fewer ideas / hold them much more provisionally, but provide a clear path to having an idea be widely held as true, that creates an incentive for people to try & jump through hoops—and this incentive is a positive one, not a punishment-driven browbeating incentive.”
Hmm, it sounds like we agree on the solution but are emphasising different parts of it. For me, the question is: who’s this “we” that should accept fewer ideas? It’s the set of people who agree with my argument that you shouldn’t believe things which haven’t been fleshed out very much. But the easiest way to add people to that set is just to make the argument, which is what I’ve done. Specifically, note that I’m not criticising anyone for producing posts that are short and speculative: I’m criticising the people who update too much on those posts.
Fair enough. I’m reminded of a time someone summarized one of my posts as being a definitive argument against some idea X and me thinking to myself “even I don’t think my post definitively settles this issue” haha.
I do think right now LessWrong should lean more in the direction the Richard is suggesting – I think it was essential to establish better Babble procedures but now we’re doing well enough on that front that I think setting clearer expectations of how the eventual pruning works is reasonable.
I wanted to register that I don’t like “babble and prune” as a model of intellectual development. I think intellectual development actually looks more like:
1. Babble
2. Prune
3. Extensive scholarship
4. More pruning
5. Distilling scholarship to form common knowledge
And that my main criticism is the lack of 3 and 5, not the lack of 2 or 4.
I also note that: a) these steps get monotonically harder, so that focusing on the first two misses *almost all* the work; b) maybe I’m being too harsh on the babble and prune framework because it’s so thematically appropriate for me to dunk on it here; I’m not sure if your use of the terminology actually reveals a substantive disagreement.
I basically agree with your 5-step model (I at least agree it’s a more accurate description than Babel and Prune, which I just meant as rough shorthand). I’d add things like “original research/empiricism” or “more rigorous theorizing” to the “Extensive Scholarship” step.
I see the LW Review as basically the first of (what I agree should essentially be at least) a 5 step process. It’s adding a stronger Step 2, and a bit of Step 5 (at least some people chose to rewrite their posts to be clearer and respond to criticism)
...
Currently, we do get non-zero Extensive Scholarship and Original Empiricism. (Kaj’s Multi-Agent Models of Mind seems like it includes real scholarship. Scott Alexander / Eli Tyre and Bucky’s exploration into Birth Order Effects seemed like real empiricism). Not nearly as much as I’d like.
If the cost of evaluating a hypothesis is high, and hypotheses are cheap to generate, I would like to generate a great deal before selecting one to evaluate.
But, honestly… I’m not sure it’s actually a question that was worth asking. I’d like to know if Eliezer’s hypothesis about mathematicians is true, but I’m not sure it ranks near the top of questions I’d want people to put serious effort into answering.
I do want LessWrong to be able to followup Good Hypotheses with Actual Research, but it’s not obvious which questions are worth answering. OpenPhil et al are paying for some types of answers, I think usually by hiring researchers full time. It’s not quite clear what the right role for LW to play in the ecosystem.
All else equal, the harder something is, the less we should do it.
My quick take is that writing lit reviews/textbooks is a comparative disadvantage of LW relative to the mainstream academic establishment.
In terms of producing reliable knowledge… if people actually care about whether something is true, they can always offer a cash prize for the best counterargument (which could of course constitute citation of academic research). The fact that people aren’t doing this suggests to me that for most claims on LW, there isn’t any (reasonably rich) person who cares deeply re: whether the claim is true. I’m a little wary of putting a lot of effort into supply if there is an absence of demand.
(I guess the counterargument is that accurate knowledge is a public good so an individual’s willingness to pay doesn’t get you complete picture of the value accurate knowledge brings. Maybe what we need is a way to crowdfund bounties for the best argument related to something.)
(I agree that LW authors would ideally engage more with each other and academic literature on the margin.)
I’ve been thinking about the idea of “social rationality” lately, and this is related. We do so much here in the way of training individual rationality—the inputs, functions, and outputs of a single human mind. But if truth is a product, then getting human minds well-coordinated to produce it might be much more important than training them to be individually stronger. Just as assembly line production is much more effective in producing almost anything than teaching each worker to be faster in assembling a complete product by themselves.
My guess is that this could be effective not only in producing useful products, but also in overcoming biases. Imagine you took 5 separate LWers and asked them to create a unified consensus response to a given article. My guess is that they’d learn more through that collective effort, and produce a more useful response, than if they spent the same amount of time individually evaluating the article and posting their separate replies.
Of course, one of the reasons we don’t to that so much is that coordination is an up-front investment and is unfamiliar. Figuring out social technology to make it easier to participate in might be a great project for LW.
There’s been a fair amount of discussion of that sort of thing here: https://www.lesswrong.com/tag/group-rationality There are also groups outside LW thinking about social technology such as RadicalxChange.
Imagine you took 5 separate LWers and asked them to create a unified consensus response to a given article. My guess is that they’d learn more through that collective effort, and produce a more useful response, than if they spent the same amount of time individually evaluating the article and posting their separate replies.
I’m not sure. If you put those 5 LWers together, I think there’s a good chance that the highest status person speaks first and then the others anchor on what they say and then it effectively ends up being like a group project for school with the highest status person in charge. Somerelatedlinks.
That’s definitely a concern too! I imagine such groups forming among people who either already share a basic common view, and collaborate to investigate more deeply. That way, any status-anchoring effects are mitigated.
Alternatively, it could be an adversarial collaboration. For me personally, some of the SSC essays in this format have led me to change my mind in a lasting way.
they’re willing to accept ideas even before they’ve been explored in depth
People also reject ideas before they’ve been explored in depth. I’ve tried to discuss similar issues with LW before but the basic response was roughly “we like chaos where no one pays attention to whether an argument has ever been answered by anyone; we all just do our own thing with no attempt at comprehensiveness or organizing who does what; having organized leadership of any sort, or anyone who is responsible for anything, would be irrational” (plus some suggestions that I’m low social status and that therefore I personally deserve to be ignored. there were also suggestions – phrased rather differently but amounting to this – that LW will listen more if published ideas are rewritten, not to improve on any flaws, but so that the new versions can be published at LW before anywhere else, because the LW community’s attention allocation is highly biased towards that).
I feel somewhat inclined to wrap up this thread at some point, even while there’s more to say. We can continue if you like and have something specific or strong you’d like to ask, but otherwise will pause here.
You have to realise that what you are doing isn’t adequate in order to gain the motivation to do it better, and that is unlikely to happen if you are mostly communicating with other people who think everything is OK.
Lesswrong is competing against philosophy as well as science, and philosophy has broader criterion of evidence still. In fact , lesswrongians are often frustrated that mainstream philosophy takes such topics as dualism or theism seriously.. even though theres an abundance of Bayesian evidence for them.
One proven claim is worth a dozen compelling hypotheses, but LW to a first approximation only produces the latter.
Depends on the claim, right?
If the cost of evaluating a hypothesis is high, and hypotheses are cheap to generate, I would like to generate a great deal before selecting one to evaluate.
Right, but this isn’t mentioned in the post? Which seems odd. Maybe that’s actually another example of the “LW mentality”: why is the fact that there has been solid empirical research into 3 layers not being enough not important enough to mention in a post on why 3 layers isn’t enough? (Maybe because the post was time-boxed? If so that seems reasonable, but then I would hope that people comment saying “Here’s a very relevant paper, why didn’t you cite it?”)
Much of the same is true of scientific journals. Creating a place to share and publish research is a pretty key piece of intellectual infrastructure, especially for researchers to create artifacts of their thinking along the way.
The point about being ‘cross-posted’ is where I disagree the most.
This is largely original content that counterfactually wouldn’t have been published, or occasionally would have been published but to a much smaller audience. What Failure Looks Like wasn’t crossposted, Anna’s piece on reality-revealing puzzles wasn’t crossposted. I think that Zvi would have still written some on mazes and simulacra, but I imagine he writes substantially more content given the cross-posting available for the LW audience. Could perhaps check his blogging frequency over the last few years to see if that tracks. I recall Zhu telling me he wrote his FAQ because LW offered an audience for it, and likely wouldn’t have done so otherwise. I love everything Abram writes, and while he did have the Intelligent Agent Foundations Forum, it had a much more concise, technical style, tiny audience, and didn’t have the conversational explanations and stories and cartoons that have been so excellent and well received on LW, and it wouldn’t as much have been focused on the implications for rationality of things like logical inductors. Rohin wouldn’t have written his coherence theorems piece or any of his value learning sequence, and I’m pretty sure about that because I personally asked him to write that sequence, which is a great resource and I’ve seen other researchers in the field physically print off to write on and study. Kaj has an excellent series of non-mystical explanations of ideas from insight meditation that started as a response to things Val wrote, and I imagine those wouldn’t have been written quite like that if that context did not exist on LW.
I could keep going, but probably have made the point. It seems weird to not call this collectively a substantial amount of intellectual progress, on a lot of important questions.
I am indeed focusing right now on how to do more ‘conversation’. I’m in the middle of trying to host some public double cruxes for events, for example, and some day we will finally have inline commenting and better draft sharing and so on. It’s obviously not finished.
Rohin wouldn’t have written his coherence theorems piece or any of his value learning sequence, and I’m pretty sure about that because I personally asked him to write that sequence
Yeah, that’s true, though it might have happened at some later point in the future as I got increasingly frustrated by people continuing to cite VNM at me (though probably it would have been a blog post and not a full sequence).
Reading through this comment tree, I feel like there’s a distinction to be made between “LW / AIAF as a platform that aggregates readership and provides better incentives for blogging”, and “the intellectual progress caused by posts on LW / AIAF”. The former seems like a clear and large positive of LW / AIAF, which I think Richard would agree with. For the latter, I tend to agree with Richard, though perhaps not as strongly as he does. Maybe I’d put it as, I only really expect intellectual progress from a few people who work on problems full time who probably would have done similar-ish work if not for LW / AIAF (but likely would not have made it public).
I’d say this mostly for the AI posts. I do read the rationality posts and don’t get a different impression from them, but I also don’t think enough about them to be confident in my opinions there.
Thanks for chiming in with this. People criticizing the epistemics is hopefully how we get better epistemics. When the Californian smoke isn’t interfering with my cognition as much, I’ll try to give your feedback (and Rohin’s) proper attention. I would generally be interested to hear your arguments/models in detail, if you get the chance to lay them out.
My default position is LW has done well enough historically (e.g. Ben Pace’s examples) for me to currently be investing in getting it even better. Epistemics and progress could definitely be a lot better, but getting there is hard. If I didn’t see much progress on the rate of progress in the next year or two, I’d probably go focus on other things, though I think it’d be tragic if we ever lost what we have now.
And another thought:
And we’re trying to produce reliable answers to much harder questions by, what, writing better blog posts
Yes and no. Journal articles have their advantages, and so do blog posts. A bunch of recent LessWrong team’s work has been around filling in the missing pieces for the system to work, e.g. Open Questions (hasn’t yet worked for coordinating research), Annual Review, Tagging, Wiki. We often talk about conferences and “campus”. My work on Open Questions involved thinking about i) a better template for articles than “Abstract, Intro, Methods, etc.”, but Open Questions didn’t work for unrelated reasons we haven’t overcome yet, ii) getting lit reviews done systematically by people, iii) coordinating groups around research agendas.
I’ve thought about re-attempting the goals of Open Questions with instead a “Research Agenda” feature that lets people communally maintain research agendas and work on them. It’s a question of priorities whether I work on that anytime soon.
I do really think many of the deficiencies of LessWrong’s current work compared to academia are “infrastructure problems” at least as much as the epistemic standards of the community. Which means the LW team should be held culpable for not having solved them yet, but it is tricky.
For the record, I think the LW team is doing a great job. There’s definitely a sense in which better infrastructure can reduce the need for high epistemic standards, but it feels like the thing I’m pointing at is more like “Many LW contributors not even realising how far away we are from being able to reliably produce and build on good ideas” (which feels like my criticism of Ben’s position in his comment, so I’ll respond more directly there).
It seems really valuable to have you sharing how you think we’re falling epistemically short and probably important for the site to integrate the insights behind that view. There are a bunch of ways I disagree with your claims about epistemic best practices, but it seems like it would be cool if I could pass your ITT more. I wish your attempt to communicate the problems you saw had worked out better. I hope there’s a way for you to help improve LW epistemics, but also get that it might be costly in time and energy.
I just noticed that a couple of those comments have been downvoted to negative karma
Now they’re positive again.
Confusing to me, their Ω-karma (karma on another website) is also positive. Does it mean they previously had negative LW-karma but positive Ω-karma? Or that their Ω-karma also improved as a result of you complaining on LW a few hours ago? Why would it?
(Feature request: graph of evolution of comment karma as a function of time.)
I’d be curious what, if any, communities you think set good examples in this regard. In particular, are there specific academic subfields or non-academic scenes that exemplify the virtues you’d like to see more of?
Maybe historians of the industrial revolution? Who grapple with really complex phenomena and large-scale patterns, like us, but unlike us use a lot of data, write a lot of thorough papers and books, and then have a lot of ongoing debate on those ideas. And then the “progress studies” crowd is an example of an online community inspired by that tradition (but still very nascent, so we’ll see how it goes).
More generally I’d say we could learn to be more rigorous by looking at any scientific discipline or econ or analytic philosophy. I don’t think most LW posters are in a position to put in as much effort as full-time researchers, but certainly we can push a bit in that direction.
Thanks for your reply! I largely agree with drossbucket’s reply.
I also wonder how much this is an incentives problem. As you mentioned and in my experience, the fields you mentioned strongly incentivize an almost fanatical level of thoroughness that I suspect is very hard for individuals to maintain without outside incentives pushing them that way. At least personally, I definitely struggle and, frankly, mostly fail to live up to the sorts of standards you mention when writing blog posts in part because the incentive gradient feels like it pushes towards hitting the publish button.
Given this, I wonder if there’s a way to shift the incentives on the margin. One minor thing I’ve been thinking of trying for my personal writing is having a Knuth or Nintil style “pay for mistakes” policy. Do you have thoughts on other incentive structures to for rewarding rigor or punishing the lack thereof?
It feels partly like an incentives problem, but also I think a lot of people around here are altruistic and truth-seeking and just don’t realise that there are much more effective ways to contribute to community epistemics than standard blog posts.
I think that most LW discussion is at the level where “paying for mistakes” wouldn’t be that helpful, since a lot of it is fuzzy. Probably the thing we need first are more reference posts that distill a range of discussion into key concepts, and place that in the wider intellectual context. Then we can get more empirical. (Although I feel pretty biased on this point, because my own style of learning about things is very top-down). I guess to encourage this, we could add a “reference” section for posts that aim to distill ongoing debates on LW.
In some cases you can get a lot of “cheap” credit by taking other people’s ideas and writing a definitive version of them aimed at more mainstream audiences. For ideas that are really worth spreading, that seems useful.
One fairly strong belief of mine is that Less Wrong’s epistemic standards are not high enough to make solid intellectual progress here. So far my best effort to make that argument has been in the comment thread starting here. Looking back at that thread, I just noticed that a couple of those comments have been downvoted to negative karma. I don’t think any of my comments have ever hit negative karma before; I find it particularly sad that the one time it happens is when I’m trying to explain why I think this community is failing at its key goal of cultivating better epistemics.
There’s all sorts of arguments to be made here, which I don’t have time to lay out in detail. But just step back for a moment. Tens or hundreds of thousands of academics are trying to figure out how the world works, spending their careers putting immense effort into reading and producing and reviewing papers. Even then, there’s a massive replication crisis. And we’re trying to produce reliable answers to much harder questions by, what, writing better blog posts, and hoping that a few of the best ideas stick? This is not what a desperate effort to find the truth looks like.
It seems to me that maybe this is what a certain stage in the desperate effort to find the truth looks like?
Like, the early stages of intellectual progress look a lot like thinking about different ideas and seeing which ones stand up robustly to scrutiny. Then the best ones can be tested more rigorously and their edges refined through experimentation.
It seems to me like there needs to be some point in the desparate search for truth in which you’re allowing for half-formed thoughts and unrefined hypotheses, or else you simply never get to a place where the hypotheses you’re creating even brush up against the truth.
In the half-formed thoughts stage, I’d expect to see a lot of literature reviews, agendas laying out problems, and attempts to identify and question fundamental assumptions. I expect that (not blog-post-sized speculation) to be the hard part of the early stages of intellectual progress, and I don’t see it right now.
Perhaps we can split this into technical AI safety and everything else. Above I’m mostly speaking about “everything else” that Less Wrong wants to solve. Since AI safety is now a substantial enough field that its problems need to be solved in more systemic ways.
I would expect that later in the process. Agendas laying out problems and fundamental assumptions don’t spring from nowhere (at least for me), they come from conversations where I’m trying to articulate some intuition, and I recognize some underlying pattern. The pattern and structure doesn’t emerge spontaneously, it comes from trying to pick around the edges of a thing, get thoughts across, explain my intuitions and see where they break.
I think it’s fair to say that crystallizing these patterns into a formal theory is a “hard part”, but the foundation for making it easy is laid out in the floundering and flailing that came before.
[Deleted]
Ironically, some people already feel threatened by the high standards here. Setting them higher probably wouldn’t result in more good content. It would result in less mediocre content, but probably also less good content, as the authors who sometimes write a mediocre article and sometimes a good one, would get discouraged and give up.
Ben Pace gives a few examples of great content in the next comment. It would be better to easier separate the good content from the rest, but that’s what the reviews are for. Well, only one review so far, if I remember correctly. I would love to see reviews of pre-2018 content (maybe multiple years in one review, if they were less productive). Then I would love to see the winning content get the same treatment as the Sequences—edit them and arrange them into a book, and make it “required reading” for the community (available as a free PDF).
[Deleted]
The top posts in the 2018 Review are filled with fascinating and well-explained ideas. Many of the new ideas are not settled science, but they’re quite original and substantive, or excellent distillations of settled science, and are often the best piece of writing on the internet about their topics.
You’re wrong about LW epistemic standards not being high enough to make solid intellectual progress, we already have. On AI alone (which I am using in large part because there’s vaguely more consensus around it than around rationality), I think you wouldn’t have seen almost any of the public write-ups (like Embedded Agency and Zhukeepa’s Paul FAQ) without LessWrong, and I think a lot of them are brilliant.
I’m not saying we can’t do far better, or that we’re sufficiently good. Many of the examples of success so far are “Things that were in people’s heads but didn’t have a natural audience to share them with”. There’s not a lot of collaboration at present, which is why I’m very keen to build the new LessWrong Docs that allows for better draft sharing and inline comments and more. We’re working on the tools for editing tags, things like edit histories and so on, that will allow us to build a functioning wiki system to have canonical writeups and explanation that people add to and refine. I want future iterations of the LW Review to have more allowance for incorporating feedback from reviewers. There’s lots of work to do, and we’re just getting started. But I disagree the direction isn’t “a desperate effort to find the truth”. That’s what I’m here for.
Even in the last month or two, how do you look at things like this and this and this and this and not think that they’re likely the best publicly available pieces of writing in the world about their subjects? Wrt rationality, I expect things like this and this and this and this will probably go down as historically important LW posts that helped us understand the world, and make a strong showing in the 2020 LW Review.
As mentioned in my reply to Ruby, this is not a critique of the LW team, but of the LW mentality. And I should have phrased my point more carefully—“epistemic standards are too low to make any progress” is clearly too strong a claim, it’s more like “epistemic standards are low enough that they’re an important bottleneck to progress”. But I do think there’s a substantive disagreement here. Perhaps the best way to spell it out is to look at the posts you linked and see why I’m less excited about them than you are.
Of the top posts in the 2018 review, and the ones you linked (excluding AI), I’d categorise them as follows:
Interesting speculation about psychology and society, where I have no way of knowing if it’s true:
Local Validity as a Key to Sanity and Civilization
The Loudest Alarm Is Probably False
Anti-social punishment (which is, unlike the others, at least based on one (1) study).
Babble
Intelligent social web
Unrolling social metacognition
Simulacra levels
Can you keep this secret?
Same as above but it’s by Scott so it’s a bit more rigorous and much more compelling:
Is Science Slowing Down?
The tails coming apart as a metaphor for life
Useful rationality content:
Toolbox-thinking and law-thinking
A sketch of good communication
Varieties of argumentative experience
Review of basic content from other fields. This seems useful for informing people on LW, but not actually indicative of intellectual progress unless we can build on them to write similar posts on things that *aren’t* basic content in other fields:
Voting theory primer
Prediction markets: when do they work
Costly coordination mechanism of common knowledge (Note: I originally said I hadn’t seen many examples of people building on these ideas, but at least for this post there seems to be a lot.)
Six economics misconceptions
Swiss political system
It’s pretty striking to me how much the original sequences drew on the best academic knowledge, and how little most of the things above draw on the best academic knowledge. And there’s nothing even close to the thoroughness of Luke’s literature reviews.
The three things I’d like to see more of are:
1. The move of saying “Ah, this is interesting speculation about a complex topic. It seems compelling, but I don’t have good ways of verifying it; I’ll treat it like a plausible hypothesis which could be explored more by further work.” (I interpret the thread I originally linked as me urging Wei to do this).
2. Actually doing that follow-up work. If it’s an empirical hypothesis, investigating empirically. If it’s a psychological hypothesis, does it apply to anyone who’s not you? If it’s more of a philosophical hypothesis, can you identify the underlying assumptions and the ways it might be wrong? In all cases, how does it fit into existing thought? (That’ll probably take much more than a single blog post).
3. Insofar as many of these scattered plausible insights are actually related in deep ways, trying to combine them so that the next generation of LW readers doesn’t have to separately learn about each of them, but can rather download a unified generative framework.
(Thanks for laying out your position in this level of depth. Sorry for how long this comment turned out. I guess I wanted to back up a bunch of my agreement with words. It’s a comment for the sake of everyone else, not just you.)
I think there’s something to what you’re saying, that the mentality itself could be better. The Sequences have been criticized because Eliezer didn’t cite previous thinkers all that much, but at least as far as the science goes, as you said, he was drawing on academic knowledge. I also think we’ve lost something precious with the absence of epic topic reviews by the likes of Luke. Kaj Sotala still brings in heavily from outside knowledge, John Wentworth did a great review on Biological Circuits, and we get SSC crossposts that have that, but otherwise posts aren’t heavily referencing or building upon outside stuff. I concede that I would like to see a lot more of that.
I think Kaj was rightly disappointed that he didn’t get more engagement with his post whose gist was “this is what the science really says about S1 & S2, one of your most cherished concepts, LW community”.
I wouldn’t say the typical approach is strictly bad, there’s value in thinking freshly for oneself or that failure to reference previous material shouldn’t be a crime or makes a text unworthy, but yeah, it’d be pretty cool if after Alkjash laid out Babble & Prune (which intuitively feels so correct), someone had dug through what empirical science we have to see whether the picture lines up. Or heck, actually gone and done some kind of experiment. I bet it would turn up something interesting.
And I think what you’re saying is that the issue isn’t just that people aren’t following up with scholarship and empiricism on new ideas and models, but that they’re actually forgetting that these are the next steps. Instead, they’re overconfident in our homegrown models, as though LessWrong were the one place able to come up with good ideas. (Sorry, some of this might be my own words.)
The category I’d label a lot of LessWrong posts with is “engaging articulation of a point which is intuitive in hindsight” / “creation of common vocabulary around such points”. That’s pretty valuable, but I do think solving the hardest problems will take more.
-----
You use the word “reliably” in a few places. It feels like it’s doing some work in your statements, and I’m not entirely sure what you mean or why it’s important.
-----
A model which is interesting but maybe not of obvious connection. I was speaking to a respected rationalist thinker this week and they classified potential writing on LessWrong into three categories:
Writing stuff to help oneself figure things out. Like a diary, but publicly shared.
People exchanging “letters” as they attempt to figure things out. Like old school academic journals.
Someone having something mostly figured out but with a large inferential distance to bridge. They write a large collection of posts trying to cover that distance. One example is The Sequences, and more recent examples are from John Wentworth and Kaj Sotala
I mention this because I recall you (alongside the rationalist thinker) complaining about the lack of people “presenting their worldviews on LessWrong”.
The kinds of epistemic norms I think you’re advocating for feel like a natural fit for 2nd kind of writing, but it’s less clear to me how they should apply to people presenting world views. Maybe it’s not more complicated than it’s fine to present your worldview without a tonne of evidence, but people shouldn’t forget that the evidence hasn’t been presented and it feeling intuitively correct isn’t enough.
-----
There’s something in here about Epistemic Modesty, something, something. Some part of me reads you as calling for more of that, which I’m wary of, but I don’t currently have more to say than flagging it as maybe a relevant variable in any disagreements here.
We probably do disagree about the value of academic sources, or what it takes to get value from them. Hmm. Maybe it’s something like there’s something to be said for thinking about models and assessing their plausibility yourself rather than relying on likely very flawed empirical studies.
Maybe I’m in favor of large careful reviews of what science knows but less in favor of trying to find sources for each idea or model that gets raised. I’m not sure.
-----
I can’t recall whether I’ve written publicly much about this, but a model I’ve had for a year or more is that for LW to make intellectual progress, we need to become a “community of practice”, not just a “community of interest”. Martial arts vs literal stamp collecting. (Streetfighting might be better still due to actual testing real fighting ability.) It’s great that many people find LessWrong a guilty pleasure they feel less guilty about than Facebook, but for us to make progress, people need to see LessWrong as a place where one of things you do is show up and do Serious Work, some of which is relatively hard and boring, like writing and reading lit reviews.
I suspect that a cap on the epistemic standards people hold stuff to is downstream of the level of effort people are calibrated on applying. But maybe it goes in other direction, so I don’t know.
Probably the 2018 Review is biased towards the posts which are most widely read, i.e., those easiest and most enjoyable to read, rather than solely rewarding those with the best contributions. Not overwhelmingly, but enough. Maybe same for karma. I’m not sure how to relate to that.
-----
This sounds partially like distillation work plus extra integration. And sounds pretty good to me too.
-----
I still remember my feeling of disillusionment in the LessWrong community relative soon after I joined in late 2012. I realized that the bulk of members didn’t seem serious about advancing the Art. I never heard people discussing new results from cognitive science and how to apply them, even though that’s what Sequences were in large part and the Sequences hardly claimed to be complete! I guess I do relate somewhat to your “desperate effort” comment, though we’ve got some people trying pretty hard that I wouldn’t want to short change.
We do good stuff, but more is possible. I appreciate the reminder. I hope we succeed at pushing the culture and mentality in directions you like.
This is only tangentially relevant, but adding it here as some of you might find it interesting:
Venkatesh Rao has an excellent Twitter thread on why most independent research only reaches this kind of initial exploratory level (he tried it for a bit before moving to consulting). It’s pretty pessimistic, but there is a somewhat more optimistic follow-up thread on potential new funding models. Key point is that the later stages are just really effortful and time-consuming, in a way that keeps out a lot of people trying to do this as a side project alongside a separate main job (which I think is the case for a lot of LW contributors?)
Quote from that thread:
Also just wanted to say good luck! I’m a relative outsider here with pretty different interests to LW core topics but I do appreciate people trying to do serious work outside academia, have been trying to do this myself, and have thought a fair bit about what’s currently missing (I wrote that in a kind of jokey style but I’m serious about the topic).
Thanks, these links seem great! I think this is a good (if slightly harsh) way of making a similar point to mine:
“I find that autodidacts who haven’t experienced institutional R&D environments have a self-congratulatory low threshold for what they count as research. It’s a bit like vanity publishing or fan fiction. This mismatch doesn’t exist as much in indie art, consulting, game dev etc”
Also, I liked your blog post! More generally, I strongly encourage bloggers to have a “best of” page, or something that directs people to good posts. I’d be keen to read more of your posts but have no idea where to start.
Thanks! I have been meaning to add a ‘start here’ page for a while, so that’s good to have the extra push :) Seems particularly worthwhile in my case because a) there’s no one clear theme and b) I’ve been trying a lot of low-quality experimental posts this year bc pandemic trashed motivation, so recent posts are not really reflective of my normal output.
For now some of my better posts in the last couple of years might be Cognitive decoupling and banana phones (tracing back the original precursor of Stanovich’s idea), The middle distance (a writeup of a useful and somewhat obscure idea from Brian Cantwell Smith’s On the Origin of Objects), and the negative probability post and its followup.
Quoting your reply to Ruby below, I agree I’d like LessWrong to be much better at “being able to reliably produce and build on good ideas”.
The reliability and focus feels most lacking to me on the building side, rather than the production, which I think we’re doing quite well at. I think we’ve successfully formed a publishing platform that provides and audience who are intensely interested in good ideas around rationality, AI, and related subjects, and a lot of very generative and thoughtful people are writing down their ideas here.
We’re low on the ability to connect people up to do more extensive work on these ideas – most good hypotheses and arguments don’t get a great deal of follow up or further discussion.
Here are some subjects where I think there’s been various people sharing substantive perspectives, but I think there’s also a lot of space for more ‘details’ to get fleshed out and subquestions to be cleanly answered:
Sabbath and Rest Days (Zvi, Lauren Lee, Jacobian, Scott)
Moloch and Slack and Mazes (Scott, Eliezer, Zvi, Swentworth, Jameson)
Inner/Outer Alignment (EvHub, Rafael, Paul, Swentworth, Steve2152)
Embedded Agency + Optimization (Abram, Scott, Swentworth, Alex Flint, nostalgebraist)
Simulacra Levels (Benquo, Zvi, Elizabeth)
AI Takeoff (Paul, Katja, Kokotajlo, Zhukeepa)
Iterated Amplification (Paul, EvHub, Zhukeepa, Vaniver, William S, Wei Dai)
Insight meditation + IFS (Kaj, Kaj, Kaj, and Kaj. Also Abram and Val and Romeo and Scott)
Coordination Problems (Eliezer, Scott, Sustrik, Swentworth, Zvi, me)
The above isn’t complete, it’s just some of the ones that come to mind as having lots of people sharing perspectives. And the list of people definitely isn’t complete.
Here examples of things that I’d like to see more of, that feel more like doing the legwork to actually dive into the details:
Eli Tyre and Bucky replicating Scott’s birth-order hypothesis
Katja and the other fine people at AI Impacts doing long-term research on a question (discontinuous progress) with lots of historical datapoints
Jameson writing up his whole research question in great detail and very well, and then an excellent commenter turning up and answering it
Zhukeepa writing up an explanation of Paul’s research, allowing many more to understand it, and allowing Eliezer to write a response
Scott writing Goodhart Taxonomy, and the commenters banding together to find a set of four similar examples to add to the post
Val writing some interesting things about insight meditation, prompting Kaj to write a non-mysterious explanation
In the LW Review when Bucky checked out the paper Zvi analysed and argued it did not support the conclusions Zvi reached (this changed my opinion of Zvi’s post from ‘true’ to ‘false’)
The discussion around covid and EMH prompting Richard Meadows to write down a lot of the crucial and core arguments around the EMH
The above is also not mentioning lots of times when the person generating the idea does a lot of the legwork, like Scott or Jameson or Sarah or someone.
I see a lot of (very high quality) raw energy here that wants shaping and directing, with the use of lots of tools for coordination (e.g. better collaboration tools).
The epistemic standards being low is one way of putting it, but it doesn’t resonate with me much and kinda feels misleading. I think our epistemic standards are way higher than the communities you mention (historians, people interested in progress studies). Bryan Caplan said he knows of no group whose beliefs are more likely to be right in general than the rationalists, this seems often accurate to me. I think we do a lot of exploration and generation and evaluation, just not in a very coordinated manner, and so could make progress at like 10x–100x the rate if we collaborated better, and I think we can get there without too much work.
“I see a lot of (very high quality) raw energy here that wants shaping and directing, with the use of lots of tools for coordination (e.g. better collaboration tools).”
Yepp, I agree with this. I guess our main disagreement is whether the “low epistemic standards” framing is a useful way to shape that energy. I think it is because it’ll push people towards realising how little evidence they actually have for many plausible-seeming hypotheses on this website. One proven claim is worth a dozen compelling hypotheses, but LW to a first approximation only produces the latter.
When you say “there’s also a lot of space for more ‘details’ to get fleshed out and subquestions to be cleanly answered”, I find myself expecting that this will involve people who believe the hypothesis continuing to build their castle in the sky, not analysis about why it might be wrong and why it’s not.
That being said, LW is very good at producing “fake frameworks”. So I don’t want to discourage this too much. I’m just arguing that this is a different thing from building robust knowledge about the world.
I will continue to be contrary and say I’m not sure I agree with this.
For one, I think in many domains new ideas are really hard to come by, as opposed to making minor progress in the existing paradigms. Fundamental theories in physics, a bunch of general insights about intelligence (in neuroscience and AI), etc.
And secondly, I am reminded of what Lukeprog wrote in his moral consciousness report, that he wished the various different philosophies-of-consciousness would stop debating each other, go away for a few decades, then come back with falsifiable predictions. I sometimes take this stance regarding many disagreements of import, such as the basic science vs engineering approaches to AI alignment. It’s not obvious to me that the correct next move is for e.g. Eliezer and Paul to debate for 1000 hours, but instead to go away and work on their ideas for a decade then come back with lots of fleshed out details and results that can be more meaningfully debated.
I feel similarly about simulacra levels, Embedded Agency, and a bunch of IFS stuff. I would like to see more experimentation and literature reviews where they make sense, but I also feel like these are implicitly making substantive and interesting claims about the world, and I’d just be interested in getting a better sense of what claims they’re making, and have them fleshed out + operationalized more. That would be a lot of progress to me, and I think each of them is seeing that sort of work (with Zvi, Abram, and Kaj respectively leading the charges on LW, alongside many others).
I think I’m concretely worried that some of those models / paradigms (and some other ones on LW) don’t seem pointed in a direction that leads obviously to “make falsifiable predictions.”
And I can imagine worlds where “make falsifiable predictions” isn’t the right next step, you need to play around with it more and get it fleshed out in your head before you can do that. But there is at least some writing on LW that feels to me like it leaps from “come up with an interesting idea” to “try to persuade people it’s correct” without enough checking.
(In the case of IFS, I think Kaj’s sequence is doing a great job of laying it out in a concrete way where it can then be meaningfully disagreed with. But the other people who’ve been playing around with IFS didn’t really seem interested in that, and I feel like we got lucky that Kaj had the time and interest to do so.)
I feel like this comment isn’t critiquing a position I actually hold. For example, I don’t believe that “the correct next move is for e.g. Eliezer and Paul to debate for 1000 hours”. I am happy for people to work towards building evidence for their hypotheses in many ways, including fleshing out details, engaging with existing literature, experimentation, and operationalisation.
Perhaps this makes “proven claim” a misleading phrase to use. Perhaps more accurate to say: “one fully fleshed out theory is more valuable than a dozen intuitively compelling ideas”. But having said that, I doubt that it’s possible to fully flesh out a theory like simulacra levels without engaging with a bunch of academic literature and then making predictions.
I also agree with Raemon’s response below.
A housemate of mine said to me they think LW has a lot of breadth, but could benefit from more depth.
I think in general when we do intellectual work we have excellent epistemic standards, capable of listening to all sorts of evidence that other communities and fields would throw out, and listening to subtler evidence than most scientists (“faster than science”), but that our level of coordination and depth is often low. “LessWrongers should collaborate more and go into more depth in fleshing out their ideas” sounds more true to me than “LessWrongers have very low epistemic standards”.
“Being more openminded about what evidence to listen to” seems like a way in which we have lower epistemic standards than scientists, and also that’s beneficial. It doesn’t rebut my claim that there are some ways in which we have lower epistemic standards than many academic communities, and that’s harmful.
In particular, the relevant question for me is: why doesn’t LW have more depth? Sure, more depth requires more work, but on the timeframe of several years, and hundreds or thousands of contributors, it seems viable. And I’m proposing, as a hypothesis, that LW doesn’t have enough depth because people don’t care enough about depth—they’re willing to accept ideas even before they’ve been explored in depth. If this explanation is correct, then it seems accurate to call it a problem with our epistemic standards—specifically, the standard of requiring (and rewarding) deep investigation and scholarship.
Your solution to the “willingness to accept ideas even before they’ve been explored in depth” problem is to explore ideas in more depth. But another solution is to accept fewer ideas, or hold them much more provisionally.
I’m a proponent of the second approach because:
I suspect even academia doesn’t hold ideas as provisionally as it should. See Hamming on expertise: https://forum.effectivealtruism.org/posts/mG6mckPHAisEbtKv5/should-you-familiarize-yourself-with-the-literature-before?commentId=SaXXQXLfQBwJc9ZaK
I suspect trying to browbeat people to explore ideas in more depth works against the grain of an online forum as an institution. Browbeating works in academia because your career is at stake, but in an online forum, it just hurts intrinsic motivation and cuts down on forum use (the forum runs on what Clay Shirky called “cognitive surplus”, essentially a term for peoples’ spare time and motivation). I’d say one big problem with LW 1.0 that LW 2.0 had to solve before flourishing was people felt too browbeaten to post much of anything.
If we accept fewer ideas / hold them much more provisionally, but provide a clear path to having an idea be widely held as true, that creates an incentive for people to try & jump through hoops—and this incentive is a positive one, not a punishment-driven browbeating incentive.
Maybe part of the issue is that on LW, peer review generally happens in the comments after you publish, not before. So there’s no publication carrot to offer in exchange for overcoming the objections of peer reviewers.
“If we accept fewer ideas / hold them much more provisionally, but provide a clear path to having an idea be widely held as true, that creates an incentive for people to try & jump through hoops—and this incentive is a positive one, not a punishment-driven browbeating incentive.”
Hmm, it sounds like we agree on the solution but are emphasising different parts of it. For me, the question is: who’s this “we” that should accept fewer ideas? It’s the set of people who agree with my argument that you shouldn’t believe things which haven’t been fleshed out very much. But the easiest way to add people to that set is just to make the argument, which is what I’ve done. Specifically, note that I’m not criticising anyone for producing posts that are short and speculative: I’m criticising the people who update too much on those posts.
Fair enough. I’m reminded of a time someone summarized one of my posts as being a definitive argument against some idea X and me thinking to myself “even I don’t think my post definitively settles this issue” haha.
Yeah, this is roughly how I think about it.
I do think right now LessWrong should lean more in the direction the Richard is suggesting – I think it was essential to establish better Babble procedures but now we’re doing well enough on that front that I think setting clearer expectations of how the eventual pruning works is reasonable.
I wanted to register that I don’t like “babble and prune” as a model of intellectual development. I think intellectual development actually looks more like:
1. Babble
2. Prune
3. Extensive scholarship
4. More pruning
5. Distilling scholarship to form common knowledge
And that my main criticism is the lack of 3 and 5, not the lack of 2 or 4.
I also note that: a) these steps get monotonically harder, so that focusing on the first two misses *almost all* the work; b) maybe I’m being too harsh on the babble and prune framework because it’s so thematically appropriate for me to dunk on it here; I’m not sure if your use of the terminology actually reveals a substantive disagreement.
I basically agree with your 5-step model (I at least agree it’s a more accurate description than Babel and Prune, which I just meant as rough shorthand). I’d add things like “original research/empiricism” or “more rigorous theorizing” to the “Extensive Scholarship” step.
I see the LW Review as basically the first of (what I agree should essentially be at least) a 5 step process. It’s adding a stronger Step 2, and a bit of Step 5 (at least some people chose to rewrite their posts to be clearer and respond to criticism)
...
Currently, we do get non-zero Extensive Scholarship and Original Empiricism. (Kaj’s Multi-Agent Models of Mind seems like it includes real scholarship. Scott Alexander / Eli Tyre and Bucky’s exploration into Birth Order Effects seemed like real empiricism). Not nearly as much as I’d like.
But John’s comment elsethread seems significant:
This reminded of a couple posts in the 2018 Review, Local Validity as Key to Sanity and Civilization, and Is Clickbait Destroying Our General Intelligence?. Both of those seemed like “sure, interesting hypothesis. Is it real tho?”
During the Review I created a followup “How would we check if Mathematicians are Generally More Law Abiding?” question, trying to move the question from Stage 2 to 3. I didn’t get much serious response, probably because, well, it was a much harder question.
But, honestly… I’m not sure it’s actually a question that was worth asking. I’d like to know if Eliezer’s hypothesis about mathematicians is true, but I’m not sure it ranks near the top of questions I’d want people to put serious effort into answering.
I do want LessWrong to be able to followup Good Hypotheses with Actual Research, but it’s not obvious which questions are worth answering. OpenPhil et al are paying for some types of answers, I think usually by hiring researchers full time. It’s not quite clear what the right role for LW to play in the ecosystem.
All else equal, the harder something is, the less we should do it.
My quick take is that writing lit reviews/textbooks is a comparative disadvantage of LW relative to the mainstream academic establishment.
In terms of producing reliable knowledge… if people actually care about whether something is true, they can always offer a cash prize for the best counterargument (which could of course constitute citation of academic research). The fact that people aren’t doing this suggests to me that for most claims on LW, there isn’t any (reasonably rich) person who cares deeply re: whether the claim is true. I’m a little wary of putting a lot of effort into supply if there is an absence of demand.
(I guess the counterargument is that accurate knowledge is a public good so an individual’s willingness to pay doesn’t get you complete picture of the value accurate knowledge brings. Maybe what we need is a way to crowdfund bounties for the best argument related to something.)
(I agree that LW authors would ideally engage more with each other and academic literature on the margin.)
I’ve been thinking about the idea of “social rationality” lately, and this is related. We do so much here in the way of training individual rationality—the inputs, functions, and outputs of a single human mind. But if truth is a product, then getting human minds well-coordinated to produce it might be much more important than training them to be individually stronger. Just as assembly line production is much more effective in producing almost anything than teaching each worker to be faster in assembling a complete product by themselves.
My guess is that this could be effective not only in producing useful products, but also in overcoming biases. Imagine you took 5 separate LWers and asked them to create a unified consensus response to a given article. My guess is that they’d learn more through that collective effort, and produce a more useful response, than if they spent the same amount of time individually evaluating the article and posting their separate replies.
Of course, one of the reasons we don’t to that so much is that coordination is an up-front investment and is unfamiliar. Figuring out social technology to make it easier to participate in might be a great project for LW.
There’s been a fair amount of discussion of that sort of thing here: https://www.lesswrong.com/tag/group-rationality There are also groups outside LW thinking about social technology such as RadicalxChange.
I’m not sure. If you put those 5 LWers together, I think there’s a good chance that the highest status person speaks first and then the others anchor on what they say and then it effectively ends up being like a group project for school with the highest status person in charge. Some related links.
That’s definitely a concern too! I imagine such groups forming among people who either already share a basic common view, and collaborate to investigate more deeply. That way, any status-anchoring effects are mitigated.
Alternatively, it could be an adversarial collaboration. For me personally, some of the SSC essays in this format have led me to change my mind in a lasting way.
People also reject ideas before they’ve been explored in depth. I’ve tried to discuss similar issues with LW before but the basic response was roughly “we like chaos where no one pays attention to whether an argument has ever been answered by anyone; we all just do our own thing with no attempt at comprehensiveness or organizing who does what; having organized leadership of any sort, or anyone who is responsible for anything, would be irrational” (plus some suggestions that I’m low social status and that therefore I personally deserve to be ignored. there were also suggestions – phrased rather differently but amounting to this – that LW will listen more if published ideas are rewritten, not to improve on any flaws, but so that the new versions can be published at LW before anywhere else, because the LW community’s attention allocation is highly biased towards that).
I feel somewhat inclined to wrap up this thread at some point, even while there’s more to say. We can continue if you like and have something specific or strong you’d like to ask, but otherwise will pause here.
You have to realise that what you are doing isn’t adequate in order to gain the motivation to do it better, and that is unlikely to happen if you are mostly communicating with other people who think everything is OK.
Lesswrong is competing against philosophy as well as science, and philosophy has broader criterion of evidence still. In fact , lesswrongians are often frustrated that mainstream philosophy takes such topics as dualism or theism seriously.. even though theres an abundance of Bayesian evidence for them.
Depends on the claim, right?
If the cost of evaluating a hypothesis is high, and hypotheses are cheap to generate, I would like to generate a great deal before selecting one to evaluate.
As mentioned in this comment, the Unrolling social metacognition paper is closely related to at least one research paper.
Right, but this isn’t mentioned in the post? Which seems odd. Maybe that’s actually another example of the “LW mentality”: why is the fact that there has been solid empirical research into 3 layers not being enough not important enough to mention in a post on why 3 layers isn’t enough? (Maybe because the post was time-boxed? If so that seems reasonable, but then I would hope that people comment saying “Here’s a very relevant paper, why didn’t you cite it?”)
[Deleted]
Much of the same is true of scientific journals. Creating a place to share and publish research is a pretty key piece of intellectual infrastructure, especially for researchers to create artifacts of their thinking along the way.
The point about being ‘cross-posted’ is where I disagree the most.
This is largely original content that counterfactually wouldn’t have been published, or occasionally would have been published but to a much smaller audience. What Failure Looks Like wasn’t crossposted, Anna’s piece on reality-revealing puzzles wasn’t crossposted. I think that Zvi would have still written some on mazes and simulacra, but I imagine he writes substantially more content given the cross-posting available for the LW audience. Could perhaps check his blogging frequency over the last few years to see if that tracks. I recall Zhu telling me he wrote his FAQ because LW offered an audience for it, and likely wouldn’t have done so otherwise. I love everything Abram writes, and while he did have the Intelligent Agent Foundations Forum, it had a much more concise, technical style, tiny audience, and didn’t have the conversational explanations and stories and cartoons that have been so excellent and well received on LW, and it wouldn’t as much have been focused on the implications for rationality of things like logical inductors. Rohin wouldn’t have written his coherence theorems piece or any of his value learning sequence, and I’m pretty sure about that because I personally asked him to write that sequence, which is a great resource and I’ve seen other researchers in the field physically print off to write on and study. Kaj has an excellent series of non-mystical explanations of ideas from insight meditation that started as a response to things Val wrote, and I imagine those wouldn’t have been written quite like that if that context did not exist on LW.
I could keep going, but probably have made the point. It seems weird to not call this collectively a substantial amount of intellectual progress, on a lot of important questions.
I am indeed focusing right now on how to do more ‘conversation’. I’m in the middle of trying to host some public double cruxes for events, for example, and some day we will finally have inline commenting and better draft sharing and so on. It’s obviously not finished.
Yeah, that’s true, though it might have happened at some later point in the future as I got increasingly frustrated by people continuing to cite VNM at me (though probably it would have been a blog post and not a full sequence).
Reading through this comment tree, I feel like there’s a distinction to be made between “LW / AIAF as a platform that aggregates readership and provides better incentives for blogging”, and “the intellectual progress caused by posts on LW / AIAF”. The former seems like a clear and large positive of LW / AIAF, which I think Richard would agree with. For the latter, I tend to agree with Richard, though perhaps not as strongly as he does. Maybe I’d put it as, I only really expect intellectual progress from a few people who work on problems full time who probably would have done similar-ish work if not for LW / AIAF (but likely would not have made it public).
I’d say this mostly for the AI posts. I do read the rationality posts and don’t get a different impression from them, but I also don’t think enough about them to be confident in my opinions there.
By “AN” do you mean the AI Alignment Forum, or “AIAF”?
[Deleted]
I did suspect you’d confused it with the Alignment Newsletter :)
Thanks for chiming in with this. People criticizing the epistemics is hopefully how we get better epistemics. When the Californian smoke isn’t interfering with my cognition as much, I’ll try to give your feedback (and Rohin’s) proper attention. I would generally be interested to hear your arguments/models in detail, if you get the chance to lay them out.
My default position is LW has done well enough historically (e.g. Ben Pace’s examples) for me to currently be investing in getting it even better. Epistemics and progress could definitely be a lot better, but getting there is hard. If I didn’t see much progress on the rate of progress in the next year or two, I’d probably go focus on other things, though I think it’d be tragic if we ever lost what we have now.
And another thought:
Yes and no. Journal articles have their advantages, and so do blog posts. A bunch of recent LessWrong team’s work has been around filling in the missing pieces for the system to work, e.g. Open Questions (hasn’t yet worked for coordinating research), Annual Review, Tagging, Wiki. We often talk about conferences and “campus”.
My work on Open Questions involved thinking about i) a better template for articles than “Abstract, Intro, Methods, etc.”, but Open Questions didn’t work for unrelated reasons we haven’t overcome yet, ii) getting lit reviews done systematically by people, iii) coordinating groups around research agendas.
I’ve thought about re-attempting the goals of Open Questions with instead a “Research Agenda” feature that lets people communally maintain research agendas and work on them. It’s a question of priorities whether I work on that anytime soon.
I do really think many of the deficiencies of LessWrong’s current work compared to academia are “infrastructure problems” at least as much as the epistemic standards of the community. Which means the LW team should be held culpable for not having solved them yet, but it is tricky.
For the record, I think the LW team is doing a great job. There’s definitely a sense in which better infrastructure can reduce the need for high epistemic standards, but it feels like the thing I’m pointing at is more like “Many LW contributors not even realising how far away we are from being able to reliably produce and build on good ideas” (which feels like my criticism of Ben’s position in his comment, so I’ll respond more directly there).
It seems really valuable to have you sharing how you think we’re falling epistemically short and probably important for the site to integrate the insights behind that view. There are a bunch of ways I disagree with your claims about epistemic best practices, but it seems like it would be cool if I could pass your ITT more. I wish your attempt to communicate the problems you saw had worked out better. I hope there’s a way for you to help improve LW epistemics, but also get that it might be costly in time and energy.
Now they’re positive again.
Confusing to me, their Ω-karma (karma on another website) is also positive. Does it mean they previously had negative LW-karma but positive Ω-karma? Or that their Ω-karma also improved as a result of you complaining on LW a few hours ago? Why would it?
(Feature request: graph of evolution of comment karma as a function of time.)
I’m confused, what is Ω-karma?
AI Alignment Forum karma (which is also displayed here on posts that are crossposted)
I’d be curious what, if any, communities you think set good examples in this regard. In particular, are there specific academic subfields or non-academic scenes that exemplify the virtues you’d like to see more of?
Maybe historians of the industrial revolution? Who grapple with really complex phenomena and large-scale patterns, like us, but unlike us use a lot of data, write a lot of thorough papers and books, and then have a lot of ongoing debate on those ideas. And then the “progress studies” crowd is an example of an online community inspired by that tradition (but still very nascent, so we’ll see how it goes).
More generally I’d say we could learn to be more rigorous by looking at any scientific discipline or econ or analytic philosophy. I don’t think most LW posters are in a position to put in as much effort as full-time researchers, but certainly we can push a bit in that direction.
Thanks for your reply! I largely agree with drossbucket’s reply.
I also wonder how much this is an incentives problem. As you mentioned and in my experience, the fields you mentioned strongly incentivize an almost fanatical level of thoroughness that I suspect is very hard for individuals to maintain without outside incentives pushing them that way. At least personally, I definitely struggle and, frankly, mostly fail to live up to the sorts of standards you mention when writing blog posts in part because the incentive gradient feels like it pushes towards hitting the publish button.
Given this, I wonder if there’s a way to shift the incentives on the margin. One minor thing I’ve been thinking of trying for my personal writing is having a Knuth or Nintil style “pay for mistakes” policy. Do you have thoughts on other incentive structures to for rewarding rigor or punishing the lack thereof?
It feels partly like an incentives problem, but also I think a lot of people around here are altruistic and truth-seeking and just don’t realise that there are much more effective ways to contribute to community epistemics than standard blog posts.
I think that most LW discussion is at the level where “paying for mistakes” wouldn’t be that helpful, since a lot of it is fuzzy. Probably the thing we need first are more reference posts that distill a range of discussion into key concepts, and place that in the wider intellectual context. Then we can get more empirical. (Although I feel pretty biased on this point, because my own style of learning about things is very top-down). I guess to encourage this, we could add a “reference” section for posts that aim to distill ongoing debates on LW.
In some cases you can get a lot of “cheap” credit by taking other people’s ideas and writing a definitive version of them aimed at more mainstream audiences. For ideas that are really worth spreading, that seems useful.