I basically agree with your 5-step model (I at least agree it’s a more accurate description than Babel and Prune, which I just meant as rough shorthand). I’d add things like “original research/empiricism” or “more rigorous theorizing” to the “Extensive Scholarship” step.
I see the LW Review as basically the first of (what I agree should essentially be at least) a 5 step process. It’s adding a stronger Step 2, and a bit of Step 5 (at least some people chose to rewrite their posts to be clearer and respond to criticism)
...
Currently, we do get non-zero Extensive Scholarship and Original Empiricism. (Kaj’s Multi-Agent Models of Mind seems like it includes real scholarship. Scott Alexander / Eli Tyre and Bucky’s exploration into Birth Order Effects seemed like real empiricism). Not nearly as much as I’d like.
If the cost of evaluating a hypothesis is high, and hypotheses are cheap to generate, I would like to generate a great deal before selecting one to evaluate.
But, honestly… I’m not sure it’s actually a question that was worth asking. I’d like to know if Eliezer’s hypothesis about mathematicians is true, but I’m not sure it ranks near the top of questions I’d want people to put serious effort into answering.
I do want LessWrong to be able to followup Good Hypotheses with Actual Research, but it’s not obvious which questions are worth answering. OpenPhil et al are paying for some types of answers, I think usually by hiring researchers full time. It’s not quite clear what the right role for LW to play in the ecosystem.
I basically agree with your 5-step model (I at least agree it’s a more accurate description than Babel and Prune, which I just meant as rough shorthand). I’d add things like “original research/empiricism” or “more rigorous theorizing” to the “Extensive Scholarship” step.
I see the LW Review as basically the first of (what I agree should essentially be at least) a 5 step process. It’s adding a stronger Step 2, and a bit of Step 5 (at least some people chose to rewrite their posts to be clearer and respond to criticism)
...
Currently, we do get non-zero Extensive Scholarship and Original Empiricism. (Kaj’s Multi-Agent Models of Mind seems like it includes real scholarship. Scott Alexander / Eli Tyre and Bucky’s exploration into Birth Order Effects seemed like real empiricism). Not nearly as much as I’d like.
But John’s comment elsethread seems significant:
This reminded of a couple posts in the 2018 Review, Local Validity as Key to Sanity and Civilization, and Is Clickbait Destroying Our General Intelligence?. Both of those seemed like “sure, interesting hypothesis. Is it real tho?”
During the Review I created a followup “How would we check if Mathematicians are Generally More Law Abiding?” question, trying to move the question from Stage 2 to 3. I didn’t get much serious response, probably because, well, it was a much harder question.
But, honestly… I’m not sure it’s actually a question that was worth asking. I’d like to know if Eliezer’s hypothesis about mathematicians is true, but I’m not sure it ranks near the top of questions I’d want people to put serious effort into answering.
I do want LessWrong to be able to followup Good Hypotheses with Actual Research, but it’s not obvious which questions are worth answering. OpenPhil et al are paying for some types of answers, I think usually by hiring researchers full time. It’s not quite clear what the right role for LW to play in the ecosystem.