Optimising Scientific Research
Traditional science is too slow Yudkowsky says. Traditional science finds it acceptable for you to waste 30 years on a false theory as long as you disconfirm it at the end. Traditional science allows you to privilege hypothesis as long as you duly test them. Traditional science does not prevent you from ‘discovering’ magic explanations. Traditional science is insufficient. Traditional science has been tested and found wanting.
And I agree, research as it is currently practiced is (to the best of my knowledge) very unoptimised. There are easily correctable inefficiencies in research that could be eliminated, to increase the speed of converging on the correct theories. Merely practicing Bayesian updating is not enough to eliminate all the inefficiences in research. I get the sense that more is possible. but as best as I can tell, the methods I propose in this post is an optimum one. ,
Table of Contents
A Mathematical Model of Scientific Research
Efficient Experimental Selection
Optimised Experiment Design
Optimised Hypothesis Selection
Brief Summary
A Mathematical Model of Scientific Research
Introduces the model I use to analyse scientific research, and lays the foundations for the entire project. Without this paper, none of the rest would be intelligible.
Efficient Experimental Selection
When confronted with a research problem, what atrategy should we adopt when selecting experiments in order to converge on the correct hypothesis as soon as possible? How do we optimise our experiments in order to make the experimental process as efficient as possible?
Optimised Experiment Design
How do we design our experiments so that they are optimal? So that we only consider the best experiments at any point in time. How do we further optimise our research to make it as efficient as possible?
Optimised Hypothesis Selection
How do we select our hypotheses so that we only consider the best hypotheses? How do we prevent ourselves from wasting time on hypothesis that are not true (and not likely to be true). How do we optimise our research in the very hypotheses we investigate?
I ~expect to publish the first post within a (week p = (0.33)/fortnight p = (0.5), month p = (0.9)) of writing this~ have vastly underestimated the project, and have no idea when I’ll finish. I think the first paper should be done before the end of this year (I’m too busy to write it right now, but I’m about halfway done, and would be free again in December). As for the remaining posts, I do not know how many they will be (apart from the 3 I’ve highlighted above), when the next post would be out (final year undergrad student, so I’m far busier than I like) ~or if I would ever complete this series~ I intend to make an extraordinary effort to see the series . After I write the first post (and possibly others), I may open the series to community input (in the sense that others can add to the series).
Some thoughts:
Arguably the biggest source of inefficiency in scientific research is perverse incentives. (See http://online.liebertpub.com/doi/pdf/10.1089/ees.2016.0223) This is a sociological problem rather than a statistical problem.
The research payoff curve is fat-tailed (some research, leads to $10^12, other research leads to -$10^2. This makes traditional optimisation methods dangerous. (Think like a Hollywood Producer or Venture Capitalist rather than a Engine designer. One “hit” can compensate for a hell of a lot of misses.)
Once a hypothesis (or hypothesis space) has been formulated to the degree that there are computable probabilities, your research work is 9/10ths done. The bit after that can arguably be outsourced or automated.
Thanks. I don’t intend to o consider perverse incentives (I want something that is applicable by individuals).
I would consider the difference in expected payoff of research.
When Feymann draw his diagrams for which he later won the nobel price he wasn’t doing it because he believed it to be the optimal way to solve issues in quantum physics. Instead he was simply curious where drawing his diagrams will lead.
I don’t believe that only the best hypothesis should be considered. After Thomas Kuhn when a new scientific paradigm begins it doesn’t offer the best hypothesis to solve most problems.
Many large discoveries are the result of investigating interesting mechanisms before you have any good idea of the problems that they will solve. It’s good to have slack that allows researchers to solve problems that they didn’t set out to solve at the beginning.
Modern grant making is likely optimizing too much for a researcher focusing all his energy on solving a very narrow problem.
A while ago we had an LW discussion where a person asked for how to best determine the probability of a woman reacting positively when he asks her out for a date. Other people thought that this was ricidulous and not the best approach to getting good at dating.
It’s likely true that it’s not the optimal strategy. On the other hand, I think it’s still a very valuable learning exercise to write down probabilities for events like that.
Thanks for this. This is a much larger project than I expected. So I’m changing the schedule. I have no idea when I’ll finish, buy I want to finish.
I have a preliminary first draft of the paper available.
The first draft is very incomplete, so I’m not posting it here as its own post yet. If you want to get the sense of what I’m doing though, you should read it: current version of paper.
Please review, criticise and make suggestions.
I already said that I don’t believe in the core assumptions you are making that it’s a good idea to try to go for an optimal research strategy.
As far as being serious about this, what’s with prior art? It’s likely that some economist has asked himself already the question of how to model scientific research. You could search for the prior art and argue why you deviate at certain points and what your motivation for the deviation is.
Do you mean “deviate” (as opposed to to “derivate”)?
Yes, I corrected the typo.
How to do Optimal Scientific Research, by DragonGod:
Create a LaTeX document summarizing your preliminary ideas
Share it on LessWrong for feedback
Create more LaTeX documents
???
Profit
One one hand, 1) and 2) pretty accurately reflect real research, except you’re usually sharing with your advisor/peers/etc. On the other hand, they’re the easy part :-)
This is technically true, but I’m writing papers on how to do research (as opposed to conducting the research itself), so I don’t see a problem with it?
What do you think does “research” happen to be? A traditional idea of research is that it’s about seeking the truth.
Are you saying that whatever you are doing has some different goal?
What I’m doing is to devise an optimum strategy for truth seeking; I do not think that is in itself truth seeking.
Why not? Do you think the thesis that a specific strategy is the optimum strategy for truth seeking can’t be true in principle?
It is not an empirical truth. It is at best a logical truth (and only do because of the axioms I choose). Maybe it is possible that there is an empirically optimal strategy—I doubt I’ll ever find it. The best I can do is show that if you accept some axioms, some specific strategy is optimal. The truth of optimality then is analytical rather than synthetic. I do not call what I’m doing truthseeking because of that.