What Makes My Attempt Special?
A crucial question towards the beginning of any research project is, why should my group succeed in elucidating an answer to a question where others may have tried and failed?
Here’s how I’m going about dividing the possible worlds, but I’m interested to see if anyone has any other strategies. First, the whole question is conditional on nobody having already answered the particular question you’re interested in. So, you first need an exhaustive lit review, that should scale in intensity based on how much effort you expect to actually expend on the project. Still nothing? These are the remaining possibilities:
1) Nobody else has ever thought of your question, even though all of the pieces of knowledge needed to formulate it have been known for years. If the field has many people involved, the probability of this is vanishingly small and you should systematically disabuse yourself of your fantasies if you think like this often. Still… if true, the prognosis: a good sign.
2) Nobody else has ever thought of your question, because it wouldn’t have been ask-able without pieces of knowledge that were discovered just recently. This is common in fast-paced fields and it’s why they can be especially exciting. The prognosis: a good sign, but work quickly!
3) Others have thought of your question, but didn’t think it was interesting enough to devote serious attention to. We should take this seriously, as how informed others choose to allocate their attention is one of our better approximations to real prediction markets. So, the prognosis: bad sign. Figure out whether you can not only answer your question but validate its usefulness / importance, too.
4) Others have thought of your question, thought it was interesting, but have never tried to answer it because of resource or tech restraints, which you do not face. Prognosis: probably the best-case scenario.
5) Others have thought of your question and run the relevant tests, but failed to get any consistent / reliable results. It’d be nice if there were no publication bias but of course there is—people are much more likely to publish statistically significant, positive results. Due to this bias, it is sometimes hard to tell precisely how many dead skeletons and dismembered brains line your path, and because of this uncertainty you must assign this possibility a non-zero probability. The prognosis: a bad sign, but do you feel lucky?
6) Others have thought of your question, run the relevant tests, and failed to get consistent / reliable results, but used a different method than the one you will use. Your new tech might clear up some of the murkiness, but it’s important here to be precise about which specific issues your method solves and which it doesn’t. The prognosis: all things equal, a good sign.
These are the considerations we make when we decide whether to pursue a given topic. But even if you do choose to pursue the question, some of these possibilities have policy recommendations for how to proceed. For example, using new tech, even if it’s not necessarily demonstrably better in all cases, seems like a good idea given the possibility of #6.
I also note that in this case other people may have competing outright opposing interests (in the ‘aggressively want to get’ sense rather than just the ‘curiosity’ sense). In such cases it isn’t a matter of not interesting enough. Some people will have an interest in outright suppressing that kind of discovery. I have a relative who has been battling to get research done on alternative fertaliser methods. Pivot really don’t like people doing that and they have purchased a lot of power in most of the relevant academic institutions.
Prognosis is good for making discoveries, but abysmal for gaining status thereby.
Voted up for clear thinking and relevance.
And, as this is Less Wrong, I would add a seventh:
7) Others have thought of your question, found it challenging to let the question hang long enough to do research, failed to meet the challenge, let their biases choose their favorite answer to the question, and ran enough tests to produce a superficial veneer of data supporting their preferred answer. Your superior rationality might be enough to attempt an honest answer to the question, but you must also consider the possibility that the question is very hard to consider fairly. The prognosis: good, unless your Bayesian Jedi mind tricks are weak.
There are two equal and opposite faults here. One is assuming that you have some special kind of genius to see the answer to a problem which others have tried and failed to answer.
The opposite mistake is thinking that everything has been tried before—that there is nothing new under the sun. There are quite a lot of old and obvious questions still out there in plain sight which nobody has answered, and which aren’t actually all that hard. There are also many easy questions out there where the given answer is wrong, and the right one can be given quite easily. The world of science is still very new, and lots of things have not yet been sorted out.
In my experience, the dumbest thing that’s often done is answering a question that’s already been answered before. It’s tempting because often it isn’t that hard to do, and a very large proportion of scientific research seems to do it. It’s another good reason to research your field.
Part of the reason this happens is simple isolation. Another group is using different jargon, different techniques and so on, and because search relies on words, you never find the work that’s been done before. It really isn’t easy to know if nobody has asked your question before, unless there’s an answer too.
I think this might underestimate the number of good questions/ideas in a field relative to the number of researchers. I think humans are already dealing with utterly huge problem spaces and carving out little areas of working technology, But there are many orders of magnitude times more of other possible technologies.
It could be that your idea is based on old information, but you only thought of it because it serves some weird specific goal, and no one in history has had that goal and knowledge of your field.
Or you’re combining old information from several obscure fields.
I think there are other hints as to whether your ideas has been though of before, but I don’t have a way to express them other than “obviousness”.
8) If the question is in a field X, but the question was posed by people not in field X. This is a good sign.
Note that this sometimes happens in math where people will pose a question in one area of math and not realize that it is really equivalent to or related to some question in another branch of math.
Another one for mathematics:
People have asked this question before and created thorough answers, but you add one to the dimension (or rank or genus or...) and see what happens!
Interesting, I’ll try to watch out for this in the future. One problem is that in current science the canonical “field” is breaking down, as everything is getting computational and uses lots of similar methods. But people who use legit cross disciplinary approaches do often have useful new contributions.
Some questions/answers are hard to think of, not because we are missing knowledge or technology but because they are hard to think of.
I guess this is basically case (1), but I think “vanishingly small” is a bad description in most fields. For example, a significant fraction of advances in theoretical fields are of this form (the exception being for rapidly developing fields). I agree that if you think you have found a proof that P!=NP you should stop and reflect on how many people have thought for a very long time about it, but in general you should not be shocked that you have thought of a question or answer that eluded other researchers, potentially for a rather long time (since there are very many questions and not that many researchers equipped to ask or answer most of them).
I guess you are claiming that the situation is fundamentally different for experimentalists, but I would still be rather shocked if (1) was really negligible. At best there may be some fields where (1) is particularly uncommon (and for people who are still deciding what they want to do with their lives, it seems like this suggests a lot of interesting data which are really hard to extract from the normal discourse about possible fields of study).
“Vanishingly small” was a bit incendiary, and it will depend on the field. This is why theoretical physics, where (1) is common, seems so scary and ego-driven, it’s all about how smart you are relative to your peers.
But in general I think it’s correct to say that the odds your idea is so special really are low. The equal odds rule says that the average publication of any particular scientist does not have any statistically different chance of having more of an impact (i.e., more citations) than any other scientist’s average publication. Most papers are lost to the ether, and published papers are really just the tip of the iceburg of most ideas.
Recently I’ve run into a case where my best guess for what’s happened is the following:
10) There’s an obvious stopping point, so the people who have previously looked at this question stopped there, without checking for easy ways to continue past. (Related to magfrump’s comment, I think.)
Was this in the context of integer complexity? If so, I have no idea what obvious stopping point you are talking about.
Oh boy, people are going to have no idea what we’re talking about. :) I’m referring to the fact I couldn’t find any references to ||2^m 3^k||=2m+3k being known except for m<=2, when there’s an easy proof that works for m<=10 (and Rawsthorne had proved enough to infer that it’s true for m<=13 if he had thought of it). And I mean, really, once I thought to actually ask the question, “can I prove this for m=3”, and then actually sit down and work on it, getting it for m=3 and then m<=10 was really straightforward...
(Background for everyone else: Here ||n||, the “complexity” of n, refers to the smallest number of 1s needed to write n using addition and multiplication; Sloane entry here. JoshuaZ and I have been doing some work on it. Actually this post brought up a very similar notion, leading JoshuaZ to ask if he actually meant integer complexity. Whether or not ||2^m 3^k||=2m+3k in general (for m,k not both zero) is unknown; we’ve managed to prove it for m<=30.)
I don’t think that 3) is a bad sign if you can argue why other didn’t find it interesting but it’s actually interesting. A lot of scientific breakthrough came from answering questions that nobody thought to be interesting at the time.
A good candidate would be a question that explores unexplored areas but that isn’t high status.
All of your bullet points involve the opinions or results of other people. That may be important, and useful as a fast heuristic, but it seems like it will promote groupthink.
9) Your question combines insides from many different fields and there aren’t many people around who are interested in the same fields.
This seems especially likely to me, given the vastness of scientific literature. It can be generalized to “your question requires a knowledge combination that is somewhat unlikely to exist.”
Somewhat related: The Fish-Scale Model of Omniscience
Related RSA Animate: “Where good ideas come from”
http://www.youtube.com/watch?v=NugRZGDbPFU
A good list, and I’m enjoying peoples’ additions. An important point to keep in mind when formulating this sort of analysis is not to generalize more than the system in question merits. In this case, it’s not emphasized that most research is institutionalized, and fueled by the social needs or the beliefs of non-researchers.
I’ve heard many people suggest a variation on #3 + #4:
11) People have thought of your question before, but research has always been smothered by non-researchers. Those who provide resources for research in that field haven’t found the question to be worth their money/approval/etc. If you find independent funding from an iconoclast, and have the chutzpah to break from the primary research institutions, you have a unique opportunity.
Yes, agree you could go much more in depth on what people’s lack of attention / resources indicates and why they might be biased. But often you can pick up on what the sentiment of people in the actual field would like to do if they had the resources, and this is useful info.
This is the one that has most stymied my creativity. Interesting research will often require resources, i.e. political backing. A researcher with powerful friends has far larger probability of success than the proverbial lone tortured genius.
A couple of years ago I had the opportunity to attend the annual meeting of the American Psychiatric Association, which rotated into the city I was living in. It was all about drugs. If you have new techniques for treating mental illness which do not involve the pharmaceutical corporations, it looks to me like you are on your own.
In my own field if I want to pursue stuff that bosses and administrators are interested in, I have a large playing field. My own ideas? Nobody is even interested in listening to them, let alone permitting me work time, computer time, &c. And if I pursue them on my own I can get the “not team player” or “hobbyist” labels of career doom.
Andy, it seems you might have experience in a particular field and I’d be interested to know which one if you’re inclined share that info.
Broadly construed, biology. You may find my neuroscience blog of interest: http://brainslab.wordpress.com/