I’m not sure if I can answer, but I can give historical background. This is by no means a new question—philosophy of science has been going at this for decades. What we are really talking about is comparing methodologies and standards of evidence. There are a lot of responses to this question, the most important probably being Thomas Kuhn’s and John Dewey’s.
Kuhn’s response is that it is almost impossible; we will always use arguments centered in our own standard of evidence to claim that our standards are good. This isn’t quite relativist, but it suggests that we cannot do much to compare standards because we would have to jump out of the system to do so. However, Kuhn did believe that better methodologies and standards of evidence have “more explaining power,” but he didn’t really quantify what this means.
Dewey’s response is a little more useful. He argued that we can compare the effectiveness of particular solutions, methodologies, and standards of evidence by looking at their ability to enhance our instrumental rationality. If one particular methodology does not give us as much instrumental rationality as another, the former is clearly inferior. (The advantage of this method is that it allows us to make a factual statement about methodologies, not just a normative one.)
In terms of deciding what’s worth pursuing, I will direct you to Larry Laudan and Irme Lakatos. They discussed the idea of competing “research programs” and “research traditions” in science and explored how we should handle this. David Hull also offers some interesting (and much more well-defined and practical) solutions. But this takes you deeper into philosophy of science and is a bit harder to understand.
I’m not sure if I can answer, but I can give historical background. This is by no means a new question—philosophy of science has been going at this for decades. What we are really talking about is comparing methodologies and standards of evidence. There are a lot of responses to this question, the most important probably being Thomas Kuhn’s and John Dewey’s.
Kuhn’s response is that it is almost impossible; we will always use arguments centered in our own standard of evidence to claim that our standards are good. This isn’t quite relativist, but it suggests that we cannot do much to compare standards because we would have to jump out of the system to do so. However, Kuhn did believe that better methodologies and standards of evidence have “more explaining power,” but he didn’t really quantify what this means.
Dewey’s response is a little more useful. He argued that we can compare the effectiveness of particular solutions, methodologies, and standards of evidence by looking at their ability to enhance our instrumental rationality. If one particular methodology does not give us as much instrumental rationality as another, the former is clearly inferior. (The advantage of this method is that it allows us to make a factual statement about methodologies, not just a normative one.)
In terms of deciding what’s worth pursuing, I will direct you to Larry Laudan and Irme Lakatos. They discussed the idea of competing “research programs” and “research traditions” in science and explored how we should handle this. David Hull also offers some interesting (and much more well-defined and practical) solutions. But this takes you deeper into philosophy of science and is a bit harder to understand.
(Edited for grammar and clarity)