Thanks for the thoughtful response. I agree that there are many instances in which it’s possible to rationally come to confident conclusions that differ from those of subject matter experts. I realize that my earlier comment was elliptical, and will try to clarify. The relevant points to my mind are:
The extraordinary intellectual caliber of the best physicists
Though difficult to formalize, I think that there’s a meaningful sense in which one can make statements of the general form “person A has intellectual caliber n times that of person B.” Of course, this is domain specific to some degree, but I think that the concept hangs together somewhat even across domains.
One operationalization of this is “if person B reaches a correct conclusion on a given subject, person A could reach it n times as fast.” Another is “it would take n copies of person B to do person A’s work.” These things are hard to estimate, but one can become better calibrated by using the rule “if person A has intellectual caliber n times that of person B and person B has intellectual caliber m times that of person C, then person A has intellectual caliber n*m times that of person C.”
In almost all domains, I think that the highest intellectual caliber people have no more than 5x my intellectual caliber. Physics is different. From what I’ve heard, the distribution of talent in physics is similar to that of math. The best mathematicians are 100x+ my intellectual caliber. I had a particularly striking illustrative experience with Don Zagier, who pinpointed a crucial weakness in an analogy that I had been exploring for 6 months (and which I had run by a number of other mathematicians) in a mere ~15 minutes. I would not be surprised if he himself were to have an analogous experience with the historical greats.
When someone is < 5x one’s intellectual caliber, an argument of the type “this person may be smarter than me, but I’ve focused a lot more on having accurate views, so I trust my judgment over him or her” seems reasonable. But when one gets to people who are 100x+ one’s intellectual caliber, the argument becomes much weaker. Model uncertainty starts to play a major role. It could be that people who are that much more powerful easily come to the correct conclusion on a given question without even needing to put conscious effort into having accurate beliefs.
The intrinsic interest of the question of interpretation of quantum mechanics
The question of what quantum mechanics means has been considered one of the universe’s great mysteries. As such, people interested in physics have been highly motivated to understand it. So I think that the question is privileged relative to other questions that physicists would have opinions on — it’s not an arbitrary question outside of the domain of their research accomplishments.
Solicitation of arguments from those with opposing views
In the Muslim theology example, you spend 40 hours engaging with the Muslim philosophers. This seems disanalogous to the present case, in that as far as I know, Eliezer’s quantum mechanics sequence hasn’t been vetted by any leading physicists who disagree with the many world’s interpretation of quantum mechanics. I also don’t know of any public record of ~40 hours of back and forth analogous to the one that you describe. I know that Eliezer might cite an example in his QM sequence, and will take a look.
The intrinsic interest of the question of interpretation of quantum mechanics
The question of what quantum mechanics means has been considered one of the universe’s great mysteries. As such, people interested in physics have been highly motivated to understand it. So I think that the question is privileged relative to other questions that physicists would have opinions on — it’s not an arbitrary question outside of the domain of their research accomplishments.
My understanding is that the interpretation of QM is (1) not regarded as a very central question in physics, being seen more as a “philosophy” question and being worked on to a reasonable extent by philosophers of physics and physicists who see it as a hobby horse, (2) is not something that physics expertise—having good physical intuition, strong math skills, detailed knowledge of how to apply QM on concrete problems—is as relevant for as many other questions physicists work on, and (3) is not something about which there is an extremely enormous amount to say. These are some of the main reasons I feel I can update at all from the expert distribution of physicists on this question. I would hardly update at all from physicist opinions on, say, quantum gravity vs. string theory, and I think it would basically be crazy for me to update substantially in one direction or the other if I had comparable experience on that question.
[ETA: As evidence of (1), I might point to the prevalence of the “shut up and calculate” mentality which seems to have been reasonably popular in physics for a while. I’d also point to the fact that Copenhagen is popular but really, really, really, really not good. And I feel that this last claim is not just Nick Beckstead’s idiosyncratic opinion, but the opinion of every philosopher of physics I have ever spoken with about this issue.]
I believe you are using bad terminology. ‘Quantum gravity’ refers to any attempt to reconcile quantum mechanics and general relativity, and string theory is one such theory (as well as a theory of everything). Perhaps you are referring to loop quantum gravity, or more broadly, to any theory of quantum gravity other than string theory?
Perhaps I should have meant loop quantum gravity. I confess that I am speaking beyond my depth, and was just trying to give an example of a central dispute in current theoretical physics. That is the type of case where I would not like to lean heavily on my own perspective.
My understanding is that the interpretation of QM is (1) not regarded as a very central question in physics, being seen more as a “philosophy” question and being worked on to a reasonable extent by philosophers of physics and physicists who see it as a hobby horse,
I agree if we’re talking about the median theoretical physicist at a top 5 school, but when one gets further toward the top of the hierarchy, one starts to see a high density of people who are all-around intellectually curious and who explore natural questions that they come across independently of whether they’re part of their official research.
(2) is not something that physics expertise—having good physical intuition, strong math skills, detailed knowledge of how to apply QM on concrete problems—is as relevant for as many other questions physicists work on
I agree, but a priori I suspect that philosophers of physics and others without heavy subject matter knowledge of quantum mechanics have leaned too heavily on this. Spending one’s life thinking about something can result subconscious acquisition of implicit knowledge of things that are obliquely related. People who haven’t had this experience may be at a disadvantage.
I would hardly update at all from physicist opinions on, say, quantum gravity vs. string theory, and I think it would basically be crazy for me to update substantially in one direction or the other if I had comparable experience on that question.
I actually think that it’s possible for somebody without subject matter knowledge to rationally develop priors that are substantially different from expert consensus here. One can do this by consulting physicists who visibly have high epistemic rationality outside of physics, by examining sociological factors that may have led to the status quo, and by watching physicists who disagree debate each other and see which of the points they respond to and which ones they don’t.
[ETA: As evidence of (1), I might point to the prevalence of the “shut up and calculate” mentality which seems to have been reasonably popular in physics for a while. I’d also point to the fact that Copenhagen is popular but really, really, really, really not good. And I feel that this last claim is not just Nick Beckstead’s idiosyncratic opinion, but the opinion of every philosopher of physics I have ever spoken with about this issue.]
I agree, but a priori I suspect that philosophers of physics and others without heavy subject matter knowledge of quantum mechanics have leaned too heavily on this. Spending one’s life thinking about something can result subconscious acquisition of implicit knowledge of things that are obliquely related. People who haven’t had this experience may be at a disadvantage.
But note that philosophers of physics sometimes make whole careers thinking about this, and they are among the most high-caliber philosophers. They may be at an advantage in terms of this criterion.
I can’t think of a reference in print for my claim about what almost all philosophers think. I think a lot of them would find it too obvious to say, and wouldn’t bother to write a paper about it. But, for what it’s worth, I attended a couple of conferences on philosophy of physics held at Rutgers, with many leading people in the field, and talked about this question and never heard anyone express an opposing opinion. And I was taught about interpretations of QM from some leading people in philosophy of physics.
What I’m anchoring on here is the situation in the field of philosophy of math, where lack of experience with the practice of math seriously undercuts most philosophers’ ability to do it well. There are exceptions, for example I consider Imre Lakatos to be one. Maybe the situation is different in philosophy of physics.
I agree, but a priori I suspect that philosophers of physics and others without heavy subject matter knowledge of quantum mechanics have leaned too heavily on this. Spending one’s life thinking about something can result subconscious acquisition of implicit knowledge of things that are obliquely related. People who haven’t had this experience may be at a disadvantage.
But note that philosophers of physics sometimes make whole careers thinking about this, and they are among the most high-caliber philosophers. They may be at an advantage in terms of this criterion.
I can’t think of a reference in print for my claim about what almost all philosophers think. I think a lot of them would find it too obvious to say, and wouldn’t bother to write a paper about it. But, for what it’s worth, I attended a couple of conferences on philosophy of physics held at Rutgers, with many leading people in the field, and talked about this question and never heard anyone express an opposing opinion. And I was taught about interpretations of QM from some leading people in philosophy of physics.
Let me first say that I find this to be an extremely interesting discussion.
In almost all domains, I think that the highest intellectual caliber people have no more than 5x my intellectual caliber. Physics is different. From what I’ve heard, the distribution of talent in physics is similar to that of math. The best mathematicians are 100x+ my intellectual caliber.
I think there is a social norm in mathematics and physics that requires people to say this, but I have serious doubts about whether it is true. Anyone 100x+ your intellectual caliber should be having much, much more impact on the world (to say nothing of mathematics itself) than any of the best mathematicians seem to be having. At the very least, if there really are people of that cognitive level running around, then the rest of the world is doing an absolutely terrible job of extracting information and value from them, and they themselves must not care too much about this fact.
More plausible to me is the hypothesis that the best mathematicians are within the same 5x limit as everyone else, and that you overestimate the difficulty of performing at their level due to cultural factors which discourage systematic study of how to imitate them.
Try this thought experiment: suppose you were a graduate student in mathematics, and went to your advisor and said: “I’d like to solve [Famous Problem X], and to start, I’m going to spend two years closely examining the work of Newton, Gauss, and Wiles, and their contemporaries, to try to discern at a higher level of generality what the cognitive stumbling blocks to solving previous problems were, and how they overcame them, and distill these meta-level insights into a meta-level technique of my own which I’ll then apply to [Famous Problem X].” What do you think the reaction would be? How many times do you think such a thing has ever been proposed, let alone attempted, by a serious student or (even) senior researcher?
Try this thought experiment: suppose you were a graduate student in mathematics, and went to your advisor and said: “I’d like to solve [Famous Problem X], and to start, I’m going to spend two years closely examining the work of Newton, Gauss, and Wiles, and their contemporaries, to try to discern at a higher level of generality what the cognitive stumbling blocks to solving previous problems were, and how they overcame them, and distill these meta-level insights into a meta-level technique of my own which I’ll then apply to [Famous Problem X].”
This is a terrible idea unless they’re spending half their time pushing their limits on object-level math problems. I just don’t think it works to try to do a meta phase before an object phase unless the process is very, very well-understood and tested already.
I’m sure that’s exactly what the advisor would say (if they bother to give a reasoned reply at all), with the result that nobody ever tries this.
(I’ll also note that it’s somewhat odd to hear this response from someone whose entire mission in life is essentially to go meta on all of humanity’s problems...)
But let me address the point, so as not to be logically rude. The person would be pushing their limits on object-level math problems in the course of “examining the work of Newton, Gauss, and Wiles”, in order to understand said work; otherwise, it can hardly be said to constitute a meaningful examination. I also think it’s important not to confuse meta-ness with (nontechnical) “outside views”; indeed I suspect that a lot of the thought processes of mathematical “geniuses” consist of abstracting over classes of technical concepts that aren’t ordinarily abstracted over, and thus if expressed explicitly (which the geniuses may lack the patience to do) would simply look like another form of mathematics. (Others of their processes, I speculate, consist in obsessive exercising of visual/dynamic mental models of various abstractions.)
Switching back to logical rudeness, I’m not sure the meta-ness is your true rejection; I suspect what you may be really worried about is making sure there are tight feedback loops to which one’s reasoning can be subjected.
(I’ll also note that it’s somewhat odd to hear this response from someone whose entire mission in life is essentially to go meta on all of humanity’s problems...)
That’s not the kind of meta I mean. The dangerous form of meta is when you spend several years preparing to do X, supposedly becoming better at doing X, but not actually doing X, and then try to do X. E.g. college. Trying to improve at doing X while doing X is much, much wiser. I would similarly advise Effective Altruists who are not literally broke to be donating $10 every three months to something while they are trying to increase their incomes and invest in human capital; furthermore, they should not donate to the same thing two seasons in a row, so that they are also practicing the skill of repeatedly assessing which charity is most important.
“Meta” for these purposes is any daily activity which is unlike the daily activity you intend to do ‘later’.
Tight feedback loops are good, but not always available. This is a separate consideration from doing meta while doing object.
The activity of understanding someone else’s proofs may be unlike the activity of producing your own new math from scratch; this would be the problem.
I would similarly advise Effective Altruists who are not literally broke to be donating $10 every three months to something while they are trying to increase their incomes and invest in human capital; furthermore, they should not donate to the same thing two seasons in a row, so that they are also practicing the skill of repeatedly assessing which charity is most important.
This is excellent advice. I have put a note in my calendar thee months hence to reevaluate my small monthly donation.
At the very least, if there really are people of that cognitive level running around, then the rest of the world is doing an absolutely terrible job of extracting information and value from them, and they themselves must not care too much about this fact.
Yes, I think that this is what the situation is.
I’ll also say that I think that there are very few such people — maybe on the order of 10 who are alive today. With such a small absolute number, I don’t think that their observed impact on math is a lot lower than what one would expect a priori, and the prior in favor them having had a huge impact in society isn’t that strong.
More plausible to me is the hypothesis that the best mathematicians are within the same 5x limit as everyone else, and that you overestimate the difficulty of performing at their level due to cultural factors which discourage systematic study of how to imitate them.
“The best mathematicians are 100+x higher in intellectual caliber than I am” and “the difference is in large part due to cultural factors which discourage systematic study of how to imitate them” aren’t mutually exclusive. I’m sympathetic to your position.
What do you think the reaction would be?
To change the subject :-)
How many times do you think such a thing has ever been proposed, let alone attempted, by a serious student or (even) senior researcher?
Anyone 100x+ your intellectual caliber should be having much, much more impact on the world (to say nothing of mathematics itself) than any of the best mathematicians seem to be having.
How do you know how little intellectual caliber JonahSinick has?
The extraordinary intellectual caliber of the best physicists
That is of course exactly why I picked QM and MWI to make my case for nihil supernum. It wouldn’t serve to break a smart person’s trust in a sane world if I demonstrated the insanity of Muslim theologians or politicians; they would just say, “But surely we should still trust in elite physicists.” It is by demonstrating that trust in a sane world fails even at the strongest point which ‘elite common sense’ would expect to find, that I would hope to actually break someone’s emotional trust, and cause them to just give up.
I haven’t fully put together my thoughts on this, but it seems like a bad test to “break someone’s trust in a sane world” for a number of reasons:
this is a case where all the views are pretty much empirically indistinguishable, so it isn’t an area where physicists really care all that much
since the views are empirically indistinguishable, it is probably a low-stakes question, so the argument doesn’t transfer well to breaking our trust in a sane world in high-stakes cases; it makes sense to assume people would apply more rationality in cases where more rationality pays off
as I said in another comment, MWI seems like a case where physics expertise is not really what matters, so this doesn’t really show that the scientific method as applied by physicists is broken; it seems it at most it shows that physics aren’t good at questions that are essentially philosophical; it would be much more persuasive if you showed that e.g., quantum gravity was obviously better than string theory and only 18% of physicists working in the relevant area thought so
From my perspective, the main point is that if you’d expect AI elites to handle FAI competently, you would expect physics elites to handle MWI competently—the risk factors in the former case are even greater. Requires some philosophical reasoning? Check. Reality does not immediately call you out on being wrong? Check. The AI problem is harder than MWI and it has additional risk factors on top of that, like losing your chance at tenure if you decide that your research actually needs to slow down. Any elite incompetence beyond the demonstrated level in MWI doesn’t really matter much to me, since we’re already way under the ‘pass’ threshold for FAI.
I feel this doesn’t address the “low stakes” issues I brought up, or that this may not even by the physicists’ area of competence. Maybe you’d get a different outcome if the fate of the world depended on this issue, as you believe it does with AI.
I also wonder if this analysis leads to wrong historical predictions. E.g., why doesn’t this reasoning suggest that the US government would totally botch the constitution? That requires philosophical reasoning and reality doesn’t immediately call you out on being wrong. And the people setting things up don’t have incentives totally properly aligned. Setting up a decent system of government strikes me as more challenging than the MWI problem in many respects.
How much weight do you actually put on this line of argument? Would you change your mind about anything practical if you found out you were wrong about MWI?
I have an overall sense that there are a lot of governments that are pretty good and that people are getting better at setting up governments over time. The question is very vague and hard to answer, so I am not going to attempt a detailed one. Perhaps you could give it a shot if you’re interested.
I agree that if it were true that the consensus of elite physicists believed that MWI is wrong when there was a decisive case in favor of it, that would be striking. But
There doesn’t seem to be a consensus among elite physicists that MWI is wrong.
Paging through your QM sequence, it doesn’t look as though you’ve systematically addressed all objections that otherwise credible people have raised against MWI. For example, have you been through all of the references that critique MWI cited in this paper? I think given that most experts don’t view the matter as decided, and given the intellectual caliber of the experts, in order have 99+% confidence in this setting, one has to cover all of one’s bases.
One will generally find that correct controversial ideas convince some physicists. There are many physicists who believe MWI (though they perhaps cannot get away with advocating it as rudely as I do), there are physicists signed up for cryonics, there were physicists advocating for Drexler’s molecular nanotechnology before it was cool, and I strongly expect that some physicists who read econblogs have by now started advocating market monetarism (if not I would update against MM). A good new idea should have some physicists in favor of it, and if not it is a warning sign. (Though the endorsement of some physicists is not a proof, obviously many bad ideas can convince a few physicists too.) If I could not convince any physicists of my views on FAI, that would be a grave warning sign indeed. (I’m pretty sure some such already exist.) But that a majority of physicists do not yet believe in MWI does not say very much one way or another.
The cognitive elites do exist and some of them are physicists, therefore you should be able to convince some physicists. But the cognitive elites don’t correspond to a majority of MIT professors or anything like that, so you shouldn’t be able to convince a majority of that particular group. A world which knew what its own elite truthfinders looked like would be a very different world from this one.
Ok, putting aside MWI, maybe our positions are significantly more similar than it initially seemed. I agree with
A world which knew what its own elite truthfinders looked like would be a very different world from this one.
I’ve taken your comments such as
Depends on how crazy the domain experts are being, in this mad world of ours.
to carry connotations of the type “the fraction of people who exhibit high epistemic rationality outside of their immediate areas of expertise is vanishingly small.”
I think that there are thousands of people worldwide who exhibit very high epistemic rationality in most domains that they think about. I think that most of these people are invisible owing to the paucity of elites online. I agree that epistemic standards are generally very poor, and that high status academics generally do poorly outside of their immediate areas of expertise.
I think that there are thousands of people worldwide who exhibit very high epistemic rationality in most domains that they think about. I think that most of these people are invisible owing to the paucity of elites online.
Where does this impression come from? Are they people you’ve encountered personally? If so, what gave you the impression that they exhibited “very high epistemic rationality in most domains that they think about”?
To clarify, when I wrote “very high epistemic rationality” I didn’t mean “very accurate beliefs,” but rather “aware of what they know and what they don’t.” I also see the qualifier “most” as significant — I think that any given person has some marked blind spots. Of course, the boundary that I’m using is fuzzy, but the standard that I have in mind is something like “at the level of the 15 most epistemically rational members of the LW community.”
“Thousands” is at the “1 in a million” level, so in relative terms, my claim is pretty weak. If one disputes the claim, one needs a story explaining how the fraction could be so small. It doesn’t suffice to say “I haven’t personally seen almost any such people,” because there are so many people who one hasn’t seen, and the relevant people may be in unexpected places.
I’ve had the subjective impression that ~ 2% of those who I know outside of the LW/EA spheres fit this description. To be sure, there’s a selection effect, but I’ve had essentially no exposure to physics, business, finance or public policy, nor to people in very populous countries such as India and China. The people who I know who fit this description don’t seem to think that they’re extremely rare, which suggests that their experiences are similar to my own (though I recognize that “it’s a small world,” i.e. these people’s social circles may overlap in nonobvious ways).
Some of the people who GiveWell has had conversations with come across very favorably in the notes. (I recognize that I’m making a jump from “their area of specialty” to “most topics that they think about” Here I’m extrapolating from the people who I know who are very strong in their area of specialty.) I think that Carl’s comment about the Gates Foundation is relevant.
I updated in the direction of people being more rational than I had thought for reason that I gave at the end of my post Many weak arguments and the typical mind.
I don’t have high confidence here: maybe ~ 50% (i.e. I’m giving a median case estimate).
I should also clarify that I don’t think that one needs a silver bullet argument of the type “the people who you would expect to be most trustworthy have the wrong belief on something that they’ve thought about, with very high probability” to conclude with high confidence that epistemic standards are generally very low.
I think that there are many weak arguments that respected authorities are very often wrong.
Vladimir M has made arguments of the type “there’s fierce disagreement among experts at X about matters pertaining to X, so one knows that at least some of them are very wrong.” I think that string theory is a good case study. There are very smart people who strongly advocate for string theory as a promising road for theoretical physics research, and other very smart people who strongly believe that the research program is misguided. If nothing else, one can tell that a lot of the actors involved are very overconfident (even if one doesn’t know who they are).
There are very smart people who strongly advocate for string theory as a promising road for theoretical physics research, and other very smart people who strongly believe that the research program is misguided. If nothing else, one can tell that a lot of the actors involved are very overconfident (even if one doesn’t know who they are).
Or, alternatively, they disagree about who the research program is promising for.
who disagree with the many world’s interpretation of quantum mechanics
That’s too strong of a statement. If you exclude some diehards like Deutsch, the Bohmian school and maybe Penrose’s gravity-induced decoherence, the prevailing attitude is “however many worlds are out there, we can only ever see one”, so, until new evidence comes along, we can safely treat MWI as a single world.
In computer science an elite coder might take 6 months to finish a very hard task (e.g. create some kind of tricky OS kernel), but a poor coder will never complete the task. This makes the elite coder infinitely better than the poor coder. Furthermore the poor coder will ask many questions of other people, impacting their productivity. Thus an elite coder is transfinitely more efficient than a poor coder ;-)
Thanks for the thoughtful response. I agree that there are many instances in which it’s possible to rationally come to confident conclusions that differ from those of subject matter experts. I realize that my earlier comment was elliptical, and will try to clarify. The relevant points to my mind are:
The extraordinary intellectual caliber of the best physicists
Though difficult to formalize, I think that there’s a meaningful sense in which one can make statements of the general form “person A has intellectual caliber n times that of person B.” Of course, this is domain specific to some degree, but I think that the concept hangs together somewhat even across domains.
One operationalization of this is “if person B reaches a correct conclusion on a given subject, person A could reach it n times as fast.” Another is “it would take n copies of person B to do person A’s work.” These things are hard to estimate, but one can become better calibrated by using the rule “if person A has intellectual caliber n times that of person B and person B has intellectual caliber m times that of person C, then person A has intellectual caliber n*m times that of person C.”
In almost all domains, I think that the highest intellectual caliber people have no more than 5x my intellectual caliber. Physics is different. From what I’ve heard, the distribution of talent in physics is similar to that of math. The best mathematicians are 100x+ my intellectual caliber. I had a particularly striking illustrative experience with Don Zagier, who pinpointed a crucial weakness in an analogy that I had been exploring for 6 months (and which I had run by a number of other mathematicians) in a mere ~15 minutes. I would not be surprised if he himself were to have an analogous experience with the historical greats.
When someone is < 5x one’s intellectual caliber, an argument of the type “this person may be smarter than me, but I’ve focused a lot more on having accurate views, so I trust my judgment over him or her” seems reasonable. But when one gets to people who are 100x+ one’s intellectual caliber, the argument becomes much weaker. Model uncertainty starts to play a major role. It could be that people who are that much more powerful easily come to the correct conclusion on a given question without even needing to put conscious effort into having accurate beliefs.
The intrinsic interest of the question of interpretation of quantum mechanics
The question of what quantum mechanics means has been considered one of the universe’s great mysteries. As such, people interested in physics have been highly motivated to understand it. So I think that the question is privileged relative to other questions that physicists would have opinions on — it’s not an arbitrary question outside of the domain of their research accomplishments.
Solicitation of arguments from those with opposing views
In the Muslim theology example, you spend 40 hours engaging with the Muslim philosophers. This seems disanalogous to the present case, in that as far as I know, Eliezer’s quantum mechanics sequence hasn’t been vetted by any leading physicists who disagree with the many world’s interpretation of quantum mechanics. I also don’t know of any public record of ~40 hours of back and forth analogous to the one that you describe. I know that Eliezer might cite an example in his QM sequence, and will take a look.
My understanding is that the interpretation of QM is (1) not regarded as a very central question in physics, being seen more as a “philosophy” question and being worked on to a reasonable extent by philosophers of physics and physicists who see it as a hobby horse, (2) is not something that physics expertise—having good physical intuition, strong math skills, detailed knowledge of how to apply QM on concrete problems—is as relevant for as many other questions physicists work on, and (3) is not something about which there is an extremely enormous amount to say. These are some of the main reasons I feel I can update at all from the expert distribution of physicists on this question. I would hardly update at all from physicist opinions on, say, quantum gravity vs. string theory, and I think it would basically be crazy for me to update substantially in one direction or the other if I had comparable experience on that question.
[ETA: As evidence of (1), I might point to the prevalence of the “shut up and calculate” mentality which seems to have been reasonably popular in physics for a while. I’d also point to the fact that Copenhagen is popular but really, really, really, really not good. And I feel that this last claim is not just Nick Beckstead’s idiosyncratic opinion, but the opinion of every philosopher of physics I have ever spoken with about this issue.]
A minor quibble.
I believe you are using bad terminology. ‘Quantum gravity’ refers to any attempt to reconcile quantum mechanics and general relativity, and string theory is one such theory (as well as a theory of everything). Perhaps you are referring to loop quantum gravity, or more broadly, to any theory of quantum gravity other than string theory?
Perhaps I should have meant loop quantum gravity. I confess that I am speaking beyond my depth, and was just trying to give an example of a central dispute in current theoretical physics. That is the type of case where I would not like to lean heavily on my own perspective.
I agree if we’re talking about the median theoretical physicist at a top 5 school, but when one gets further toward the top of the hierarchy, one starts to see a high density of people who are all-around intellectually curious and who explore natural questions that they come across independently of whether they’re part of their official research.
I agree, but a priori I suspect that philosophers of physics and others without heavy subject matter knowledge of quantum mechanics have leaned too heavily on this. Spending one’s life thinking about something can result subconscious acquisition of implicit knowledge of things that are obliquely related. People who haven’t had this experience may be at a disadvantage.
I actually think that it’s possible for somebody without subject matter knowledge to rationally develop priors that are substantially different from expert consensus here. One can do this by consulting physicists who visibly have high epistemic rationality outside of physics, by examining sociological factors that may have led to the status quo, and by watching physicists who disagree debate each other and see which of the points they respond to and which ones they don’t.
Can you give a reference?
But note that philosophers of physics sometimes make whole careers thinking about this, and they are among the most high-caliber philosophers. They may be at an advantage in terms of this criterion.
I can’t think of a reference in print for my claim about what almost all philosophers think. I think a lot of them would find it too obvious to say, and wouldn’t bother to write a paper about it. But, for what it’s worth, I attended a couple of conferences on philosophy of physics held at Rutgers, with many leading people in the field, and talked about this question and never heard anyone express an opposing opinion. And I was taught about interpretations of QM from some leading people in philosophy of physics.
What I’m anchoring on here is the situation in the field of philosophy of math, where lack of experience with the practice of math seriously undercuts most philosophers’ ability to do it well. There are exceptions, for example I consider Imre Lakatos to be one. Maybe the situation is different in philosophy of physics.
But note that philosophers of physics sometimes make whole careers thinking about this, and they are among the most high-caliber philosophers. They may be at an advantage in terms of this criterion.
I can’t think of a reference in print for my claim about what almost all philosophers think. I think a lot of them would find it too obvious to say, and wouldn’t bother to write a paper about it. But, for what it’s worth, I attended a couple of conferences on philosophy of physics held at Rutgers, with many leading people in the field, and talked about this question and never heard anyone express an opposing opinion. And I was taught about interpretations of QM from some leading people in philosophy of physics.
Er, http://lesswrong.com/lw/r8/and_the_winner_is_manyworlds/ maybe.
Let me first say that I find this to be an extremely interesting discussion.
I think there is a social norm in mathematics and physics that requires people to say this, but I have serious doubts about whether it is true. Anyone 100x+ your intellectual caliber should be having much, much more impact on the world (to say nothing of mathematics itself) than any of the best mathematicians seem to be having. At the very least, if there really are people of that cognitive level running around, then the rest of the world is doing an absolutely terrible job of extracting information and value from them, and they themselves must not care too much about this fact.
More plausible to me is the hypothesis that the best mathematicians are within the same 5x limit as everyone else, and that you overestimate the difficulty of performing at their level due to cultural factors which discourage systematic study of how to imitate them.
Try this thought experiment: suppose you were a graduate student in mathematics, and went to your advisor and said: “I’d like to solve [Famous Problem X], and to start, I’m going to spend two years closely examining the work of Newton, Gauss, and Wiles, and their contemporaries, to try to discern at a higher level of generality what the cognitive stumbling blocks to solving previous problems were, and how they overcame them, and distill these meta-level insights into a meta-level technique of my own which I’ll then apply to [Famous Problem X].” What do you think the reaction would be? How many times do you think such a thing has ever been proposed, let alone attempted, by a serious student or (even) senior researcher?
This is a terrible idea unless they’re spending half their time pushing their limits on object-level math problems. I just don’t think it works to try to do a meta phase before an object phase unless the process is very, very well-understood and tested already.
I’m sure that’s exactly what the advisor would say (if they bother to give a reasoned reply at all), with the result that nobody ever tries this.
(I’ll also note that it’s somewhat odd to hear this response from someone whose entire mission in life is essentially to go meta on all of humanity’s problems...)
But let me address the point, so as not to be logically rude. The person would be pushing their limits on object-level math problems in the course of “examining the work of Newton, Gauss, and Wiles”, in order to understand said work; otherwise, it can hardly be said to constitute a meaningful examination. I also think it’s important not to confuse meta-ness with (nontechnical) “outside views”; indeed I suspect that a lot of the thought processes of mathematical “geniuses” consist of abstracting over classes of technical concepts that aren’t ordinarily abstracted over, and thus if expressed explicitly (which the geniuses may lack the patience to do) would simply look like another form of mathematics. (Others of their processes, I speculate, consist in obsessive exercising of visual/dynamic mental models of various abstractions.)
Switching back to logical rudeness, I’m not sure the meta-ness is your true rejection; I suspect what you may be really worried about is making sure there are tight feedback loops to which one’s reasoning can be subjected.
That’s not the kind of meta I mean. The dangerous form of meta is when you spend several years preparing to do X, supposedly becoming better at doing X, but not actually doing X, and then try to do X. E.g. college. Trying to improve at doing X while doing X is much, much wiser. I would similarly advise Effective Altruists who are not literally broke to be donating $10 every three months to something while they are trying to increase their incomes and invest in human capital; furthermore, they should not donate to the same thing two seasons in a row, so that they are also practicing the skill of repeatedly assessing which charity is most important.
“Meta” for these purposes is any daily activity which is unlike the daily activity you intend to do ‘later’.
Tight feedback loops are good, but not always available. This is a separate consideration from doing meta while doing object.
The activity of understanding someone else’s proofs may be unlike the activity of producing your own new math from scratch; this would be the problem.
This is excellent advice. I have put a note in my calendar thee months hence to reevaluate my small monthly donation.
Nice to hear from you :-)
Yes, I think that this is what the situation is.
I’ll also say that I think that there are very few such people — maybe on the order of 10 who are alive today. With such a small absolute number, I don’t think that their observed impact on math is a lot lower than what one would expect a priori, and the prior in favor them having had a huge impact in society isn’t that strong.
“The best mathematicians are 100+x higher in intellectual caliber than I am” and “the difference is in large part due to cultural factors which discourage systematic study of how to imitate them” aren’t mutually exclusive. I’m sympathetic to your position.
To change the subject :-)
Basically never.
Not to mention that some of them might be working on Wall Street or something, and not have worked on unsolved problems in mathematics in decades.
How do you know how little intellectual caliber JonahSinick has?
looks at JonahSinick’s profile
follows link to his website
skims the “About me” page
Yes, you have a point.
That is of course exactly why I picked QM and MWI to make my case for nihil supernum. It wouldn’t serve to break a smart person’s trust in a sane world if I demonstrated the insanity of Muslim theologians or politicians; they would just say, “But surely we should still trust in elite physicists.” It is by demonstrating that trust in a sane world fails even at the strongest point which ‘elite common sense’ would expect to find, that I would hope to actually break someone’s emotional trust, and cause them to just give up.
I haven’t fully put together my thoughts on this, but it seems like a bad test to “break someone’s trust in a sane world” for a number of reasons:
this is a case where all the views are pretty much empirically indistinguishable, so it isn’t an area where physicists really care all that much
since the views are empirically indistinguishable, it is probably a low-stakes question, so the argument doesn’t transfer well to breaking our trust in a sane world in high-stakes cases; it makes sense to assume people would apply more rationality in cases where more rationality pays off
as I said in another comment, MWI seems like a case where physics expertise is not really what matters, so this doesn’t really show that the scientific method as applied by physicists is broken; it seems it at most it shows that physics aren’t good at questions that are essentially philosophical; it would be much more persuasive if you showed that e.g., quantum gravity was obviously better than string theory and only 18% of physicists working in the relevant area thought so
[Edited to add a missing “not”]
From my perspective, the main point is that if you’d expect AI elites to handle FAI competently, you would expect physics elites to handle MWI competently—the risk factors in the former case are even greater. Requires some philosophical reasoning? Check. Reality does not immediately call you out on being wrong? Check. The AI problem is harder than MWI and it has additional risk factors on top of that, like losing your chance at tenure if you decide that your research actually needs to slow down. Any elite incompetence beyond the demonstrated level in MWI doesn’t really matter much to me, since we’re already way under the ‘pass’ threshold for FAI.
I feel this doesn’t address the “low stakes” issues I brought up, or that this may not even by the physicists’ area of competence. Maybe you’d get a different outcome if the fate of the world depended on this issue, as you believe it does with AI.
I also wonder if this analysis leads to wrong historical predictions. E.g., why doesn’t this reasoning suggest that the US government would totally botch the constitution? That requires philosophical reasoning and reality doesn’t immediately call you out on being wrong. And the people setting things up don’t have incentives totally properly aligned. Setting up a decent system of government strikes me as more challenging than the MWI problem in many respects.
How much weight do you actually put on this line of argument? Would you change your mind about anything practical if you found out you were wrong about MWI?
What different evidence would you expect to observe in a world where amateur attempts to set up systems of government were usually botched?
(Edit: reworded for (hopefully) clarity.)
I have an overall sense that there are a lot of governments that are pretty good and that people are getting better at setting up governments over time. The question is very vague and hard to answer, so I am not going to attempt a detailed one. Perhaps you could give it a shot if you’re interested.
You meant “is not really”?
Yes, thank you for catching.
I agree that if it were true that the consensus of elite physicists believed that MWI is wrong when there was a decisive case in favor of it, that would be striking. But
There doesn’t seem to be a consensus among elite physicists that MWI is wrong.
Paging through your QM sequence, it doesn’t look as though you’ve systematically addressed all objections that otherwise credible people have raised against MWI. For example, have you been through all of the references that critique MWI cited in this paper? I think given that most experts don’t view the matter as decided, and given the intellectual caliber of the experts, in order have 99+% confidence in this setting, one has to cover all of one’s bases.
One will generally find that correct controversial ideas convince some physicists. There are many physicists who believe MWI (though they perhaps cannot get away with advocating it as rudely as I do), there are physicists signed up for cryonics, there were physicists advocating for Drexler’s molecular nanotechnology before it was cool, and I strongly expect that some physicists who read econblogs have by now started advocating market monetarism (if not I would update against MM). A good new idea should have some physicists in favor of it, and if not it is a warning sign. (Though the endorsement of some physicists is not a proof, obviously many bad ideas can convince a few physicists too.) If I could not convince any physicists of my views on FAI, that would be a grave warning sign indeed. (I’m pretty sure some such already exist.) But that a majority of physicists do not yet believe in MWI does not say very much one way or another.
The cognitive elites do exist and some of them are physicists, therefore you should be able to convince some physicists. But the cognitive elites don’t correspond to a majority of MIT professors or anything like that, so you shouldn’t be able to convince a majority of that particular group. A world which knew what its own elite truthfinders looked like would be a very different world from this one.
Ok, putting aside MWI, maybe our positions are significantly more similar than it initially seemed. I agree with
I’ve taken your comments such as
to carry connotations of the type “the fraction of people who exhibit high epistemic rationality outside of their immediate areas of expertise is vanishingly small.”
I think that there are thousands of people worldwide who exhibit very high epistemic rationality in most domains that they think about. I think that most of these people are invisible owing to the paucity of elites online. I agree that epistemic standards are generally very poor, and that high status academics generally do poorly outside of their immediate areas of expertise.
Where does this impression come from? Are they people you’ve encountered personally? If so, what gave you the impression that they exhibited “very high epistemic rationality in most domains that they think about”?
To clarify, when I wrote “very high epistemic rationality” I didn’t mean “very accurate beliefs,” but rather “aware of what they know and what they don’t.” I also see the qualifier “most” as significant — I think that any given person has some marked blind spots. Of course, the boundary that I’m using is fuzzy, but the standard that I have in mind is something like “at the level of the 15 most epistemically rational members of the LW community.”
“Thousands” is at the “1 in a million” level, so in relative terms, my claim is pretty weak. If one disputes the claim, one needs a story explaining how the fraction could be so small. It doesn’t suffice to say “I haven’t personally seen almost any such people,” because there are so many people who one hasn’t seen, and the relevant people may be in unexpected places.
I’ve had the subjective impression that ~ 2% of those who I know outside of the LW/EA spheres fit this description. To be sure, there’s a selection effect, but I’ve had essentially no exposure to physics, business, finance or public policy, nor to people in very populous countries such as India and China. The people who I know who fit this description don’t seem to think that they’re extremely rare, which suggests that their experiences are similar to my own (though I recognize that “it’s a small world,” i.e. these people’s social circles may overlap in nonobvious ways).
Some of the people who GiveWell has had conversations with come across very favorably in the notes. (I recognize that I’m making a jump from “their area of specialty” to “most topics that they think about” Here I’m extrapolating from the people who I know who are very strong in their area of specialty.) I think that Carl’s comment about the Gates Foundation is relevant.
I updated in the direction of people being more rational than I had thought for reason that I gave at the end of my post Many weak arguments and the typical mind.
I don’t have high confidence here: maybe ~ 50% (i.e. I’m giving a median case estimate).
I should also clarify that I don’t think that one needs a silver bullet argument of the type “the people who you would expect to be most trustworthy have the wrong belief on something that they’ve thought about, with very high probability” to conclude with high confidence that epistemic standards are generally very low.
I think that there are many weak arguments that respected authorities are very often wrong.
Vladimir M has made arguments of the type “there’s fierce disagreement among experts at X about matters pertaining to X, so one knows that at least some of them are very wrong.” I think that string theory is a good case study. There are very smart people who strongly advocate for string theory as a promising road for theoretical physics research, and other very smart people who strongly believe that the research program is misguided. If nothing else, one can tell that a lot of the actors involved are very overconfident (even if one doesn’t know who they are).
Or, alternatively, they disagree about who the research program is promising for.
That’s too strong of a statement. If you exclude some diehards like Deutsch, the Bohmian school and maybe Penrose’s gravity-induced decoherence, the prevailing attitude is “however many worlds are out there, we can only ever see one”, so, until new evidence comes along, we can safely treat MWI as a single world.
In computer science an elite coder might take 6 months to finish a very hard task (e.g. create some kind of tricky OS kernel), but a poor coder will never complete the task. This makes the elite coder infinitely better than the poor coder. Furthermore the poor coder will ask many questions of other people, impacting their productivity. Thus an elite coder is transfinitely more efficient than a poor coder ;-)
See also Stephen Hsu’s comments on this.