I agree with your conclusion, that the important takeaway is to build models of whom to trust when and on what matters.
Nonetheless, I disagree that it requires as much work to decide to trust the academic field of math as to trust MIRI. Whenever you’re using the outside view, you need to define a reference class. I’ve never come across this used as an objection to the outside view. That’s probably because there often is one such class more salient than others: “people who have social status within field X”. After all, one of the key explanations for the evolution of human intelligence is that of an arms race in social cognition. For example, you see this in studies where people are clueless at solving logic problems, unless you phrase them in terms of detecting cheaters or other breaches of social contracts (see e.g. Cheng & Holyoak, 1985 and Gigerenzer & Hug, 1992). So we should expect humans to easily figure out who has status within a field, but to have a very hard time figuring out who gets the field closer to truth.
Isn’t this exactly why modesty is such an appealing and powerful view in the first place? Because choosing the reference class is so easy (not requiring much object-level investigation), and experts are correct sufficiently often, that any inside view is mistaken in expectation.
Right. I will say that the heuristic of trusting “people who have social status within field X” seems to have very obvious flaws and biases, and it’s important to build models to work past those mistakes. Furthermore, in the world around us, most high status people in institutions are messing up most of the time. Bank of Japan is fine example, but also the people running my prestigious high school, my university, researchers in my field, etc. It’s important to learn where they’re inadequate so you can find extra value.
I’m not saying that in general “trust the high status people” isn’t probably okay if you have zero further info, but you definitely need to start building detailed models and get better than that otherwise you’re definitely not going to save the world.
I agree with your conclusion, that the important takeaway is to build models of whom to trust when and on what matters.
Nonetheless, I disagree that it requires as much work to decide to trust the academic field of math as to trust MIRI. Whenever you’re using the outside view, you need to define a reference class. I’ve never come across this used as an objection to the outside view. That’s probably because there often is one such class more salient than others: “people who have social status within field X”. After all, one of the key explanations for the evolution of human intelligence is that of an arms race in social cognition. For example, you see this in studies where people are clueless at solving logic problems, unless you phrase them in terms of detecting cheaters or other breaches of social contracts (see e.g. Cheng & Holyoak, 1985 and Gigerenzer & Hug, 1992). So we should expect humans to easily figure out who has status within a field, but to have a very hard time figuring out who gets the field closer to truth.
Isn’t this exactly why modesty is such an appealing and powerful view in the first place? Because choosing the reference class is so easy (not requiring much object-level investigation), and experts are correct sufficiently often, that any inside view is mistaken in expectation.
Right. I will say that the heuristic of trusting “people who have social status within field X” seems to have very obvious flaws and biases, and it’s important to build models to work past those mistakes. Furthermore, in the world around us, most high status people in institutions are messing up most of the time. Bank of Japan is fine example, but also the people running my prestigious high school, my university, researchers in my field, etc. It’s important to learn where they’re inadequate so you can find extra value.
I’m not saying that in general “trust the high status people” isn’t probably okay if you have zero further info, but you definitely need to start building detailed models and get better than that otherwise you’re definitely not going to save the world.