I think there are some important qualifications to make about this post, as others have noted.
My hunch is that most significant problem with the MWA approach is the assumption of (weak) independence, in the sense that in practice, when sophisticated use of MWA fails, it’s usually because the weak lines of evidence are all being driven by the same selection effect. A hypothetical example that jumps to mind is:
A VC is evaluating a startup. He or she reasons
The sector is growing
My colleagues think that the sector is good to invest in
On an object level, their plan looks good
The people are impressive
and the situation is
Re: #1 — The reason that the sector is growing is because there’s a bubble
Re: #2 — The reason that the VC’s colleagues think that the sector is good to invest in is because, like the VC, they don’t recognize that there’s a bubble.
Re: #3 — The VC’s views on the object level merit of the project are colored by the memes that have been spreading around that are causing the bubble
Re: #4 — The reason that impressive people are going into the sector is because there’s a bubble, so everyone’s going into the sector – the people’s impressiveness isn’t manifesting itself in their choosing a good focus.
I don’t know whether this situation occurs in practice, but it seems very possible.
Givewell tends to emphasize the MWA approach, and has been remarkably successful at figuring out the parts of the world they’re trying to understand.
GiveWell is an interesting case, insofar as it’s done more ORSA work than I’ve seen in most contexts. The page on long lasting insecticide treated nets provides examples. Part of why I’m favoring MWA is because GiveWell has done both and of the two, leans toward MWA.
This is a great example. It’s often very hard to tell whether MWA are independent or not. They could all derive from the same factors. Or they could all be made up by the same type of motivated reasoning.
I think that’s the judgment of being a good “Fox” ala Tetlock’s Hedgehog vs the Fox.
Which situation? The VC startup thing? The OSRA style is to scrutinize arguments carefully to see if they break down. If you can recognize that the arguments all break down in the same way, then you can conclude that the arguments are dependent and that even collectively, they don’t constitute much evidence.
But that still doesn’t tell you whether to invest in the startup. If an ORSA-er is just paralyzed by indecision here and decides to leave VC and go into theoretical math or whatever, he or she is not really winning.
Unrelatedly, a fun example of MWA triumphing over ORSA could be geologists vs. physicists on the age of the Earth.
But that still doesn’t tell you whether to invest in the startup. If an ORSA-er is just paralyzed by indecision here and decides to leave VC and go into theoretical math or whatever, he or she is not really winning.
I would guess that ORSA doesn’t suffice to be a successful VC. The claim is that it could help, in conjunction with MWAs.
If you scrutinize the weak arguments and find that the break down in different ways, then that suggests that the arguments are independent, and that you should invest in the start-up. If you find that they break down in the same way, then that suggests that you shouldn’t invest in the start-up.
Thanks for the feedback.
My hunch is that most significant problem with the MWA approach is the assumption of (weak) independence, in the sense that in practice, when sophisticated use of MWA fails, it’s usually because the weak lines of evidence are all being driven by the same selection effect. A hypothetical example that jumps to mind is:
A VC is evaluating a startup. He or she reasons
The sector is growing
My colleagues think that the sector is good to invest in
On an object level, their plan looks good
The people are impressive
and the situation is
Re: #1 — The reason that the sector is growing is because there’s a bubble
Re: #2 — The reason that the VC’s colleagues think that the sector is good to invest in is because, like the VC, they don’t recognize that there’s a bubble.
Re: #3 — The VC’s views on the object level merit of the project are colored by the memes that have been spreading around that are causing the bubble
Re: #4 — The reason that impressive people are going into the sector is because there’s a bubble, so everyone’s going into the sector – the people’s impressiveness isn’t manifesting itself in their choosing a good focus.
I don’t know whether this situation occurs in practice, but it seems very possible.
GiveWell is an interesting case, insofar as it’s done more ORSA work than I’ve seen in most contexts. The page on long lasting insecticide treated nets provides examples. Part of why I’m favoring MWA is because GiveWell has done both and of the two, leans toward MWA.
This is a great example. It’s often very hard to tell whether MWA are independent or not. They could all derive from the same factors. Or they could all be made up by the same type of motivated reasoning.
I think that’s the judgment of being a good “Fox” ala Tetlock’s Hedgehog vs the Fox.
Do you know ORSA that gets you out of this situation?
Which situation? The VC startup thing? The OSRA style is to scrutinize arguments carefully to see if they break down. If you can recognize that the arguments all break down in the same way, then you can conclude that the arguments are dependent and that even collectively, they don’t constitute much evidence.
But that still doesn’t tell you whether to invest in the startup. If an ORSA-er is just paralyzed by indecision here and decides to leave VC and go into theoretical math or whatever, he or she is not really winning.
Unrelatedly, a fun example of MWA triumphing over ORSA could be geologists vs. physicists on the age of the Earth.
I would guess that ORSA doesn’t suffice to be a successful VC. The claim is that it could help, in conjunction with MWAs.
If you scrutinize the weak arguments and find that the break down in different ways, then that suggests that the arguments are independent, and that you should invest in the start-up. If you find that they break down in the same way, then that suggests that you shouldn’t invest in the start-up.
I’d definitely be interested to learn (in more detail, with more examples) why this is. They may very well have good reasons for it.