If the system allocates all resources into the highest EV option,
and that sole option does not pay off, then the system fails. This is a known fact in finance and
many other fields that take a portfolio approach to investments
I think this is a mistaken analogy. In finance, you want to minimise variance (more broadly, people are willing to pay some expected value to reduce variance). Diversifying reduces variance at the cost of EV, which generally makes sense.
In comparison, in AI Safety we probably don’t care about variance—the goal is to minimise the probability of x-risk. Instead, the key question is how to maximise the expected reduction of x risk. When allocating people between fields, the two questions are the person’s comparative advantage in each field, and the diminishing/increasing returns to each person in the field. I expect there are initially increasing returns to each new people in a field (it’s very hard to do research when totally alone in the field), and then diminishing returns. But given how small a field safety is relative to ML, it is not obvious to me when you hit the diminishing returns. Then there are other considerations, eg value of information convincing people that the subfield is valuable, etc. Comparative advantage also seems like a big deal, eg I’m much more productive at interpretability research than anything else I’ve tried. You might also get increasing returns because plausibly alignment work only reduces x risk if it’s fully mature,and this is really hard. In which case putting all our eggs in the path most likely to get to AGI is the correct decision.
On net I probably agree that having a diverse portfolio of research is good for value of information reasons, but it’s definitely not obvious!
I think this is a mistaken analogy. In finance, you want to minimise variance (more broadly, people are willing to pay some expected value to reduce variance). Diversifying reduces variance at the cost of EV, which generally makes sense.
In comparison, in AI Safety we probably don’t care about variance—the goal is to minimise the probability of x-risk. Instead, the key question is how to maximise the expected reduction of x risk. When allocating people between fields, the two questions are the person’s comparative advantage in each field, and the diminishing/increasing returns to each person in the field. I expect there are initially increasing returns to each new people in a field (it’s very hard to do research when totally alone in the field), and then diminishing returns. But given how small a field safety is relative to ML, it is not obvious to me when you hit the diminishing returns. Then there are other considerations, eg value of information convincing people that the subfield is valuable, etc. Comparative advantage also seems like a big deal, eg I’m much more productive at interpretability research than anything else I’ve tried. You might also get increasing returns because plausibly alignment work only reduces x risk if it’s fully mature,and this is really hard. In which case putting all our eggs in the path most likely to get to AGI is the correct decision.
On net I probably agree that having a diverse portfolio of research is good for value of information reasons, but it’s definitely not obvious!