Appreciation for sharing the reasoning. Disagreement with the reasoning.
eeegnu is saying they go where their ideas take them and expressing ethical qualms, which both seem like excellent reasons to want someone considering AI safety work rather than reasons to drive them away from AI safety work. Their decision to continue doing AI safety work seems likely to be correlated with whether they could be productive by doing additional AI safety work—if their ideas take them elsewhere it is unlikely anything would have come of them staying.
This is especially true if one subscribes to the theory that we are worried about sign mistakes rather than ‘wasting’ funding—if we are funding unproven individuals in AI Safety and think that is good, then this is unusually ‘safe’ in the sense of it being more non-negative.
So to the extent that I was running the LTFF, I would have said yes.
I don’t think the policy of “I will fund people to do work that I don’t expect to be useful” is a good one, unless there is some positive externality.
It seems to me that your comment is also saying that the positive externality you are looking for is “this will make this person more productive in helping with AI Safety”, or maybe “this will make them more likely to work on AI Safety”. But you are separately saying that I shouldn’t take their self-reported prediction that they will not continue working in AI Safety, independently of the outcome of the experiment, at face value, and instead bet that by working on this, they will change their mind, which seems weird to me.
Separately, I think there are bad cultural effects of having people work on projects that seem very unlikely to work, especially if the people working on them are self-reportedly not doing so with a long-term safety motivation, but because they found the specific idea they had appealing (or wanted to play around with technologies in the space). I think this will predictably attract a large number of grifters and generally make the field a much worse place to be.
“I don’t think the policy of “I will fund people to do work that I don’t expect to be useful” is a good one, unless there is some positive externality.”
By this, do you mean you think it’s not good to fund work that you expect to be useful with < 50% probability, even if the downside risk is zero?
Or do you mean you don’t expect it’s useful to fund work you strongly expect to have no positive value when you also expect it to have a significant risk of causing harm?
50% is definitely not my cutoff, and I don’t have any probability cutoff. More something in the expected value space. Like, if you have an idea that could be really great but only has a 1% chance of working, that still feels definitely worth funding. But if you have an idea that seems like it only improves things a bit, and has a 10% chance of working, that doesn’t feel worth it.
Appreciation for sharing the reasoning. Disagreement with the reasoning.
eeegnu is saying they go where their ideas take them and expressing ethical qualms, which both seem like excellent reasons to want someone considering AI safety work rather than reasons to drive them away from AI safety work. Their decision to continue doing AI safety work seems likely to be correlated with whether they could be productive by doing additional AI safety work—if their ideas take them elsewhere it is unlikely anything would have come of them staying.
This is especially true if one subscribes to the theory that we are worried about sign mistakes rather than ‘wasting’ funding—if we are funding unproven individuals in AI Safety and think that is good, then this is unusually ‘safe’ in the sense of it being more non-negative.
So to the extent that I was running the LTFF, I would have said yes.
I don’t think the policy of “I will fund people to do work that I don’t expect to be useful” is a good one, unless there is some positive externality.
It seems to me that your comment is also saying that the positive externality you are looking for is “this will make this person more productive in helping with AI Safety”, or maybe “this will make them more likely to work on AI Safety”. But you are separately saying that I shouldn’t take their self-reported prediction that they will not continue working in AI Safety, independently of the outcome of the experiment, at face value, and instead bet that by working on this, they will change their mind, which seems weird to me.
Separately, I think there are bad cultural effects of having people work on projects that seem very unlikely to work, especially if the people working on them are self-reportedly not doing so with a long-term safety motivation, but because they found the specific idea they had appealing (or wanted to play around with technologies in the space). I think this will predictably attract a large number of grifters and generally make the field a much worse place to be.
“I don’t think the policy of “I will fund people to do work that I don’t expect to be useful” is a good one, unless there is some positive externality.”
By this, do you mean you think it’s not good to fund work that you expect to be useful with < 50% probability, even if the downside risk is zero?
Or do you mean you don’t expect it’s useful to fund work you strongly expect to have no positive value when you also expect it to have a significant risk of causing harm?
50% is definitely not my cutoff, and I don’t have any probability cutoff. More something in the expected value space. Like, if you have an idea that could be really great but only has a 1% chance of working, that still feels definitely worth funding. But if you have an idea that seems like it only improves things a bit, and has a 10% chance of working, that doesn’t feel worth it.