Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Concretely, the current presidential election seems extremely important to me from an AI safety perspective, I expect that importance to only go up in future elections, and I think OpenPhil is correct on what candidates are best from an AI safety perspective. Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs, and probably the most important other stuff to do is governance and policy work, which this strategy seems helpful for.
I don’t know the actual marginal increase in political influence that they’re buying here, but my guess would be that the numbers pencil and OpenPhil is making the right call.
I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding.
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Jan Leike, Paul (as you mention), etc.
Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs,
It might be the case that most of the quality weighted safety research involving working with large models is happening at labs, but I’m pretty skeptical that having this mostly happen at labs is the best approach and it seems like OpenPhil should be actively interested in building up a robust safety research ecosystem outside of labs.
(Better model access seems substantially overrated in its importance and large fractions of research can and should happen with just prompting or on smaller models. Additionally, at the moment, open weight models are pretty close to the best models.)
(This argument is also locally invalid at a more basic level. Just because this research seems to be mostly happening at large AI companies (which I’m also more skeptical of I think) doesn’t imply that this is the way it should be and funding should try to push people to do better stuff rather than merely reacting to the current allocation.)
Yeah, I think that’s a pretty fair criticism, but afaict that is the main thing that OpenPhil is still funding in AI safety? E.g. all the RFPs that they’ve been doing, I think they funded Jacob Steinhardt, etc. Though I don’t know much here; I could be wrong.
Wasn’t the relevant part of your argument like, “AI safety research outside of the labs is not that good, so that’s a contributing factor among many to it not being bad to lose the ability to do safety funding for governance work”? If so, I think that “most of OpenPhil’s actual safety funding has gone to building a robust safety research ecosystem outside of the labs” is not a good rejoinder to “isn’t there a large benefit to building a robust safety research ecosystem outside of the labs?”, because the rejoinder is focusing on relative allocations within “(technical) safety research”, and the complaint was about the allocation between “(technical) safety research” vs “other AI x-risk stuff”.
Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Yeah, I currently think Open Phil’s policy activism has been harmful for the world, and will probably continue to be, so by my lights this is causing harm with the justification of causing even more harm. I agree they will probably get the bit right about what major political party would be better, but sadly the effects of policy work are much more nuanced and detailed than that, and also they will have extremely little influence on who wins the general elections.
We could talk more about this sometime. I also have some docs with more of my thoughts here (which I maybe already shared with you, but would be happy to do so if not).
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Paul (as you mention), etc.
I genuinely don’t know whether Rohin would get funding to pursue what he thinks is most important, if he wanted it. I agree that some others don’t “need” funding anymore, though as I said, lab incentives are even worse on these dimensions and is of very little solace to me. I agree you might be able to get funding, though also see my other discussion with Eli on the boundaries I was trying to draw (which I agree are fuzzy and up-to-debate).
sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade
Sacrificing half of it to avoid things associated with one of the two major political parties and being deceptive about doing this is of course not equal to half the cost of sacrificing all of such funding, it is a much more unprincipled and distorting and actively deceptive decision that messes up everyone’s maps of the world in a massive way and reduces our ability to trust each other or understand what is happening.
Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Concretely, the current presidential election seems extremely important to me from an AI safety perspective, I expect that importance to only go up in future elections, and I think OpenPhil is correct on what candidates are best from an AI safety perspective. Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs, and probably the most important other stuff to do is governance and policy work, which this strategy seems helpful for.
I don’t know the actual marginal increase in political influence that they’re buying here, but my guess would be that the numbers pencil and OpenPhil is making the right call.
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Jan Leike, Paul (as you mention), etc.
It might be the case that most of the quality weighted safety research involving working with large models is happening at labs, but I’m pretty skeptical that having this mostly happen at labs is the best approach and it seems like OpenPhil should be actively interested in building up a robust safety research ecosystem outside of labs.
(Better model access seems substantially overrated in its importance and large fractions of research can and should happen with just prompting or on smaller models. Additionally, at the moment, open weight models are pretty close to the best models.)
(This argument is also locally invalid at a more basic level. Just because this research seems to be mostly happening at large AI companies (which I’m also more skeptical of I think) doesn’t imply that this is the way it should be and funding should try to push people to do better stuff rather than merely reacting to the current allocation.)
Yeah, I think that’s a pretty fair criticism, but afaict that is the main thing that OpenPhil is still funding in AI safety? E.g. all the RFPs that they’ve been doing, I think they funded Jacob Steinhardt, etc. Though I don’t know much here; I could be wrong.
Wasn’t the relevant part of your argument like, “AI safety research outside of the labs is not that good, so that’s a contributing factor among many to it not being bad to lose the ability to do safety funding for governance work”? If so, I think that “most of OpenPhil’s actual safety funding has gone to building a robust safety research ecosystem outside of the labs” is not a good rejoinder to “isn’t there a large benefit to building a robust safety research ecosystem outside of the labs?”, because the rejoinder is focusing on relative allocations within “(technical) safety research”, and the complaint was about the allocation between “(technical) safety research” vs “other AI x-risk stuff”.
Yeah, I currently think Open Phil’s policy activism has been harmful for the world, and will probably continue to be, so by my lights this is causing harm with the justification of causing even more harm. I agree they will probably get the bit right about what major political party would be better, but sadly the effects of policy work are much more nuanced and detailed than that, and also they will have extremely little influence on who wins the general elections.
We could talk more about this sometime. I also have some docs with more of my thoughts here (which I maybe already shared with you, but would be happy to do so if not).
I genuinely don’t know whether Rohin would get funding to pursue what he thinks is most important, if he wanted it. I agree that some others don’t “need” funding anymore, though as I said, lab incentives are even worse on these dimensions and is of very little solace to me. I agree you might be able to get funding, though also see my other discussion with Eli on the boundaries I was trying to draw (which I agree are fuzzy and up-to-debate).
Sacrificing half of it to avoid things associated with one of the two major political parties and being deceptive about doing this is of course not equal to half the cost of sacrificing all of such funding, it is a much more unprincipled and distorting and actively deceptive decision that messes up everyone’s maps of the world in a massive way and reduces our ability to trust each other or understand what is happening.