Epistemic status: Speculating about adversarial and somewhat deceptive PR optimization, which is inherently very hard and somewhat paranoia inducing. I am quite confident of the broad trends here, but it’s definitely more likely that I am getting things wrong here than in other domains where evidence is more straightforward to interpret, and people are less likely to shape their behavior in ways that includes plausible deniability and defensibility.
I agree with this, but I actually think the issues with Open Phil are substantially broader. As a concrete example, as far as I can piece together from various things I have heard, Open Phil does not want to fund anything that is even slightly right of center in any policy work. I don’t think this is because of any COIs, it’s because Dustin is very active in the democratic party and doesn’t want to be affiliated with anything that is right-coded. Of course, this has huge effects by incentivizing polarization of AI policy work with billions of dollars, since any AI Open Phil funded policy organization that wants to engage with people on the right might just lose all of their funding because of that, and so you can be confident they will steer away from that.
Open Phil is also very limited in what they can say about what they can or cannot fund, because that itself is something that they are worried will make people annoyed with Dustin, which creates a terrible fog around how OP is thinking about stuff.[1]
Honestly, I think there might no longer a single organization that I have historically been excited about that OpenPhil wants to fund. MIRI could not get OP funding, FHI could not get OP funding, Lightcone cannot get OP funding, my best guess is Redwood could not get OP funding if they tried today (though I am quite uncertain of this), most policy work I am excited about cannot get OP funding, the LTFF cannot get OP funding, any kind of intelligence enhancement work cannot get OP funding, CFAR cannot get OP funding, SPARC cannot get OP funding, FABRIC (ESPR etc.) and Epistea (FixedPoint and other Prague-based projects) cannot get OP funding, not even ARC is being funded by OP these days (in that case because of COIs between Paul and Ajeya).[2] I would be very surprised if Wentworth’s work, or Wei Dai’s work, or Daniel Kokotajlo’s work, or Brian Tomasik’s work could get funding from them these days. I might be missing some good ones, but the funding landscape is really quite thoroughly fucked in that respect. My best guess is Scott Alexander could not get funding, but I am not totally sure.[3]
I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding. Bostrom, Eliezer, Hanson, Gwern, Tomasik, Kokotajlo, Sandberg, Armstrong, Jessicata, Garrabrant, Demski, Critch, Carlsmith, would all be unable to get funding[4] as far as I can tell. In as much as OP is the most powerful actor in the space, the original geeks are being thoroughly ousted.[5]
In-general my sense is if you want to be an OP longtermist grantee these days, you have to be the kind of person that OP thinks is not and will not be a PR risk, and who OP thinks has “good judgement” on public comms, and who isn’t the kind of person who might say weird or controversial stuff, and is not at risk of becoming politically opposed to OP. This includes not annoying any potential allies that OP might have, or associating with anything that Dustin doesn’t like, or that might strain Dustin’s relationships with others in any non-trivial way.
Of course OP will never ask you to fit these constraints directly, since that itself could explode reputationally (and also because OP staff themselves seem miscalibrated on this and do not seem in-sync with their leadership). Instead you will just get less and less funding, or just be defunded fully, if you aren’t the kind of person who gets the hint that this is how the game is played now.
And to provide some pushback on things you say, I think now that OPs bridges with OpenAI are thoroughly burned after the Sam firing drama, OP is pretty OK with people criticizing OpenAI (since what social capital is there left to protect here?). My sense is criticizing Anthropic is slightly risky, especially if you do it in a way that doesn’t signal what OP considers good judgement on maintaining and spending your social capital appropriately (i.e. telling them that they are harmful for the world, or should really stop, is bad, but doing a mixture of praise and criticism without taking any controversial top-level stance is fine), but mostly also isn’t the kind of thing that OP will totally freak out about. I think OP used to be really crazy about this, but now is a bit more reasonable, and it’s not the domain where OP’s relationship to reputation-management is causing the worst failures.
I think all of this is worse in the longtermist space, though I am not confident. At the present it wouldn’t surprise me very much if OP would defund a global health grantee because their CEO endorsed Trump for president, so I do think there is also a lot of distortion and skew there, but my sense is that it’s less, mostly because the field is much more professionalized and less political (though I don’t know how they think, for example, about funding on corporate campaign stuff which feels like it would be more political and invite more of these kinds of skewed considerations).
Also, to balance things, sometimes OP does things that seem genuinely good to me. The lead reduction fund stuff seems good, genuinely neglected, and I don’t see that many of these dynamics at play there (I do also genuinely care about it vastly less than OPs effect on AI Safety and Rationality things).
As an example of this, in the announcement of recent Open Phil defunding decisions, they communicated that they are “withdrawing funding from some cause areas”, which was really a quite misleading way to describe what actually happened and basically no one I talked to who read the post understood what happened correctly.
If you “withdraw from a cause area” you would expect that if you have an organization that does good work in multiple cause areas, then you would expect you would still fund the organization for work in cause areas that funding wasn’t withdrawn from. However, what actually happened is that Open Phil blacklisted a number of ill-defined broad associations and affiliations, where if you are associated with a certain set of ideas, or identities or causes, then no matter how cost-effective your other work is, you cannot get funding from OP. This is of course an enormously different situations, with an enormously different set of incentives. But as far as I can tell all the comms here was dictated by Dustin, and nobody from OP felt comfortable clarifying what is actually happening in public, and so tons of people were misled about what actually happened.
To elaborate on this a bit: Mostly ARC and METR have been trying to avoid taking Open Phil money due to minimizing conflicts of interest. My guess is if those COIs were not a concern they could get funding. But as a result, a really huge fraction of the non-OP funding has been going to ARC and METR in a way that has substantially contributed to centralizing funding decisions under OP, making the effects of this case quite similar to all the other cases (though with different dynamics involved).
(Edit: Two organizations that came to mind that OP has funded somewhat recently that I do think are good are FAR AI and AI Impacts. My sense is that FAR AI was quite unhappy with the pressures OP kept putting on them, with them pushing them away from doing what they considered important intellectual work, but instead do more talent funnel building, and they are less dependent on OP funding than others these days, and I am also not confident they could get more funding for things like FAR Labs if they applied today. I do think them funding AI Impacts is good, but also wouldn’t be that surprised to see that end because for example AI Impacts wants to think about AI sentience and OP cannot fund orgs that do that kind of work.)
(Edit: “Unable to get funding” meaning here “unable to get funding for the work that they think is most important”. I think many of these people are smart and of course could just subjugate their labor to OP themselves or some other entity that OP approves of, but that of course mostly makes the situation worse, especially in that they would be paid an order of magnitude less than their expected market wage if they did so)
(Edit: As I mention above, Paul is a weird case. He could very likely get funding, but has been avoiding taking it for other reasons. Now editing this a few hours after it was originally written, I think Carl Shulman is also someone who could get funding, and I do think is in the same reference class as the others here, and so is at least one counterexample, though I am not confident on Carl.)
Adding my two cents as someone who has a pretty different lens from Habryka but has still been fairly disappointed with OpenPhil, especially in the policy domain.
Relative to Habryka, I am generally more OK with people “playing politics”. I think it’s probably good for AI safely folks to exhibit socially-common levels of “playing the game”– networking, finding common ground, avoiding offending other people, etc. I think some people in the rationalist sphere have a very strong aversion to some things in this genre, and labels like “power-seeking” and “deceptive” get thrown around too liberally. I also think I’m pretty with OpenPhil deciding it doesn’t want to fund certain parts of the rationalist ecosystem (and probably less bothered than Habryka about how their comms around this wasn’t direct/clear).
In that sense, I don’t penalize OP much for trying to “play politics” or for breaking deontological norms. Nonetheless, I still feel pretty disappointed with them, particularly for their impact on comms/policy. Some thoughts here:
I agree with Habryka that it is quite bad that OP is not willing to fund right-coded things. Even many of the “bipartisan” things funded by OP are quite left-coded. (As a useful heuristic, whenever you hear of someone launching a bipartisan initiative, I think one should ask “what % of the staff of this organization is Republican?” Obviously just a heuristic– there are some cases in which a 90%-Dem staff can actually truly engage in “real” bipartisan efforts. But in some cases, you will have a 90%-Dem staff claiming to be interested in bipartisan work without any real interest in Republican ideas, few if any Republican contacts, and only a cursory understanding of Republican stances.)
I also agree with Habryka that OP seems overly focused on PR risks and not doing things that are weird/controversial. “To be a longtermist grantee these days you have to be the kind of person that OP thinks is not and will not be a PR risk, IE will not say weird or controversial stuff” sounds pretty accurate to me. OP cannot publicly admit this because this would be bad for its reputation– instead, it operates more subtly.
Separately, I have seen OpenPhil attempt to block or restrain multiple efforts in which people were trying to straightforwardly explain AI risks to policymakers. My understanding is that OpenPhil would say that they believed the messengers weren’t the right people (e.g., too inexperienced), and they thought the downside risks were too high. In practice, there are some real tradeoffs here: there are often people who seem to have strong models of AGI risk but little/no policy experience, and sometimes people who have extensive policy experience but only recently started engaging with AI/AGI issues. With that in mind, I think OpenPhil has systematically made poor tradeoffs here and failed to invest into (or in some cases, actively blocked) people who were willing to be explicit about AGI risks, loss of control risks, capability progress, and the need for regulation. (I also think the “actively blocking” thing has gotten less severe over time, perhaps in part because OpenPhil changed its mind a bit on the value of direct advocacy or perhaps because OpenPhil just decided to focus its efforts on things like research and advocacy projects found funding elsewhere.)
I think OpenPhil has an intellectual monoculture and puts explicit/implicit cultural pressure on people in the OP orbit to “stay in line.” There is a lot of talk about valuing people who can think for themselves, but I think the groupthink problems are pretty real. There is a strong emphasis on “checking-in” with people before saying/doing things, and the OP bubble is generally much more willing to criticize action than inaction. I suspect that something like the CAIS statement or even a lot of the early Bengio comms would not have occured if Dan Hendrycks or Yoshua were deeply ingrained in the OP orbit. It is both the case that they would’ve been forced to write 10+ page Google Docs defending their theories of change and the case that the intellectual culture simply wouldn’t have fostered this kind of thinking.
I think the focus on evals/RSPs can largely be explained by a bias toward trusting labs. OpenPhil steered a lot of talent toward the evals/RSPs theory of change (specifically, if I recall correctly, OpenPhil leadership on AI was especially influential in steering a lot of the ecosystem to support and invest in the evals/RSPs theory of change.) I expect that when we look back in a few years, there will be a pretty strong feeling that this was the wrong call & that this should’ve been more apparent even without the benefit of hindsight.
I would be more sympathetic to OpenPhil in a world where their aversion to weirdness/PR risks resulted in them having a strong reputation, a lot of political capital, and real-world influence that matched the financial resources they possess. Sadly, I think we’re in a “lose-lose” world: OpenPhil’s reputation tends to be poor in many policy/journalism circles even while OpenPhil pursues a strategy that seems to be largely focused on avoiding PR risks. I think some of this is unjustified (e.g., a result of propaganda campaigns designed to paint anyone who cares about AI risk as awful). But then some of it actually is kind of reasonable (e.g., impartial observers viewing OpenPhil as kind of shady, not direct in its communications, not very willing to engage directly or openly with policymakers or journalists, having lots of conflicts of interests, trying to underplay the extent to which its funding priorities are influenced/constrained by a single Billionaire, being pretty left-coded, etc.)
To defend OpenPhil a bit, I do think it’s quite hard to navigate trade-offs and I think sometimes people don’t seem to recognize these tradeoffs. In AI policy, I think the biggest tradeoff is something like “lots of people who have engaged with technical AGI arguments and AGI threat models don’t have policy experience, and lots of people who have policy experience don’t have technical expertise or experience engaging with AGI threat models” (this is a bit of an oversimplification– there are some shining stars who have both.)
I also think OpenPhil folks probably tend to have a different probability distribution over threat models (compared to me and probably also Habryka). For instance, it seems likely to me that OpenPhil employees operate in more of a “there are a lot of ways AGI could play out and a lot of uncertainty– we just need smart people thinking seriously about the problem. And who really know how hard alignment will be, maybe Anthropic will just figure it out” lens and less of a “ASI is coming and our priority needs to be making sure humanity understands the dangers associated with a reckless race toward ASI, and there’s a substantial chance that we are seriously not on track to solve the necessary safety and security challenges unless we fundamentally reorient our whole approach” lens.
And finally, I think despite these criticisms, OpenPhil is also responsible for some important wins (e.g., building the field, raising awareness about AGI risk on university campuses, funding some people early on before AI safety was a “big deal”, jumpstarting the careers of some leaders in the policy space [especially in the UK]. It’s also plausible to me that there are some cases in which OpenPhil gatekeeping was actually quite useful in preventing people from causing harm, even though I probably disagree with OpenPhil about the # and magnitude of these cases).
What are the norms here? Can I just copy/paste this exact text and put it into a top-level post? I got the sense that a top-level post should be more well thought out than this but I don’t actually have anything else useful to say. I would be happy to co-author a post if someone else thinks they can flesh it out.
Edit: Didn’t realize you were replying to Habryka, not me. That makes more sense.
It feels sorta understandable to me (albeit frustrating) that OpenPhil faces these assorted political constraints. In my view this seems to create a big unfilled niche in the rationalist ecosystem: a new, more right-coded, EA-adjacent funding organization could optimize itself for being able to enter many of those blacklisted areas with enthusiasm.
If I was a billionare, I would love to put together a kind of “completion portfolio” to complement some of OP’s work. Rationality community building, macrostrategy stuff, AI-related advocacy to try and influence republican politicians, plus a big biotechnology emphasis focused on intelligence enhancement, reproductive technologies, slowing aging, cryonics, gene drives for eradicating diseases, etc. Basically it seems like there is enough edgy-but-promising stuff out there (like studying geoengineering for climate, or advocating for charter cities, or just funding oddball substack intellectuals to do their thing) that you could hope to create a kind of “alt-EA” (obviously IRL it shouldn’t have EA in the name) where you batten down the hatches, accept that the media will call you an evil villain mastermind forever, and hope to create a kind of protective umbrella for all the work that can’t get done elsewhere. As a bonus, you could engage more in actual politics (like having some hot takes on the US budget deficit, or on how to increase marriage & fertility rates, or whatever), in some areas that OP in its quest for center-left non-polarization can’t do.
Peter Thiel already lives this life, kinda? But his model seems 1. much more secretive, and 2. less directly EA-adjacent, than what I’d try if I was a billionare.
Dustin himself talks about how he is really focused on getting more “multipolarity” to the EA landscape, by bringing in other high-net-worth funders. For all the reasons discussed, he obviously can’t say “hi, somebody please start an edgier right-wing offshoot of EA!!” But it seems like a major goal that the movement should have, nonetheless.
Seems like you could potentially also run this play with a more fully-left-coded organization. The gains there would probably be smaller, since there’s less “room” to OP’s left than to their right. But maybe you could group together wild animal welfare, invertebrate welfare, digital minds, perhaps some David Pearce / Project Far Out-style “suffering abolition” transhumanist stuff, other mental-wellbeing stuff like the Organization for the Prevention of Intense Suffering, S-risk work, etc. Toss in some more aggressive political activism on AI (like PauseAI) and other issues (like Georgist land value taxation), and maybe some forward-looking political stuff on avoiding stable totalitarianism, regulation of future AI-enabled technologies, and how to distribute the gains from a positive / successful singularity (akin to Sam Altman’s vision of UBI supported by georgist/pigouvian taxes, but more thought-through and detailed and up-to-date.)
Finding some funders to fill these niches seems like it should be a very high priority of the rationalist / EA movement. Even if the funders were relatively small at first (like say they have $10M - $100M in crypto that they are preparing to give away), I think there could be a lot of value in being “out and proud” (publicising much of their research and philosophy and grantmaking like OP, rather than being super-secretive like Peter Thiel). If a small funder manages to build a small successful “alt-EA” ecosystem on either the left or right, that might attract larger funders in time.
I think OP has funded almost everyone I have listed here in 2022 (directly or indirectly), so I don’t really think that is evidence of anything (though it is a bit more evidence for ARC because it means the COI is overcomable).
Hm, this timing suggests the change could be a consequence of Karnofsky stepping away from the organization.
Which makes sense, now that I think about it. He’s by far the most politically strategic leader Open Philanthropy has had, so with him gone, it’s not shocking they might revert towards standard risk-averse optionality-maxxing foundation behavior.
Isn’t it just the case that OpenPhil just generally doesn’t fund that many technical AI safety things these days? If you look at OP’s team on their website, they have only two technical AI safety grantmakers. Also, you list all the things OP doesn’t fund, but what are the things in technical AI safety that they do fund? Looking at their grants, it’s mostly MATS and METR and Apollo and FAR and some scattered academics I mostly haven’t heard of. It’s not that many things. I have the impression that the story is less like “OP is a major funder in technical AI safety, but unfortunately they blacklisted all the rationalist-adjacent orgs and people” and more like “AI safety is still a very small field, especially if you only count people outside the labs, and there are just not that many exciting funding opportunities, and OpenPhil is not actually a very big funder in the field”.
Open Phil is definitely by far the biggest funder in the field. I agree that their technical grantmaking has been a limited over the past few years (though still on the order of $50M/yr, I think), but they also fund a huge amount of field-building and talent-funnel work, as well as a lot of policy stuff (I wasn’t constraining myself to technical AI Safety, the people listed have been as influential, if not more, on public discourse and policy).
AI Safety is still relatively small, but more like $400M/yr small. The primary other employers/funders in the space these days are big capability labs. As you can imagine, their funding does not have great incentives either.
Yeah, I agree, and I don’t know that much about OpenPhil’s policy work, and their fieldbuilding seems decent to me, though maybe not from you perspective. I just wanted to flag that many people (including myself until recently) overestimate how big a funder OP is in technical AI safety, and I think it’s important to flag that they actually have pretty limited scope in this area.
Yep, agree that this is a commonly overlooked aspect (and one that I think sadly has also contributed to the dominant force in AI Safety researchers becoming the labs, which I think has been quite sad).
If you “withdraw from a cause area” you would expect that if you have an organization that does good work in multiple cause areas, then you would expect you would still fund the organization for work in cause areas that funding wasn’t withdrawn from. However, what actually happened is that Open Phil blacklisted a number of ill-defined broad associations and affiliations, where if you are associated with a certain set of ideas, or identities or causes, then no matter how cost-effective your other work is, you cannot get funding from OP
I’m wondering if you have a list of organizations where Open Phil would have funded their other work, but because they withdrew from funding part of the organization they decided to withdraw totally.
This feels very importantly different from good ventures choosing not to fund certain cause areas (and I think you agree, which is why you put that footnote).
I don’t have a long list, but I know this is true for Lightcone, SPARC, ESPR, any of the Czech AI-Safety/Rationality community building stuff, and I’ve heard a bunch of stories since then from other organizations that got pretty strong hints from Open Phil that if they start working in an area at all, they might lose all funding (and also, the “yes, it’s more like a blacklist, if you work in these areas at all we can’t really fund you, though we might make occasional exceptions if it’s really only a small fraction of what you do” story was confirmed to me by multiple OP staff, so I am quite confident in this, and my guess is OP staff would be OK with confirming to you as well if you ask them).
Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Concretely, the current presidential election seems extremely important to me from an AI safety perspective, I expect that importance to only go up in future elections, and I think OpenPhil is correct on what candidates are best from an AI safety perspective. Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs, and probably the most important other stuff to do is governance and policy work, which this strategy seems helpful for.
I don’t know the actual marginal increase in political influence that they’re buying here, but my guess would be that the numbers pencil and OpenPhil is making the right call.
I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding.
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Jan Leike, Paul (as you mention), etc.
Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs,
It might be the case that most of the quality weighted safety research involving working with large models is happening at labs, but I’m pretty skeptical that having this mostly happen at labs is the best approach and it seems like OpenPhil should be actively interested in building up a robust safety research ecosystem outside of labs.
(Better model access seems substantially overrated in its importance and large fractions of research can and should happen with just prompting or on smaller models. Additionally, at the moment, open weight models are pretty close to the best models.)
(This argument is also locally invalid at a more basic level. Just because this research seems to be mostly happening at large AI companies (which I’m also more skeptical of I think) doesn’t imply that this is the way it should be and funding should try to push people to do better stuff rather than merely reacting to the current allocation.)
Yeah, I think that’s a pretty fair criticism, but afaict that is the main thing that OpenPhil is still funding in AI safety? E.g. all the RFPs that they’ve been doing, I think they funded Jacob Steinhardt, etc. Though I don’t know much here; I could be wrong.
Wasn’t the relevant part of your argument like, “AI safety research outside of the labs is not that good, so that’s a contributing factor among many to it not being bad to lose the ability to do safety funding for governance work”? If so, I think that “most of OpenPhil’s actual safety funding has gone to building a robust safety research ecosystem outside of the labs” is not a good rejoinder to “isn’t there a large benefit to building a robust safety research ecosystem outside of the labs?”, because the rejoinder is focusing on relative allocations within “(technical) safety research”, and the complaint was about the allocation between “(technical) safety research” vs “other AI x-risk stuff”.
Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Yeah, I currently think Open Phil’s policy activism has been harmful for the world, and will probably continue to be, so by my lights this is causing harm with the justification of causing even more harm. I agree they will probably get the bit right about what major political party would be better, but sadly the effects of policy work are much more nuanced and detailed than that, and also they will have extremely little influence on who wins the general elections.
We could talk more about this sometime. I also have some docs with more of my thoughts here (which I maybe already shared with you, but would be happy to do so if not).
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Paul (as you mention), etc.
I genuinely don’t know whether Rohin would get funding to pursue what he thinks is most important, if he wanted it. I agree that some others don’t “need” funding anymore, though as I said, lab incentives are even worse on these dimensions and is of very little solace to me. I agree you might be able to get funding, though also see my other discussion with Eli on the boundaries I was trying to draw (which I agree are fuzzy and up-to-debate).
sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade
Sacrificing half of it to avoid things associated with one of the two major political parties and being deceptive about doing this is of course not equal to half the cost of sacrificing all of such funding, it is a much more unprincipled and distorting and actively deceptive decision that messes up everyone’s maps of the world in a massive way and reduces our ability to trust each other or understand what is happening.
As a concrete example, as far as I can piece together from various things I have heard, Open Phil does not want to fund anything that is even slightly right of center in any policy work. I don’t think this is because of any COIs, it’s because Dustin is very active in the democratic party and doesn’t want to be affiliated with anything that is even slightly right-coded. Of course, this has huge effects by incentivizing polarization of AI policy work with billions of dollars, since any AI Open Phil funded policy organization that wants to engage with people on the right might just lose all of their funding because of that, and so you can be confident they will steer away from that.
Thanks for sharing, I was curious if you could elaborate on this (e.g. if there are examples of AI policy work funded by OP that come to mind that are clearly left of center). I am not familiar with policy, but my one data point is the Horizon Fellowship, which is non-partisan and intentionally places congressional fellows in both Democratic and Republican offices. This straightforwardly seems to me like a case where they are trying to engage with people on the right, though maybe you mean not-right-of-center at the organizational level? In general though, (in my limited exposure) I don’t model any AI governance orgs as having a particular political affiliation (which might just be because I’m uninformed / ignorant).
Yep, my model is that OP does fund things that are explicitly bipartisan (like, they are not currently filtering on being actively affiliated with the left). My sense is in-practice it’s a fine balance and if there was some high-profile thing where Horizon became more associated with the right (like maybe some alumni becomes prominent in the republican party and very publicly credits Horizon for that, or there is some scandal involving someone on the right who is a Horizon alumni), then I do think their OP funding would have a decent chance of being jeopardized, and the same is not true on the left.
Another part of my model is that one of the key things about Horizon is that they are of a similar school of PR as OP themselves. They don’t make public statements. They try to look very professional. They are probably very happy to compromise on messaging and public comms with Open Phil and be responsive to almost any request that OP would have messaging wise. That makes up for a lot. I think if you had a more communicative and outspoken organization with a similar mission to Horizon, I think the funding situation would be a bunch dicier (though my guess is if they were competent, an organization like that could still get funding).
More broadly, I am not saying “OP staff want to only support organizations on the left”. My sense is that many individual OP staff would love to fund more organizations on the right, and would hate for polarization to occur, but that organizationally and because of constraints by Dustin, they can’t, and so you will see them fund organizations that aim for more engagement with the right, but there will be relatively hard lines and constraints that will mostly prevent that.
Thanks for the reply. When I wrote “Many people would have more useful things to say about this than I do”, you were one of the people I was thinking of.
AI Impacts wants to think about AI sentience and OP cannot fund orgs that do that kind of work
Related to this, I think GW/OP has always been too unwilling to fund weird causes, but it’s generally gotten better over time: originally recommending US charities over global poverty b/c global poverty was too weird, taking years to remove their recommendations for US charities that were ~100x less effective than their global poverty recs, then taking years to start funding animal welfare and x-risk, then still not funding weirder stuff like wild animal welfare and AI sentience. I’ve criticized them for this in the past but I liked that they were moving in the right direction. Now I get the sense that recently they’ve gotten worse on AI safety (and weird causes in general).
(I like Buck, but he is one generation later than the one I was referencing. Also, I am currently like 50⁄50 whether Buck would indeed be blacklisted. I agree that Carl is a decent counterexample, though he is a bit of a weirder case)
I remember running into her a bunch before I ran into Buck. Scott/Abram are also second generation. Overall, seems reasonable to include Buck (but communicating my more complicated epistemic state with regard to him would have been harder).
Out of curiosity—“it’s because Dustin is very active in the democratic party and doesn’t want to be affiliated with anything that is right-coded” Are these projects related to AI safety or just generally? And what are some examples?
I am not sure I am understanding your question. Are you asking about examples of left-leaning projects that Dustin is involved in, or right-leaning projects that cannot get funding? On the left, Dustin is one of the biggest donors to the democratic party (with Asana donating $45M and him donating $24M to Joe Biden in 2020).
I don’t currently know of any public examples and feel weird publicly disclosing details about organizations that I privately heard about. If more people are interested I can try to dig up some more concrete details (but can’t make any promises on things I’ll end up able sharing).
OP appears to have been one of FHI’s biggest funders according to Sandberg:[1]
Eventually, Open Philanthropy became FHI’s most important funder, making two major grants: £1.6m in 2017, and £13.3m in 2018. Indeed, the donation behind this second grant was at the time the largest in the Faculty of Philosophy’s history (although, owing to limited faculty administrative capacity for hiring and the subsequent hiring freezes it imposed, a large part of this grant would remain unspent). With generous and unrestricted funding from a foundation that was aligned with FHI’s mission, we were free to expand our research in ways we thought would make the most difference.
The hiring (and fundraising) freeze imposed by Oxf began in 2020.
In 2023/2024 OP drastically changed it’s funding process and priorities (in part in response to FTX, in part in response to Dustin’s preferences). This whole conversation is about the shift in OPs giving in this recent time period.
I agree with the claim you’re making: that if FHI still existed and they applied for a grant from OP it would be rejected. This seems true to me.
I don’t mean to nitpick, but it still feels misleading to claim “FHI could not get OP funding” when they did in fact get lots of funding from OP. It implies that FHI operated without any help from OP, which isn’t true.
The “could” here is (in context) about “could not get funding from modern OP”. The whole point of my comment was about the changes that OP underwent. Sorry if that wasn’t as clear, it might not be as obvious to others that of course OP was very different in the past.
Epistemic status: Speculating about adversarial and somewhat deceptive PR optimization, which is inherently very hard and somewhat paranoia inducing. I am quite confident of the broad trends here, but it’s definitely more likely that I am getting things wrong here than in other domains where evidence is more straightforward to interpret, and people are less likely to shape their behavior in ways that includes plausible deniability and defensibility.
I agree with this, but I actually think the issues with Open Phil are substantially broader. As a concrete example, as far as I can piece together from various things I have heard, Open Phil does not want to fund anything that is even slightly right of center in any policy work. I don’t think this is because of any COIs, it’s because Dustin is very active in the democratic party and doesn’t want to be affiliated with anything that is right-coded. Of course, this has huge effects by incentivizing polarization of AI policy work with billions of dollars, since any AI Open Phil funded policy organization that wants to engage with people on the right might just lose all of their funding because of that, and so you can be confident they will steer away from that.
Open Phil is also very limited in what they can say about what they can or cannot fund, because that itself is something that they are worried will make people annoyed with Dustin, which creates a terrible fog around how OP is thinking about stuff.[1]
Honestly, I think there might no longer a single organization that I have historically been excited about that OpenPhil wants to fund. MIRI could not get OP funding, FHI could not get OP funding, Lightcone cannot get OP funding, my best guess is Redwood could not get OP funding if they tried today (though I am quite uncertain of this), most policy work I am excited about cannot get OP funding, the LTFF cannot get OP funding, any kind of intelligence enhancement work cannot get OP funding, CFAR cannot get OP funding, SPARC cannot get OP funding, FABRIC (ESPR etc.) and Epistea (FixedPoint and other Prague-based projects) cannot get OP funding, not even ARC is being funded by OP these days (in that case because of COIs between Paul and Ajeya).[2] I would be very surprised if Wentworth’s work, or Wei Dai’s work, or Daniel Kokotajlo’s work, or Brian Tomasik’s work could get funding from them these days. I might be missing some good ones, but the funding landscape is really quite thoroughly fucked in that respect. My best guess is Scott Alexander could not get funding, but I am not totally sure.[3]
I cannot think of anyone who I would credit with the creation or shaping of the field of AI Safety or Rationality who could still get OP funding. Bostrom, Eliezer, Hanson, Gwern, Tomasik, Kokotajlo, Sandberg, Armstrong, Jessicata, Garrabrant, Demski, Critch, Carlsmith, would all be unable to get funding[4] as far as I can tell. In as much as OP is the most powerful actor in the space, the original geeks are being thoroughly ousted.[5]
In-general my sense is if you want to be an OP longtermist grantee these days, you have to be the kind of person that OP thinks is not and will not be a PR risk, and who OP thinks has “good judgement” on public comms, and who isn’t the kind of person who might say weird or controversial stuff, and is not at risk of becoming politically opposed to OP. This includes not annoying any potential allies that OP might have, or associating with anything that Dustin doesn’t like, or that might strain Dustin’s relationships with others in any non-trivial way.
Of course OP will never ask you to fit these constraints directly, since that itself could explode reputationally (and also because OP staff themselves seem miscalibrated on this and do not seem in-sync with their leadership). Instead you will just get less and less funding, or just be defunded fully, if you aren’t the kind of person who gets the hint that this is how the game is played now.
And to provide some pushback on things you say, I think now that OPs bridges with OpenAI are thoroughly burned after the Sam firing drama, OP is pretty OK with people criticizing OpenAI (since what social capital is there left to protect here?). My sense is criticizing Anthropic is slightly risky, especially if you do it in a way that doesn’t signal what OP considers good judgement on maintaining and spending your social capital appropriately (i.e. telling them that they are harmful for the world, or should really stop, is bad, but doing a mixture of praise and criticism without taking any controversial top-level stance is fine), but mostly also isn’t the kind of thing that OP will totally freak out about. I think OP used to be really crazy about this, but now is a bit more reasonable, and it’s not the domain where OP’s relationship to reputation-management is causing the worst failures.
I think all of this is worse in the longtermist space, though I am not confident. At the present it wouldn’t surprise me very much if OP would defund a global health grantee because their CEO endorsed Trump for president, so I do think there is also a lot of distortion and skew there, but my sense is that it’s less, mostly because the field is much more professionalized and less political (though I don’t know how they think, for example, about funding on corporate campaign stuff which feels like it would be more political and invite more of these kinds of skewed considerations).
Also, to balance things, sometimes OP does things that seem genuinely good to me. The lead reduction fund stuff seems good, genuinely neglected, and I don’t see that many of these dynamics at play there (I do also genuinely care about it vastly less than OPs effect on AI Safety and Rationality things).
As an example of this, in the announcement of recent Open Phil defunding decisions, they communicated that they are “withdrawing funding from some cause areas”, which was really a quite misleading way to describe what actually happened and basically no one I talked to who read the post understood what happened correctly.
If you “withdraw from a cause area” you would expect that if you have an organization that does good work in multiple cause areas, then you would expect you would still fund the organization for work in cause areas that funding wasn’t withdrawn from. However, what actually happened is that Open Phil blacklisted a number of ill-defined broad associations and affiliations, where if you are associated with a certain set of ideas, or identities or causes, then no matter how cost-effective your other work is, you cannot get funding from OP. This is of course an enormously different situations, with an enormously different set of incentives. But as far as I can tell all the comms here was dictated by Dustin, and nobody from OP felt comfortable clarifying what is actually happening in public, and so tons of people were misled about what actually happened.
To elaborate on this a bit: Mostly ARC and METR have been trying to avoid taking Open Phil money due to minimizing conflicts of interest. My guess is if those COIs were not a concern they could get funding. But as a result, a really huge fraction of the non-OP funding has been going to ARC and METR in a way that has substantially contributed to centralizing funding decisions under OP, making the effects of this case quite similar to all the other cases (though with different dynamics involved).
(Edit: Two organizations that came to mind that OP has funded somewhat recently that I do think are good are FAR AI and AI Impacts. My sense is that FAR AI was quite unhappy with the pressures OP kept putting on them, with them pushing them away from doing what they considered important intellectual work, but instead do more talent funnel building, and they are less dependent on OP funding than others these days, and I am also not confident they could get more funding for things like FAR Labs if they applied today. I do think them funding AI Impacts is good, but also wouldn’t be that surprised to see that end because for example AI Impacts wants to think about AI sentience and OP cannot fund orgs that do that kind of work.)
(Edit: “Unable to get funding” meaning here “unable to get funding for the work that they think is most important”. I think many of these people are smart and of course could just subjugate their labor to OP themselves or some other entity that OP approves of, but that of course mostly makes the situation worse, especially in that they would be paid an order of magnitude less than their expected market wage if they did so)
(Edit: As I mention above, Paul is a weird case. He could very likely get funding, but has been avoiding taking it for other reasons. Now editing this a few hours after it was originally written, I think Carl Shulman is also someone who could get funding, and I do think is in the same reference class as the others here, and so is at least one counterexample, though I am not confident on Carl.)
Adding my two cents as someone who has a pretty different lens from Habryka but has still been fairly disappointed with OpenPhil, especially in the policy domain.
Relative to Habryka, I am generally more OK with people “playing politics”. I think it’s probably good for AI safely folks to exhibit socially-common levels of “playing the game”– networking, finding common ground, avoiding offending other people, etc. I think some people in the rationalist sphere have a very strong aversion to some things in this genre, and labels like “power-seeking” and “deceptive” get thrown around too liberally. I also think I’m pretty with OpenPhil deciding it doesn’t want to fund certain parts of the rationalist ecosystem (and probably less bothered than Habryka about how their comms around this wasn’t direct/clear).
In that sense, I don’t penalize OP much for trying to “play politics” or for breaking deontological norms. Nonetheless, I still feel pretty disappointed with them, particularly for their impact on comms/policy. Some thoughts here:
I agree with Habryka that it is quite bad that OP is not willing to fund right-coded things. Even many of the “bipartisan” things funded by OP are quite left-coded. (As a useful heuristic, whenever you hear of someone launching a bipartisan initiative, I think one should ask “what % of the staff of this organization is Republican?” Obviously just a heuristic– there are some cases in which a 90%-Dem staff can actually truly engage in “real” bipartisan efforts. But in some cases, you will have a 90%-Dem staff claiming to be interested in bipartisan work without any real interest in Republican ideas, few if any Republican contacts, and only a cursory understanding of Republican stances.)
I also agree with Habryka that OP seems overly focused on PR risks and not doing things that are weird/controversial. “To be a longtermist grantee these days you have to be the kind of person that OP thinks is not and will not be a PR risk, IE will not say weird or controversial stuff” sounds pretty accurate to me. OP cannot publicly admit this because this would be bad for its reputation– instead, it operates more subtly.
Separately, I have seen OpenPhil attempt to block or restrain multiple efforts in which people were trying to straightforwardly explain AI risks to policymakers. My understanding is that OpenPhil would say that they believed the messengers weren’t the right people (e.g., too inexperienced), and they thought the downside risks were too high. In practice, there are some real tradeoffs here: there are often people who seem to have strong models of AGI risk but little/no policy experience, and sometimes people who have extensive policy experience but only recently started engaging with AI/AGI issues. With that in mind, I think OpenPhil has systematically made poor tradeoffs here and failed to invest into (or in some cases, actively blocked) people who were willing to be explicit about AGI risks, loss of control risks, capability progress, and the need for regulation. (I also think the “actively blocking” thing has gotten less severe over time, perhaps in part because OpenPhil changed its mind a bit on the value of direct advocacy or perhaps because OpenPhil just decided to focus its efforts on things like research and advocacy projects found funding elsewhere.)
I think OpenPhil has an intellectual monoculture and puts explicit/implicit cultural pressure on people in the OP orbit to “stay in line.” There is a lot of talk about valuing people who can think for themselves, but I think the groupthink problems are pretty real. There is a strong emphasis on “checking-in” with people before saying/doing things, and the OP bubble is generally much more willing to criticize action than inaction. I suspect that something like the CAIS statement or even a lot of the early Bengio comms would not have occured if Dan Hendrycks or Yoshua were deeply ingrained in the OP orbit. It is both the case that they would’ve been forced to write 10+ page Google Docs defending their theories of change and the case that the intellectual culture simply wouldn’t have fostered this kind of thinking.
I think the focus on evals/RSPs can largely be explained by a bias toward trusting labs. OpenPhil steered a lot of talent toward the evals/RSPs theory of change (specifically, if I recall correctly, OpenPhil leadership on AI was especially influential in steering a lot of the ecosystem to support and invest in the evals/RSPs theory of change.) I expect that when we look back in a few years, there will be a pretty strong feeling that this was the wrong call & that this should’ve been more apparent even without the benefit of hindsight.
I would be more sympathetic to OpenPhil in a world where their aversion to weirdness/PR risks resulted in them having a strong reputation, a lot of political capital, and real-world influence that matched the financial resources they possess. Sadly, I think we’re in a “lose-lose” world: OpenPhil’s reputation tends to be poor in many policy/journalism circles even while OpenPhil pursues a strategy that seems to be largely focused on avoiding PR risks. I think some of this is unjustified (e.g., a result of propaganda campaigns designed to paint anyone who cares about AI risk as awful). But then some of it actually is kind of reasonable (e.g., impartial observers viewing OpenPhil as kind of shady, not direct in its communications, not very willing to engage directly or openly with policymakers or journalists, having lots of conflicts of interests, trying to underplay the extent to which its funding priorities are influenced/constrained by a single Billionaire, being pretty left-coded, etc.)
To defend OpenPhil a bit, I do think it’s quite hard to navigate trade-offs and I think sometimes people don’t seem to recognize these tradeoffs. In AI policy, I think the biggest tradeoff is something like “lots of people who have engaged with technical AGI arguments and AGI threat models don’t have policy experience, and lots of people who have policy experience don’t have technical expertise or experience engaging with AGI threat models” (this is a bit of an oversimplification– there are some shining stars who have both.)
I also think OpenPhil folks probably tend to have a different probability distribution over threat models (compared to me and probably also Habryka). For instance, it seems likely to me that OpenPhil employees operate in more of a “there are a lot of ways AGI could play out and a lot of uncertainty– we just need smart people thinking seriously about the problem. And who really know how hard alignment will be, maybe Anthropic will just figure it out” lens and less of a “ASI is coming and our priority needs to be making sure humanity understands the dangers associated with a reckless race toward ASI, and there’s a substantial chance that we are seriously not on track to solve the necessary safety and security challenges unless we fundamentally reorient our whole approach” lens.
And finally, I think despite these criticisms, OpenPhil is also responsible for some important wins (e.g., building the field, raising awareness about AGI risk on university campuses, funding some people early on before AI safety was a “big deal”, jumpstarting the careers of some leaders in the policy space [especially in the UK]. It’s also plausible to me that there are some cases in which OpenPhil gatekeeping was actually quite useful in preventing people from causing harm, even though I probably disagree with OpenPhil about the # and magnitude of these cases).
This should be a top-level post.
What are the norms here? Can I just copy/paste this exact text and put it into a top-level post? I got the sense that a top-level post should be more well thought out than this but I don’t actually have anything else useful to say. I would be happy to co-author a post if someone else thinks they can flesh it out.
Edit: Didn’t realize you were replying to Habryka, not me. That makes more sense.
It feels sorta understandable to me (albeit frustrating) that OpenPhil faces these assorted political constraints. In my view this seems to create a big unfilled niche in the rationalist ecosystem: a new, more right-coded, EA-adjacent funding organization could optimize itself for being able to enter many of those blacklisted areas with enthusiasm.
If I was a billionare, I would love to put together a kind of “completion portfolio” to complement some of OP’s work. Rationality community building, macrostrategy stuff, AI-related advocacy to try and influence republican politicians, plus a big biotechnology emphasis focused on intelligence enhancement, reproductive technologies, slowing aging, cryonics, gene drives for eradicating diseases, etc. Basically it seems like there is enough edgy-but-promising stuff out there (like studying geoengineering for climate, or advocating for charter cities, or just funding oddball substack intellectuals to do their thing) that you could hope to create a kind of “alt-EA” (obviously IRL it shouldn’t have EA in the name) where you batten down the hatches, accept that the media will call you an evil villain mastermind forever, and hope to create a kind of protective umbrella for all the work that can’t get done elsewhere. As a bonus, you could engage more in actual politics (like having some hot takes on the US budget deficit, or on how to increase marriage & fertility rates, or whatever), in some areas that OP in its quest for center-left non-polarization can’t do.
Peter Thiel already lives this life, kinda? But his model seems 1. much more secretive, and 2. less directly EA-adjacent, than what I’d try if I was a billionare.
Dustin himself talks about how he is really focused on getting more “multipolarity” to the EA landscape, by bringing in other high-net-worth funders. For all the reasons discussed, he obviously can’t say “hi, somebody please start an edgier right-wing offshoot of EA!!” But it seems like a major goal that the movement should have, nonetheless.
Seems like you could potentially also run this play with a more fully-left-coded organization. The gains there would probably be smaller, since there’s less “room” to OP’s left than to their right. But maybe you could group together wild animal welfare, invertebrate welfare, digital minds, perhaps some David Pearce / Project Far Out-style “suffering abolition” transhumanist stuff, other mental-wellbeing stuff like the Organization for the Prevention of Intense Suffering, S-risk work, etc. Toss in some more aggressive political activism on AI (like PauseAI) and other issues (like Georgist land value taxation), and maybe some forward-looking political stuff on avoiding stable totalitarianism, regulation of future AI-enabled technologies, and how to distribute the gains from a positive / successful singularity (akin to Sam Altman’s vision of UBI supported by georgist/pigouvian taxes, but more thought-through and detailed and up-to-date.)
Finding some funders to fill these niches seems like it should be a very high priority of the rationalist / EA movement. Even if the funders were relatively small at first (like say they have $10M - $100M in crypto that they are preparing to give away), I think there could be a lot of value in being “out and proud” (publicising much of their research and philosophy and grantmaking like OP, rather than being super-secretive like Peter Thiel). If a small funder manages to build a small successful “alt-EA” ecosystem on either the left or right, that might attract larger funders in time.
As context, note that OP funded ARC in March 2022.
I think OP has funded almost everyone I have listed here in 2022 (directly or indirectly), so I don’t really think that is evidence of anything (though it is a bit more evidence for ARC because it means the COI is overcomable).
Hm, this timing suggests the change could be a consequence of Karnofsky stepping away from the organization.
Which makes sense, now that I think about it. He’s by far the most politically strategic leader Open Philanthropy has had, so with him gone, it’s not shocking they might revert towards standard risk-averse optionality-maxxing foundation behavior.
Isn’t it just the case that OpenPhil just generally doesn’t fund that many technical AI safety things these days? If you look at OP’s team on their website, they have only two technical AI safety grantmakers. Also, you list all the things OP doesn’t fund, but what are the things in technical AI safety that they do fund? Looking at their grants, it’s mostly MATS and METR and Apollo and FAR and some scattered academics I mostly haven’t heard of. It’s not that many things. I have the impression that the story is less like “OP is a major funder in technical AI safety, but unfortunately they blacklisted all the rationalist-adjacent orgs and people” and more like “AI safety is still a very small field, especially if you only count people outside the labs, and there are just not that many exciting funding opportunities, and OpenPhil is not actually a very big funder in the field”.
A lot of OP’s funding to technical AI safety goes to people outside the main x-risk community (e.g. applications to Ajeya’s RFPs).
Open Phil is definitely by far the biggest funder in the field. I agree that their technical grantmaking has been a limited over the past few years (though still on the order of $50M/yr, I think), but they also fund a huge amount of field-building and talent-funnel work, as well as a lot of policy stuff (I wasn’t constraining myself to technical AI Safety, the people listed have been as influential, if not more, on public discourse and policy).
AI Safety is still relatively small, but more like $400M/yr small. The primary other employers/funders in the space these days are big capability labs. As you can imagine, their funding does not have great incentives either.
Yeah, I agree, and I don’t know that much about OpenPhil’s policy work, and their fieldbuilding seems decent to me, though maybe not from you perspective. I just wanted to flag that many people (including myself until recently) overestimate how big a funder OP is in technical AI safety, and I think it’s important to flag that they actually have pretty limited scope in this area.
Yep, agree that this is a commonly overlooked aspect (and one that I think sadly has also contributed to the dominant force in AI Safety researchers becoming the labs, which I think has been quite sad).
is there a list of these somewhere/details on what happened?
You can see some of the EA Forum discussion here: https://forum.effectivealtruism.org/posts/foQPogaBeNKdocYvF/linkpost-an-update-from-good-ventures?commentId=RQX56MAk6RmvRqGQt
The current list of areas that I know about are:
Anything to do with the rationality community (“Rationality community building”)
Anything to do with moral relevance of digital minds
Anything to do with wild animal welfare and invertebrate welfare
Anything to do with human genetic engineering and reproductive technology
Anything that is politically right-leaning
There are a bunch of other domains where OP hasn’t had an active grantmaking program but where my guess is most grants aren’t possible:
Most forms of broad public communication about AI (where you would need to align very closely with OP goals to get any funding)
Almost any form of macrostrategy work of the kind that FHI used to work on (i.e. Eternity in Six Hours and stuff like that)
Anything about acausal trade of cooperation in large worlds (and more broadly anything that is kind of weird game theory)
Huh, are there examples of right leaning stuff they stopped funding? That’s new to me
You said
I’m wondering if you have a list of organizations where Open Phil would have funded their other work, but because they withdrew from funding part of the organization they decided to withdraw totally.
This feels very importantly different from good ventures choosing not to fund certain cause areas (and I think you agree, which is why you put that footnote).
I don’t have a long list, but I know this is true for Lightcone, SPARC, ESPR, any of the Czech AI-Safety/Rationality community building stuff, and I’ve heard a bunch of stories since then from other organizations that got pretty strong hints from Open Phil that if they start working in an area at all, they might lose all funding (and also, the “yes, it’s more like a blacklist, if you work in these areas at all we can’t really fund you, though we might make occasional exceptions if it’s really only a small fraction of what you do” story was confirmed to me by multiple OP staff, so I am quite confident in this, and my guess is OP staff would be OK with confirming to you as well if you ask them).
Thanks!
Imo sacrificing a bunch of OpenPhil AI safety funding in exchange for improving OpenPhil’s ability to influence politics seems like a pretty reasonable trade to me, at least depending on the actual numbers. As an extreme case, I would sacrifice all current OpenPhil AI safety funding in exchange for OpenPhil getting to pick which major party wins every US presidential election until the singularity.
Concretely, the current presidential election seems extremely important to me from an AI safety perspective, I expect that importance to only go up in future elections, and I think OpenPhil is correct on what candidates are best from an AI safety perspective. Furthermore, I don’t think independent AI safety funding is that important anymore; models are smart enough now that most of the work to do in AI safety is directly working with them, most of that is happening at labs, and probably the most important other stuff to do is governance and policy work, which this strategy seems helpful for.
I don’t know the actual marginal increase in political influence that they’re buying here, but my guess would be that the numbers pencil and OpenPhil is making the right call.
Separately, this is just obviously false. A lot of the old AI safety people just don’t need OpenPhil funding anymore because they’re working at labs or governments, e.g. me, Rohin Shah, Geoffrey Irving, Jan Leike, Paul (as you mention), etc.
It might be the case that most of the quality weighted safety research involving working with large models is happening at labs, but I’m pretty skeptical that having this mostly happen at labs is the best approach and it seems like OpenPhil should be actively interested in building up a robust safety research ecosystem outside of labs.
(Better model access seems substantially overrated in its importance and large fractions of research can and should happen with just prompting or on smaller models. Additionally, at the moment, open weight models are pretty close to the best models.)
(This argument is also locally invalid at a more basic level. Just because this research seems to be mostly happening at large AI companies (which I’m also more skeptical of I think) doesn’t imply that this is the way it should be and funding should try to push people to do better stuff rather than merely reacting to the current allocation.)
Yeah, I think that’s a pretty fair criticism, but afaict that is the main thing that OpenPhil is still funding in AI safety? E.g. all the RFPs that they’ve been doing, I think they funded Jacob Steinhardt, etc. Though I don’t know much here; I could be wrong.
Wasn’t the relevant part of your argument like, “AI safety research outside of the labs is not that good, so that’s a contributing factor among many to it not being bad to lose the ability to do safety funding for governance work”? If so, I think that “most of OpenPhil’s actual safety funding has gone to building a robust safety research ecosystem outside of the labs” is not a good rejoinder to “isn’t there a large benefit to building a robust safety research ecosystem outside of the labs?”, because the rejoinder is focusing on relative allocations within “(technical) safety research”, and the complaint was about the allocation between “(technical) safety research” vs “other AI x-risk stuff”.
Yeah, I currently think Open Phil’s policy activism has been harmful for the world, and will probably continue to be, so by my lights this is causing harm with the justification of causing even more harm. I agree they will probably get the bit right about what major political party would be better, but sadly the effects of policy work are much more nuanced and detailed than that, and also they will have extremely little influence on who wins the general elections.
We could talk more about this sometime. I also have some docs with more of my thoughts here (which I maybe already shared with you, but would be happy to do so if not).
I genuinely don’t know whether Rohin would get funding to pursue what he thinks is most important, if he wanted it. I agree that some others don’t “need” funding anymore, though as I said, lab incentives are even worse on these dimensions and is of very little solace to me. I agree you might be able to get funding, though also see my other discussion with Eli on the boundaries I was trying to draw (which I agree are fuzzy and up-to-debate).
Sacrificing half of it to avoid things associated with one of the two major political parties and being deceptive about doing this is of course not equal to half the cost of sacrificing all of such funding, it is a much more unprincipled and distorting and actively deceptive decision that messes up everyone’s maps of the world in a massive way and reduces our ability to trust each other or understand what is happening.
Thanks for sharing, I was curious if you could elaborate on this (e.g. if there are examples of AI policy work funded by OP that come to mind that are clearly left of center). I am not familiar with policy, but my one data point is the Horizon Fellowship, which is non-partisan and intentionally places congressional fellows in both Democratic and Republican offices. This straightforwardly seems to me like a case where they are trying to engage with people on the right, though maybe you mean not-right-of-center at the organizational level? In general though, (in my limited exposure) I don’t model any AI governance orgs as having a particular political affiliation (which might just be because I’m uninformed / ignorant).
Yep, my model is that OP does fund things that are explicitly bipartisan (like, they are not currently filtering on being actively affiliated with the left). My sense is in-practice it’s a fine balance and if there was some high-profile thing where Horizon became more associated with the right (like maybe some alumni becomes prominent in the republican party and very publicly credits Horizon for that, or there is some scandal involving someone on the right who is a Horizon alumni), then I do think their OP funding would have a decent chance of being jeopardized, and the same is not true on the left.
Another part of my model is that one of the key things about Horizon is that they are of a similar school of PR as OP themselves. They don’t make public statements. They try to look very professional. They are probably very happy to compromise on messaging and public comms with Open Phil and be responsive to almost any request that OP would have messaging wise. That makes up for a lot. I think if you had a more communicative and outspoken organization with a similar mission to Horizon, I think the funding situation would be a bunch dicier (though my guess is if they were competent, an organization like that could still get funding).
More broadly, I am not saying “OP staff want to only support organizations on the left”. My sense is that many individual OP staff would love to fund more organizations on the right, and would hate for polarization to occur, but that organizationally and because of constraints by Dustin, they can’t, and so you will see them fund organizations that aim for more engagement with the right, but there will be relatively hard lines and constraints that will mostly prevent that.
Thanks for the reply. When I wrote “Many people would have more useful things to say about this than I do”, you were one of the people I was thinking of.
Related to this, I think GW/OP has always been too unwilling to fund weird causes, but it’s generally gotten better over time: originally recommending US charities over global poverty b/c global poverty was too weird, taking years to remove their recommendations for US charities that were ~100x less effective than their global poverty recs, then taking years to start funding animal welfare and x-risk, then still not funding weirder stuff like wild animal welfare and AI sentience. I’ve criticized them for this in the past but I liked that they were moving in the right direction. Now I get the sense that recently they’ve gotten worse on AI safety (and weird causes in general).
Nitpick, but this statement seems obviously false given what I understand your views to be? Paul, Carl, Buck, for starters.
[edit: I now see that Oliver had already made a footnote to that effect.]
(I like Buck, but he is one generation later than the one I was referencing. Also, I am currently like 50⁄50 whether Buck would indeed be blacklisted. I agree that Carl is a decent counterexample, though he is a bit of a weirder case)
I agree that I didn’t really have much of an effect on this community’s thinking about AIS until like 2021.
Jessica Taylor seems like she’s also second generation?
I remember running into her a bunch before I ran into Buck. Scott/Abram are also second generation. Overall, seems reasonable to include Buck (but communicating my more complicated epistemic state with regard to him would have been harder).
Out of curiosity—“it’s because Dustin is very active in the democratic party and doesn’t want to be affiliated with anything that is right-coded” Are these projects related to AI safety or just generally? And what are some examples?
I am not sure I am understanding your question. Are you asking about examples of left-leaning projects that Dustin is involved in, or right-leaning projects that cannot get funding? On the left, Dustin is one of the biggest donors to the democratic party (with Asana donating $45M and him donating $24M to Joe Biden in 2020).
Examples of right leaning projects that got rejected by him due to his political affiliation, and if these examples are AI safety related
I don’t currently know of any public examples and feel weird publicly disclosing details about organizations that I privately heard about. If more people are interested I can try to dig up some more concrete details (but can’t make any promises on things I’ll end up able sharing).
No worries; thanks!
Can you elaborate on what you mean by this?
OP appears to have been one of FHI’s biggest funders according to Sandberg:[1]
The hiring (and fundraising) freeze imposed by Oxf began in 2020.
See page 15
In 2023/2024 OP drastically changed it’s funding process and priorities (in part in response to FTX, in part in response to Dustin’s preferences). This whole conversation is about the shift in OPs giving in this recent time period.
See also: https://forum.effectivealtruism.org/posts/foQPogaBeNKdocYvF/linkpost-an-update-from-good-ventures
I agree with the claim you’re making: that if FHI still existed and they applied for a grant from OP it would be rejected. This seems true to me.
I don’t mean to nitpick, but it still feels misleading to claim “FHI could not get OP funding” when they did in fact get lots of funding from OP. It implies that FHI operated without any help from OP, which isn’t true.
The “could” here is (in context) about “could not get funding from modern OP”. The whole point of my comment was about the changes that OP underwent. Sorry if that wasn’t as clear, it might not be as obvious to others that of course OP was very different in the past.
I understand the claim you were making now and I hope the nitpicking isn’t irritable.
If Open Phil is unwilling to fund some/most of the best orgs, that makes earning to give look more compelling.
(There are some other big funders in AI safety like Jaan Tallinn, but I think all of them combined still have <10% as much money as Open Phil.)