In the interest of saying more things publicly on this, some relevant thoughts:
I don’t know what it means for somebody to be a “doomer,” but I used to be a MIRI researcher and I am pretty publicly one of the people with the highest probabilities of unconditional AI existential risk (I usually say ~80%).
When Conjecture was looking for funding initially to get started, I was one of the people who was most vocal in supporting them. In particular, before FTX collapsed, I led FTX regranting towards Conjecture, and recommended an allocation in the single-digit millions.
I no longer feel excited about Conjecture. I view a lot of the stuff they’re doing as net-negative and I wouldn’t recommend anyone work there anymore. Almost all of the people at Conjecture initially that I really liked have left (e.g. they fired janus, their interpretability people left for Apollo, etc.), the concrete CoEms stuff they’re doing now doesn’t seem exciting to me, and I think their comms efforts have actively hurt our ability to enact good AI policy. In particular, I think their usage of Dario’s statements on x-risk as a rhetorical weapon against RSPs creates a structural disincentive against lab heads being clear about existential risk and reduces the probability of us getting good RSPs from other labs and good RSP-based regulation.
In particular, I think their usage of Dario’s statements on x-risk as a rhetorical weapon against RSPs creates a structural disincentive against lab heads being clear about existential risk
I’m not sure how to articulate this, exactly, but I want to say something like “it’s not on us to make sure the incentives line up so that lab heads state their true beliefs about the amount of risk they’re putting the entire world in.” Stating their beliefs is just something they should be doing, on a matter this important, no matter the consequences. That’s on them. The counterfactual world—where they keep quiet or are unclear in order to hide their true (and alarming) beliefs about the harm they might impose on everyone—is deceptive. And it is indeed pretty unfortunate that the people who are most clear about this (such as Dario), will get the most pushback. But if people are upset about what they’re saying, then they should still be getting the pushback.
When I was an SRE at Google, we had a motto that I really like, which is: “hope is not a strategy.” It would be nice if all the lab heads would be perfectly honest here, but just hoping for that to happen is not an actual strategy.
Furthermore, I would say that I see the main goal of outside-game advocacy work as setting up external incentives in such a way that pushes labs to good things rather than bad things. Either through explicit regulation or implicit pressure, I think controlling the incentives is absolutely critical and the main lever that you have externally for controlling the actions of large companies.
I don’t think aysja was endorsing “hope” as a strategy– at least, that’s not how I read it. I read it as “we should hold leaders accountable and make it clear that we think it’s important for people to state their true beliefs about important matters.”
To be clear, I think it’s reasonable for people to discuss the pros and cons of various advocacy tactics, and I think asking “to what extent do I expect X advocacy tactic will affect peoples’ incentives to openly state their beliefs?” makes sense.
Separately, though, I think the “accountability frame” is important. Accountability can involve putting pressure on them to express their true beliefs, pushing back when we suspect people are trying to find excuses to hide their beliefs, and making it clear that we think openness and honesty are important virtues even when they might provoke criticism– perhaps especially when they might provoke criticism. I think this is especially important in the case of lab leaders and others who have clear financial interests or power interests in the current AGI development ecosystem.
It’s not about hoping that people are honest– it’s about upholding standards of honesty, and recognizing that we have some ability to hold people accountable if we suspect that they’re not being honest.
I would say that I see the main goal of outside-game advocacy work as setting up external incentives in such a way that pushes labs to good things rather than bad things
I’m currently most excited about outside-game advocacy that tries to get governments to implement regulations that make good things happen. I think this technically falls under the umbrella of “controlling the incentives through explicit regulation”, but I think it’s sufficiently different from outside-game advocacy work that is trying to get labs to do things voluntarily.
Setting aside my personal models of Connor/Gabe/etc, the only way this action reads to me as making sense if one feels compelled to go all in on “so-called Responsible Scaling Policies are primarily a fig leaf of responsibility from ML labs, as the only viable responsible option is to regulate them / shut them down”. I assign at least 10% to that perspective being accurate, so I am not personally ruling it out as a fine tactic.
I agree it is otherwise disincentivizing[1] in worlds where open discussion and publication of scaling policies (even I cannot bring myself to calling them ‘responsible’) is quite reasonable.
Probably Evan/others agree with this, but I want to explicitly point out that the CEOs of the labs such as Amodei and Altman and Hassabis should answer the question honestly regardless of how it’s used by those they’re in conflict with, the matter is too important for it to be forgivable that they would otherwise be strategically avoidant in order to prop up their businesses.
There are all kinds of benefits to acting with good faith, and people should not feel licensed to abandon good faith dialogue just because they’re SUPER confident and this issue is REALLY IMPORTANT.
When something is really serious it becomes even more important to do boring +EV things like “remember that you can be wrong sometimes” and “don’t take people’s quotes out of context, misrepresent their position, and run smear campaigns on them; and definitely don’t make that your primary contribution to the conversation”.
Like, for Connor & people who support him (not saying this is you Ben): don’t you think it’s a little bit suspicious that you ended up in a place where you concluded that the very best use of your time in helping with AI risk was tweet-dunking and infighting among the AI safety community?
In the interest of saying more things publicly on this, some relevant thoughts:
I don’t know what it means for somebody to be a “doomer,” but I used to be a MIRI researcher and I am pretty publicly one of the people with the highest probabilities of unconditional AI existential risk (I usually say ~80%).
When Conjecture was looking for funding initially to get started, I was one of the people who was most vocal in supporting them. In particular, before FTX collapsed, I led FTX regranting towards Conjecture, and recommended an allocation in the single-digit millions.
I no longer feel excited about Conjecture. I view a lot of the stuff they’re doing as net-negative and I wouldn’t recommend anyone work there anymore. Almost all of the people at Conjecture initially that I really liked have left (e.g. they fired janus, their interpretability people left for Apollo, etc.), the concrete CoEms stuff they’re doing now doesn’t seem exciting to me, and I think their comms efforts have actively hurt our ability to enact good AI policy. In particular, I think their usage of Dario’s statements on x-risk as a rhetorical weapon against RSPs creates a structural disincentive against lab heads being clear about existential risk and reduces the probability of us getting good RSPs from other labs and good RSP-based regulation.
Unlike Conjecture’s comms efforts, I’ve been really happy with MIRI’s comms efforts. I thought Eliezer’s Time article was great, I really liked Nate’s analysis of the different labs’ policies, etc.
I’m not sure how to articulate this, exactly, but I want to say something like “it’s not on us to make sure the incentives line up so that lab heads state their true beliefs about the amount of risk they’re putting the entire world in.” Stating their beliefs is just something they should be doing, on a matter this important, no matter the consequences. That’s on them. The counterfactual world—where they keep quiet or are unclear in order to hide their true (and alarming) beliefs about the harm they might impose on everyone—is deceptive. And it is indeed pretty unfortunate that the people who are most clear about this (such as Dario), will get the most pushback. But if people are upset about what they’re saying, then they should still be getting the pushback.
When I was an SRE at Google, we had a motto that I really like, which is: “hope is not a strategy.” It would be nice if all the lab heads would be perfectly honest here, but just hoping for that to happen is not an actual strategy.
Furthermore, I would say that I see the main goal of outside-game advocacy work as setting up external incentives in such a way that pushes labs to good things rather than bad things. Either through explicit regulation or implicit pressure, I think controlling the incentives is absolutely critical and the main lever that you have externally for controlling the actions of large companies.
I don’t think aysja was endorsing “hope” as a strategy– at least, that’s not how I read it. I read it as “we should hold leaders accountable and make it clear that we think it’s important for people to state their true beliefs about important matters.”
To be clear, I think it’s reasonable for people to discuss the pros and cons of various advocacy tactics, and I think asking “to what extent do I expect X advocacy tactic will affect peoples’ incentives to openly state their beliefs?” makes sense.
Separately, though, I think the “accountability frame” is important. Accountability can involve putting pressure on them to express their true beliefs, pushing back when we suspect people are trying to find excuses to hide their beliefs, and making it clear that we think openness and honesty are important virtues even when they might provoke criticism– perhaps especially when they might provoke criticism. I think this is especially important in the case of lab leaders and others who have clear financial interests or power interests in the current AGI development ecosystem.
It’s not about hoping that people are honest– it’s about upholding standards of honesty, and recognizing that we have some ability to hold people accountable if we suspect that they’re not being honest.
I’m currently most excited about outside-game advocacy that tries to get governments to implement regulations that make good things happen. I think this technically falls under the umbrella of “controlling the incentives through explicit regulation”, but I think it’s sufficiently different from outside-game advocacy work that is trying to get labs to do things voluntarily.
Setting aside my personal models of Connor/Gabe/etc, the only way this action reads to me as making sense if one feels compelled to go all in on “so-called Responsible Scaling Policies are primarily a fig leaf of responsibility from ML labs, as the only viable responsible option is to regulate them / shut them down”. I assign at least 10% to that perspective being accurate, so I am not personally ruling it out as a fine tactic.
I agree it is otherwise disincentivizing[1] in worlds where open discussion and publication of scaling policies (even I cannot bring myself to calling them ‘responsible’) is quite reasonable.
Probably Evan/others agree with this, but I want to explicitly point out that the CEOs of the labs such as Amodei and Altman and Hassabis should answer the question honestly regardless of how it’s used by those they’re in conflict with, the matter is too important for it to be forgivable that they would otherwise be strategically avoidant in order to prop up their businesses.
There are all kinds of benefits to acting with good faith, and people should not feel licensed to abandon good faith dialogue just because they’re SUPER confident and this issue is REALLY IMPORTANT.
When something is really serious it becomes even more important to do boring +EV things like “remember that you can be wrong sometimes” and “don’t take people’s quotes out of context, misrepresent their position, and run smear campaigns on them; and definitely don’t make that your primary contribution to the conversation”.
Like, for Connor & people who support him (not saying this is you Ben): don’t you think it’s a little bit suspicious that you ended up in a place where you concluded that the very best use of your time in helping with AI risk was tweet-dunking and infighting among the AI safety community?