I disagree with some of the claims made here, and I think there several worldview assumptions that go into a lot of these claims. Examples include things like “what do we expect the trajectory to ASI to look like”, “how much should we worry about AI takeover risks”, “what happens if a single actor ends up controlling [aligned] ASI”, “what kinds of regulations can we reasonably expect absent some sort of centralized USG project”, and “how much do we expect companies to race to the top on safety absent meaningful USG involvement.” (TBC though I don’t think it’s the responsibility of the authors to go into all of these background assumptions– I think it’s good for people to present claims like this even if they don’t have time/space to give their Entire Model of Everything.)
Nonetheless, I agree with the bottom-line conclusion: on the margin, I suspect it’s more valuable for people to figure out how to make different worlds go well than to figure out which “world” is better. In other words, asking “how do I make Centralized World or Noncentralized World more likely to go well” rather than “which one is better: Centralized World or Noncentralized World?”
More specifically, I think more people should be thinking: “Assume the USG decides to centralize AGI development or pursue some sort of AGI Manhattan Project. At that point, the POTUS or DefSec calls you in and asks you if you have any suggestions for how to maximize the chance of this going well. What do you say?”
One part of my rationale: the decisions about whether or not to centralize will be much harder to influence than decisions about what particular kind of centralized model to go with or what the implementation details of a centralized project should look like. I imagine scenarios in which the “whether to centralize” decision is largely a policy decision that the POTUS and the POTUS’s close advisors make, whereas the decision of “how do we actually do this” is something that would be delegated to people lower down the chain (who are both easier to access and more likely to be devoting a lot of time to engaging with arguments about what’s desirable.)
I disagree with some of the claims made here, and I think there several worldview assumptions that go into a lot of these claims. Examples include things like “what do we expect the trajectory to ASI to look like”, “how much should we worry about AI takeover risks”, “what happens if a single actor ends up controlling [aligned] ASI”, “what kinds of regulations can we reasonably expect absent some sort of centralized USG project”, and “how much do we expect companies to race to the top on safety absent meaningful USG involvement.” (TBC though I don’t think it’s the responsibility of the authors to go into all of these background assumptions– I think it’s good for people to present claims like this even if they don’t have time/space to give their Entire Model of Everything.)
Nonetheless, I agree with the bottom-line conclusion: on the margin, I suspect it’s more valuable for people to figure out how to make different worlds go well than to figure out which “world” is better. In other words, asking “how do I make Centralized World or Noncentralized World more likely to go well” rather than “which one is better: Centralized World or Noncentralized World?”
More specifically, I think more people should be thinking: “Assume the USG decides to centralize AGI development or pursue some sort of AGI Manhattan Project. At that point, the POTUS or DefSec calls you in and asks you if you have any suggestions for how to maximize the chance of this going well. What do you say?”
One part of my rationale: the decisions about whether or not to centralize will be much harder to influence than decisions about what particular kind of centralized model to go with or what the implementation details of a centralized project should look like. I imagine scenarios in which the “whether to centralize” decision is largely a policy decision that the POTUS and the POTUS’s close advisors make, whereas the decision of “how do we actually do this” is something that would be delegated to people lower down the chain (who are both easier to access and more likely to be devoting a lot of time to engaging with arguments about what’s desirable.)