Are you imagining that a company develops AGI, starts becoming more and more powerful, and after 10 years of being confused and disoriented the government says “you’re too big, you need to be broken up” and the company says “no” and takes over the government?
Sort of, but worse – I’m imagining something more like “the government has already has a lot of regulatory capture going on, so the system-as-is is already fairly broken. Even given slow-ish takeoff assumptions, it seems like within 2-3 years there will either be one-or-several-companies that have gained unprecedented amounts of power. And by the time the government has even figured out an action to take, it will either have already been taken over, regulatory-captured in ways much deeper than previously, or rendered irrelevant.”
Okay, I see, that makes sense and seems plausible, though I’d bet against it happening. But you’ve convinced me that I should qualify that sentence more.
Sort of, but worse – I’m imagining something more like “the government has already has a lot of regulatory capture going on, so the system-as-is is already fairly broken. Even given slow-ish takeoff assumptions, it seems like within 2-3 years there will either be one-or-several-companies that have gained unprecedented amounts of power. And by the time the government has even figured out an action to take, it will either have already been taken over, regulatory-captured in ways much deeper than previously, or rendered irrelevant.”
Okay, I see, that makes sense and seems plausible, though I’d bet against it happening. But you’ve convinced me that I should qualify that sentence more.