However, after looking back on it more than four years later, I think the general picture it gave missed some crucial details about how AI will go.
I feel like this is understating things a bit.
In my view (Drexler probably disagrees?), there are two important parts of CAIS:
Most sectors of the economy are under human supervision and iteratively reduce human oversight as trust is gradually built in engineered subsystems (which are understood ‘well enough’).
The existing economy is slightly more sophisticated than what SOTA general AI can produce, and so there’s not much free energy for rogue AI systems to eat or to attract investment in creating general AI. (General AI is dangerous, and the main goal of CAIS is to get the benefits without the drawbacks.)
I think a ‘foundation model’ world probably wrecks both. I think they might be recoverable—and your post goes some of the way to making that visualizable to me—but it still doesn’t seem like the default outcome.
[In particular, I like the point about models with broad world models can still have narrow responsibilities, and think that likely makes them more likely to be safe, at least in the medium term. Having one global moral/law-abiding foundational AI model that many people then slot into their organizations seems way better than everyone training whatever AI model they need for their use case.]
I feel like this is understating things a bit.
In my view (Drexler probably disagrees?), there are two important parts of CAIS:
Most sectors of the economy are under human supervision and iteratively reduce human oversight as trust is gradually built in engineered subsystems (which are understood ‘well enough’).
The existing economy is slightly more sophisticated than what SOTA general AI can produce, and so there’s not much free energy for rogue AI systems to eat or to attract investment in creating general AI. (General AI is dangerous, and the main goal of CAIS is to get the benefits without the drawbacks.)
I think a ‘foundation model’ world probably wrecks both. I think they might be recoverable—and your post goes some of the way to making that visualizable to me—but it still doesn’t seem like the default outcome.
[In particular, I like the point about models with broad world models can still have narrow responsibilities, and think that likely makes them more likely to be safe, at least in the medium term. Having one global moral/law-abiding foundational AI model that many people then slot into their organizations seems way better than everyone training whatever AI model they need for their use case.]