How can we solve that coordination problem? I have yet to hear a workable idea.
We agree that far, then! I just don’t think that’s a workable strategy (you also didn’t state that big assumption in your post—that AGI is still dangerous as hell, we just have a route to really useful AI that isn’t).
The problem is that we don’t know whether agents based on LLMs are alignable. We don’t have enough people working on the conjunction of LLM/deep nets and real AGI. So everyone building it is going to optmistically assume it’s alignable. The Yudkowsky et al arguments for alignment being very difficult are highly incomplete; they aren’t convincing because they shouldn’t be. But they make good points.
If we refuse to think about aligning AGI LLM architectures because it sounds risky, it seems pretty certain that people will try it without our help. Even convincing them not to would require grappling in depth with why alignment would or wouldn’t work for that type of AGI.
How can we solve that coordination problem? I have yet to hear a workable idea.
We agree that far, then! I just don’t think that’s a workable strategy (you also didn’t state that big assumption in your post—that AGI is still dangerous as hell, we just have a route to really useful AI that isn’t).
The problem is that we don’t know whether agents based on LLMs are alignable. We don’t have enough people working on the conjunction of LLM/deep nets and real AGI. So everyone building it is going to optmistically assume it’s alignable. The Yudkowsky et al arguments for alignment being very difficult are highly incomplete; they aren’t convincing because they shouldn’t be. But they make good points.
If we refuse to think about aligning AGI LLM architectures because it sounds risky, it seems pretty certain that people will try it without our help. Even convincing them not to would require grappling in depth with why alignment would or wouldn’t work for that type of AGI.
This is my next project!