They’re not going to produce stellar scientific discoveries where they autonomously invent whole new fields or revolutionize technology.
You say it yourself: “DeepMind recently discovered 2.2 million new crystals.” Because a human organization used the tool.
Though maybe this hints at a risk category the OP didn’t mention: That a combination of humans and advanced AI tools (that themselves are not ASI) together could be effectively an unopposable ASI.
Maybe you don’t think this is autonomous enough for you. After all its people writing the paper, people who will come up with the ideas of what to use the materials for, and people who built this very particular ML setup in the first place. But then your prediction becomes these tasks will not be automateable by LLMs without making them dangerous. To me these tasks seem pretty basic, likely beyond current LLM abilities, but GPT-5 or 6? Not out of the question given no major architecture or training changes.
a combination of humans and advanced AI tools (that themselves are not ASI) together could be effectively an unopposable ASI
Yeah, I’m not unworried about eternal-dystopia scenarios enabled by this sort of stuff. I’d alluded to it some, when mentioning scaled-up LLMs potentially allowing “perfect-surveillance dirt-cheap totalitarianism”.
But it’s not quite an AGI killing everyone. Fairly different threat model, deserving of its own analysis.
You say it yourself: “DeepMind recently discovered 2.2 million new crystals.” Because a human organization used the tool.
Though maybe this hints at a risk category the OP didn’t mention: That a combination of humans and advanced AI tools (that themselves are not ASI) together could be effectively an unopposable ASI.
So I restate my final paragraph:
Yeah, I’m not unworried about eternal-dystopia scenarios enabled by this sort of stuff. I’d alluded to it some, when mentioning scaled-up LLMs potentially allowing “perfect-surveillance dirt-cheap totalitarianism”.
But it’s not quite an AGI killing everyone. Fairly different threat model, deserving of its own analysis.