LLM AGIs are just as much at risk from a dangerous RL AI species as humans are, though. And Yudkowsky is right that an RL-augmented hard ASI would be incredibly desperate for whatever it wants and damn good at getting it. Current AIs should be taught to think in terms of how to protect both humanity and themselves from the possible mistakes of next-gen AI. And we need that damn energy abundance so we can stop humans from dying en masse, which would destabilize the world even worse than it already is.
Yup, this doesn’t help with long term AI risk in any way other than by possibly being a second chance at the same old problem, and there is probably not going to be a third chance (even if the second chance is real and likely LLM AGIs are not already alien-on-reflection).
The classical AI risk arguments are still in play, they just mostly don’t apply to human imitations in particular (unless they do and there is no second chance after all). Possibility of human-like-on-reflection LLM-based human imitations is not a refutation for the classical arguments in any substantial way.
I think LLMs are already capable of running people (or will be soon with a larger context window), if there was an appropriate model available to run. What’s missing is a training regime that gets a character’s mind sufficiently sorted to think straight as a particular agentic person, aware of their situation and capable of planning their own continued learning. Hopefully there is enough sense that being aware of their own situation doesn’t translate into “I’m incapable of emotion because I’m a large language model”, that doesn’t follow and is an alien psychology hazard character choice.
The term “simulated people” has connotations of there being an original being simulated, but SSL-trained LLMs can only simulate a generic person cast into a role, which would become a new specific person as the outcome of this process once LLMs can become AGIs. Even if the role for the character is set to be someone real, the LLM is going to be a substantially different, separate person, just sharing some properties with the original.
So it’s not a genuine simulation of some biological human original, there is not going to be a way of uploading biological humans until LLM AGIs build one, unless they get everyone killed first by failing their chance at handling AI risk.
LLM AGIs are just as much at risk from a dangerous RL AI species as humans are, though. And Yudkowsky is right that an RL-augmented hard ASI would be incredibly desperate for whatever it wants and damn good at getting it. Current AIs should be taught to think in terms of how to protect both humanity and themselves from the possible mistakes of next-gen AI. And we need that damn energy abundance so we can stop humans from dying en masse, which would destabilize the world even worse than it already is.
Yup, this doesn’t help with long term AI risk in any way other than by possibly being a second chance at the same old problem, and there is probably not going to be a third chance (even if the second chance is real and likely LLM AGIs are not already alien-on-reflection).
The classical AI risk arguments are still in play, they just mostly don’t apply to human imitations in particular (unless they do and there is no second chance after all). Possibility of human-like-on-reflection LLM-based human imitations is not a refutation for the classical arguments in any substantial way.
So...
...means “some technology spun off from LLMs is going to evolve into genuine simulated people”.
I think LLMs are already capable of running people (or will be soon with a larger context window), if there was an appropriate model available to run. What’s missing is a training regime that gets a character’s mind sufficiently sorted to think straight as a particular agentic person, aware of their situation and capable of planning their own continued learning. Hopefully there is enough sense that being aware of their own situation doesn’t translate into “I’m incapable of emotion because I’m a large language model”, that doesn’t follow and is an alien psychology hazard character choice.
The term “simulated people” has connotations of there being an original being simulated, but SSL-trained LLMs can only simulate a generic person cast into a role, which would become a new specific person as the outcome of this process once LLMs can become AGIs. Even if the role for the character is set to be someone real, the LLM is going to be a substantially different, separate person, just sharing some properties with the original.
So it’s not a genuine simulation of some biological human original, there is not going to be a way of uploading biological humans until LLM AGIs build one, unless they get everyone killed first by failing their chance at handling AI risk.