If you look over your bullet list and then ask “is this a capability groups training proto-AGI in sims (like DM) already have today?” or “are likely to have by the time for AGI?”, I get yes for all basically.
So to the extent the safe path here differs from business as usual, it seems to be around:
1.) coordinating on standardized sim environments where AGI teams can still compete over model design while coordinating on shared sim sandbox and safety evaluation infrastructure.
and 2.) advocating the importance of leakage prevention/data isolation as we approach AGI (isolating the sim environment so that it doesn’t reveal too much about our world).
For an example of 2, when Carmack was talking about his AGI approach he ponders whether AGI’s will need a full sim with virtual tv screens or whether you can skip the sim and just hook the tv/browser directly to their input. It’s clear there he isn’t really considering the danger of letting potentially future dangerous AGI read the internet.
So long as the AI retains no modifications or artifacts from the evaluation, then it can’t learn from it, and thus it should be safe to make the simulation as accurate as possible. And I agree that some things like this are already sort of happening, but I think not with the evaluation being focused on the most dangerous paths (agentive strategic planning to acquire resources, self-modification to increase abilities, deceptive manipulation to avoid detection and run cons, etc.). And not in a widespread consistent way, where all papers published have a little note saying that the model passed it’s standardized safety evals.
If the AI learns it is in a sim that could completely undermine or invalidate any evaluations of it’s ethical/moral/altruistic behavior. I am assuming that the agent’s entire life and education training process is thus an evaluation. The sim can be ‘accurate’, it just needs to be knowledge constrained. A medieval tech era sim would be fine for example.
If you look over your bullet list and then ask “is this a capability groups training proto-AGI in sims (like DM) already have today?” or “are likely to have by the time for AGI?”, I get yes for all basically.
So to the extent the safe path here differs from business as usual, it seems to be around: 1.) coordinating on standardized sim environments where AGI teams can still compete over model design while coordinating on shared sim sandbox and safety evaluation infrastructure. and 2.) advocating the importance of leakage prevention/data isolation as we approach AGI (isolating the sim environment so that it doesn’t reveal too much about our world).
For an example of 2, when Carmack was talking about his AGI approach he ponders whether AGI’s will need a full sim with virtual tv screens or whether you can skip the sim and just hook the tv/browser directly to their input. It’s clear there he isn’t really considering the danger of letting potentially future dangerous AGI read the internet.
So long as the AI retains no modifications or artifacts from the evaluation, then it can’t learn from it, and thus it should be safe to make the simulation as accurate as possible. And I agree that some things like this are already sort of happening, but I think not with the evaluation being focused on the most dangerous paths (agentive strategic planning to acquire resources, self-modification to increase abilities, deceptive manipulation to avoid detection and run cons, etc.). And not in a widespread consistent way, where all papers published have a little note saying that the model passed it’s standardized safety evals.
If the AI learns it is in a sim that could completely undermine or invalidate any evaluations of it’s ethical/moral/altruistic behavior. I am assuming that the agent’s entire life and education training process is thus an evaluation. The sim can be ‘accurate’, it just needs to be knowledge constrained. A medieval tech era sim would be fine for example.