I like this analysis, and I agree with except that I do think it’s missing a likely intermediate scenario.
I think the “fully under lab control” is a super advantageous situation for the humans, especially if the AI has been trained on censored simulation data that doesn’t mention humans or computers or have accurate physics.
I think the current world has an unfortunately dangerous intermediate situation where LLMs age given full access to human knowledge, and allowed to interact with society. And yet, in the case of the SotA models like GPT-4, aren’t quite at “loose in the world” levels of freedom. They don’t have access to their own weights or source code and neither do any accomplices they might recruit outside the company. Indeed, even most employees at the company couldn’t exfiltrate the weights. Thus, the current default starting state for a rogue AI is posed right on that dangerous margin of “difficult but not impossible to escape”. I think this “brains vs brawn” style analysis does then make a big difference for the initial escape.
I agree that once the escape has been accomplished it’s really hard for humanity to claw back a win. But before the escape has occurred, it’s a much more even game.
I like this analysis, and I agree with except that I do think it’s missing a likely intermediate scenario. I think the “fully under lab control” is a super advantageous situation for the humans, especially if the AI has been trained on censored simulation data that doesn’t mention humans or computers or have accurate physics. I think the current world has an unfortunately dangerous intermediate situation where LLMs age given full access to human knowledge, and allowed to interact with society. And yet, in the case of the SotA models like GPT-4, aren’t quite at “loose in the world” levels of freedom. They don’t have access to their own weights or source code and neither do any accomplices they might recruit outside the company. Indeed, even most employees at the company couldn’t exfiltrate the weights. Thus, the current default starting state for a rogue AI is posed right on that dangerous margin of “difficult but not impossible to escape”. I think this “brains vs brawn” style analysis does then make a big difference for the initial escape. I agree that once the escape has been accomplished it’s really hard for humanity to claw back a win. But before the escape has occurred, it’s a much more even game.