I am not covering training setups where we purposefully train an AI to be agentic and autonomous. I just think it’s not plausible that we just keep scaling up networks, run pretraining + light RLHF, and then produce a schemer.[2]
Like Ryan, I’m interested in how much of this claim is conditional on “just keep scaling up networks” being insufficient to produce relevantly-superhuman systems (i.e. systems capable of doing scientific R&D better and faster than humans, without humans in the intellectual part of the loop). If it’s “most of it”, then my guess is that accounts for a good chunk of the disagreement.
I don’t expect the current paradigm will be insufficient (though it seems totally possible). Off the cuff I expect 75% that something like the current paradigm will be sufficient, with some probability that something else happens first. (Note that “something like the current paradigm” doesn’t just involve scaling up networks.)
Like Ryan, I’m interested in how much of this claim is conditional on “just keep scaling up networks” being insufficient to produce relevantly-superhuman systems (i.e. systems capable of doing scientific R&D better and faster than humans, without humans in the intellectual part of the loop). If it’s “most of it”, then my guess is that accounts for a good chunk of the disagreement.
I don’t expect the current paradigm will be insufficient (though it seems totally possible). Off the cuff I expect 75% that something like the current paradigm will be sufficient, with some probability that something else happens first. (Note that “something like the current paradigm” doesn’t just involve scaling up networks.)