Note also that it will probably be easier to act cautiously if you don’t have to be constantly in negotiations with an escaped scheming AI that is currently working on becoming more powerful, perhaps attacking you with bioweapons, etc!
Hmm, when I imagine “Scheming AI that is not easy to shut down with concerted nation-state effort, are attacking you with bioweapons, but are weak enough such that you can bargain/negotiate with them” I can imagine this outcome inspiring a lot more caution relative to many other worlds where control techniques work well but we can’t get any convincing demos/evidence to inspire caution (especially if control techniques inspire overconfidence).
But the ‘is currently working on becoming more powerful’ part of your statement does carry a lot of weight.
Note also that it will probably be easier to act cautiously if you don’t have to be constantly in negotiations with an escaped scheming AI that is currently working on becoming more powerful, perhaps attacking you with bioweapons, etc!
Hmm, when I imagine “Scheming AI that is not easy to shut down with concerted nation-state effort, are attacking you with bioweapons, but are weak enough such that you can bargain/negotiate with them” I can imagine this outcome inspiring a lot more caution relative to many other worlds where control techniques work well but we can’t get any convincing demos/evidence to inspire caution (especially if control techniques inspire overconfidence).
But the ‘is currently working on becoming more powerful’ part of your statement does carry a lot of weight.
People will sure be scared of AI, but the arms race pressure will be very strong, and I think that is a bigger consideration