I found the argument compelling, but if I put on my techno-optimist ML researcher hat I think the least believable part of the story is the deployment:
relatively shortly after deployment, Magma’s datacenter would essentially contain a populous “virtual civilization” running ahead of human civilization in its scientific and technological sophistication
It’s hard to imagine that this is the way Alex would be deployed. BigTech executives are already terrified of deploying large-scale open-ended AI models with impacts on the real world, due to liability and PR risk. Faced with an AI system this powerful they would be under enormous pressure, both external and internal, to shut the system down, improve its interpretability, or deploy it incrementally while carefully monitoring for unintended consequences.
Not that this pressure will necessarily prevent takeover if Alex is sufficiently patient and deceptive, but I think it would help to flesh out why we expect humanity not to react appropriately to the terrifying trend of “understanding and control of Alex’s actions becoming looser and more tenuous after deployment”. Maybe the crux is that I expect the relevant institutions to be extremely risk-averse, and happily forego “dramatically important humanitarian, economic, and military benefits” to remove the chance that they will be held accountable for the downside.
I found the argument compelling, but if I put on my techno-optimist ML researcher hat I think the least believable part of the story is the deployment:
It’s hard to imagine that this is the way Alex would be deployed. BigTech executives are already terrified of deploying large-scale open-ended AI models with impacts on the real world, due to liability and PR risk. Faced with an AI system this powerful they would be under enormous pressure, both external and internal, to shut the system down, improve its interpretability, or deploy it incrementally while carefully monitoring for unintended consequences.
Not that this pressure will necessarily prevent takeover if Alex is sufficiently patient and deceptive, but I think it would help to flesh out why we expect humanity not to react appropriately to the terrifying trend of “understanding and control of Alex’s actions becoming looser and more tenuous after deployment”. Maybe the crux is that I expect the relevant institutions to be extremely risk-averse, and happily forego “dramatically important humanitarian, economic, and military benefits” to remove the chance that they will be held accountable for the downside.