This post was oriented around the goal of “be ready to safely train and deploy a powerful AI”. I felt like I could make the case for that fairly straightforwardly, mostly within the paradigm that I expect many AI labs are operating under.
But one of the reasons I think it’s important to have a strong culture of safety/carefulness, is in the leadup to strong AI. I think the world is going to be changing rapidly, and that means your organization may need to change strategies quickly, and track your impact on various effects on society.
Some examples of problems you might need to address:
Accidentally accelerating race dynamics (even if you’re really careful to avoid hyping and demonstrating new capabilities publicly, if it’s clear that you’re working on fairly advanced stuff that’ll likely get released someday it can still contribute to FOMO)
Failing to capitalize on opportunities to reduce race dynamics, which you might not have noticed due to distraction or pressures from within your company
Publishing some research that turned out to be more useful for capabilities than you thought.
Employees leaving and taking insights with them to other less careful orgs (in theory you can have noncompete clauses that mediate this, but I’m skeptical about how that works out in practice)
AIs interacting with society, or each other, in ways that destabilize humanity.
I’m working on a longer comment (which will maybe turn out to just be a whole post) about this, but wanted to flag it here for now.
This post was oriented around the goal of “be ready to safely train and deploy a powerful AI”. I felt like I could make the case for that fairly straightforwardly, mostly within the paradigm that I expect many AI labs are operating under.
But one of the reasons I think it’s important to have a strong culture of safety/carefulness, is in the leadup to strong AI. I think the world is going to be changing rapidly, and that means your organization may need to change strategies quickly, and track your impact on various effects on society.
Some examples of problems you might need to address:
Accidentally accelerating race dynamics (even if you’re really careful to avoid hyping and demonstrating new capabilities publicly, if it’s clear that you’re working on fairly advanced stuff that’ll likely get released someday it can still contribute to FOMO)
Failing to capitalize on opportunities to reduce race dynamics, which you might not have noticed due to distraction or pressures from within your company
Publishing some research that turned out to be more useful for capabilities than you thought.
Employees leaving and taking insights with them to other less careful orgs (in theory you can have noncompete clauses that mediate this, but I’m skeptical about how that works out in practice)
AIs interacting with society, or each other, in ways that destabilize humanity.
I’m working on a longer comment (which will maybe turn out to just be a whole post) about this, but wanted to flag it here for now.