There’s a limit on how much siloing and information security a state sponsored AGI research team can afford when it’s competing with teams like DeepMind that don’t necessarily have to operate under the same constraints
Mm, I’m not sure we’re talking about the same problem? I’m saying that a lot of people will have read-access, and each of them would be able to notice that there’s something very wrong with the AI’s target, and then they’ll need not to raise a fuss about it, for the conspiracy to succeed.
there is much less slack available for selecting AI researchers for loyalty
In which case there’ll be more oversight over them, no?
In addition, the leading engineers won’t necessarily need to be genius-level. The people doing foundational alignment research would need to be, but if we’re in a hypothetical where we have alignment tools good enough to avoid omnicide, we’re past the stage where theory was the bottleneck. In that world, verifying the AI’s preferences should be trivial for only normally-competent ML engineers.
Which you’d then select for loyalty and put in the oversight team.
Mm, I’m not sure we’re talking about the same problem? I’m saying that a lot of people will have read-access, and each of them would be able to notice that there’s something very wrong with the AI’s target, and then they’ll need not to raise a fuss about it, for the conspiracy to succeed.
In which case there’ll be more oversight over them, no?
In addition, the leading engineers won’t necessarily need to be genius-level. The people doing foundational alignment research would need to be, but if we’re in a hypothetical where we have alignment tools good enough to avoid omnicide, we’re past the stage where theory was the bottleneck. In that world, verifying the AI’s preferences should be trivial for only normally-competent ML engineers.
Which you’d then select for loyalty and put in the oversight team.