I think this is making a five-inch fence half an inch higher. It’s just not relevant on the scale of an agent to which a human is a causal system made of brain areas and a group of humans is just another causal system made of several interacting copies of those brain areas.
I agree that the AI you envision would be dangerously likely to escape a “competent” box too; and in any case, even if you manage to keep the AI in the box, attempts to actually use any advice it gives are extremely dangerous.
That said, I think your “half an inch” is off by multiple orders of magnitude.
I think this is making a five-inch fence half an inch higher. It’s just not relevant on the scale of an agent to which a human is a causal system made of brain areas and a group of humans is just another causal system made of several interacting copies of those brain areas.
I agree that the AI you envision would be dangerously likely to escape a “competent” box too; and in any case, even if you manage to keep the AI in the box, attempts to actually use any advice it gives are extremely dangerous.
That said, I think your “half an inch” is off by multiple orders of magnitude.