I was a big fan and ready to show this to the curious layperson before it turned from informal but compelling arguments to demanding formal proof.
I’m puzzled by your statement that “there is so far neither a strong theoretical nor empirical basis for power-seeking”. It conflicts with your initial arguments. I’d say there’s a very strong theoretical basis for expecting that behavior. More resources enable agents to achieve goals better and with more certainty. We’re not certain that agents will do this in all cases, but it seems like the burden of proof is strongly on those who think this isn’t a risk.
This seems analogous to saying that we had no strong theoretical basis for gravity before we had a clear mathematical description that was confirmed with formal experiments. Everyone still expected things to fall when unsupported, since informal observation showed that they usually did.
People seek money and collaborators when they have goals. Power helps accomplish goals on average. It’s how the world clearly and obviously works. I’d need a proof against it to stop believing it; a mathematical formalism would do little to boost my confidence that this is a simple fact about a complex world.
I was a big fan and ready to show this to the curious layperson before it turned from informal but compelling arguments to demanding formal proof.
I’m puzzled by your statement that “there is so far neither a strong theoretical nor empirical basis for power-seeking”. It conflicts with your initial arguments. I’d say there’s a very strong theoretical basis for expecting that behavior. More resources enable agents to achieve goals better and with more certainty. We’re not certain that agents will do this in all cases, but it seems like the burden of proof is strongly on those who think this isn’t a risk.
This seems analogous to saying that we had no strong theoretical basis for gravity before we had a clear mathematical description that was confirmed with formal experiments. Everyone still expected things to fall when unsupported, since informal observation showed that they usually did.
People seek money and collaborators when they have goals. Power helps accomplish goals on average. It’s how the world clearly and obviously works. I’d need a proof against it to stop believing it; a mathematical formalism would do little to boost my confidence that this is a simple fact about a complex world.