Optimization power is the source of the danger, not agency. Agents merely wield optimality to achieve their goals.
Agency is orthogonal to optimization power.
Where “agency” is defined as the ability to optimize for an objective, given some internal or external optimization power, and “optimality” (of a system) is defined as having an immense amount of optimization power, either during its creation (the nuclear bomb) or its runtime (Solomonoff induction).
This hints at the notion that there’s a minimum Kolmogorov complexity (aka algorithmic description length) that needs to be met by an objective of an AI to be considered safe, assuming that we want the AI to be safe in the worst case scenario when it has access to extreme optimization power.
I would not generally put the Kolmogorov section the way you did, but I suspect that’s more a disagreement on what Kolmogorov complexity is like than what agents are like. (I think the statement is still literally true.)
Elegant. Here’s my summary:
Optimization power is the source of the danger, not agency. Agents merely wield optimality to achieve their goals.
Agency is orthogonal to optimization power.
Where “agency” is defined as the ability to optimize for an objective, given some internal or external optimization power, and “optimality” (of a system) is defined as having an immense amount of optimization power, either during its creation (the nuclear bomb) or its runtime (Solomonoff induction).
This hints at the notion that there’s a minimum Kolmogorov complexity (aka algorithmic description length) that needs to be met by an objective of an AI to be considered safe, assuming that we want the AI to be safe in the worst case scenario when it has access to extreme optimization power.
I’d love to know if I’m missing something.
That seems a reasonable takeaway to me.
I would not generally put the Kolmogorov section the way you did, but I suspect that’s more a disagreement on what Kolmogorov complexity is like than what agents are like. (I think the statement is still literally true.)