I recommend Two Sense of “Optimizer”. I agree with you, roughly. I think that it will be relatively easy to build a tool AI that has very strong capabilities, and much harder to build something with world optimization capabilities. This implies that powerful tool-AI (or AI services) will come first, and for the most part they will be safe in the way that agentic AI isn’t safe.
However, two potential things which may trouble this analysis:
1. Tool AI’s might become agentic AIs because of some emergent mesa-optimization program which encourages world optimization. It’s hard to see how likely this would be.
2. Gwern wrote about how Tool AI’s have an instrumental reason to become agentic. At one point I believed this argument, but I no longer think that it is predictive of real AI systems. Practically speaking, even if there was an instrumental advantage to becoming agentic, AGZ just isn’t optimizing a utility function (or at least, it’s not useful to describe it as such) and therefore arguments about instrumental convergence aren’t predictive of AGZ scaled up.
I recommend Two Sense of “Optimizer”. I agree with you, roughly. I think that it will be relatively easy to build a tool AI that has very strong capabilities, and much harder to build something with world optimization capabilities. This implies that powerful tool-AI (or AI services) will come first, and for the most part they will be safe in the way that agentic AI isn’t safe.
However, two potential things which may trouble this analysis:
1. Tool AI’s might become agentic AIs because of some emergent mesa-optimization program which encourages world optimization. It’s hard to see how likely this would be.
2. Gwern wrote about how Tool AI’s have an instrumental reason to become agentic. At one point I believed this argument, but I no longer think that it is predictive of real AI systems. Practically speaking, even if there was an instrumental advantage to becoming agentic, AGZ just isn’t optimizing a utility function (or at least, it’s not useful to describe it as such) and therefore arguments about instrumental convergence aren’t predictive of AGZ scaled up.