It seems like eventually people are going to make competent goal-directed agents, and at that point we will indeed have the problems of their exerting more optimisation power than humanity.
In fact it seems like these non-agentic AIs might make things worse, because the goal-maximisation agents will be able to use the non-agentic AIs.
The solution is obviously to prohibit the creation of goal-maximization agents and use scaffolded LLMs, instead.
Unfortunately it seems like people are going to make AI agents anyway, because ML researchers love making things.
I bet geneticists would also love to make some new things with clonning. And yet we have a noticeable lack of clones. Do not underestimate the ability of our civilization to limit its own progress.
So an alternative possible conclusion would be that we should actually try to accelerate agentic AI research as much as possible, because eventually we are going to have influential AI maximisers, and we want them to occur before the forecasting/planning overhang (and the hardware overhang) get too large.
We are currently living in the luckiest possible world where we have powerful AI models which are nevertheless existentially harmless specifically because they lack the agentic part. Moreover, we can use these models to develop agentic-but-not-really systems that can satisfy the demand, without doing the risky research into developping coherent goal maximizators. This is a miracle. We didn’t expect that thing could be this good. Suddenly there is a comprehensive way we may not be doomed. And you are proposing to dismiss our incredible advantage and return back to the course of being doomed, anyway.
The solution is obviously to prohibit the creation of goal-maximization agents and use scaffolded LLMs, instead.
I bet geneticists would also love to make some new things with clonning. And yet we have a noticeable lack of clones. Do not underestimate the ability of our civilization to limit its own progress.
We are currently living in the luckiest possible world where we have powerful AI models which are nevertheless existentially harmless specifically because they lack the agentic part. Moreover, we can use these models to develop agentic-but-not-really systems that can satisfy the demand, without doing the risky research into developping coherent goal maximizators. This is a miracle. We didn’t expect that thing could be this good. Suddenly there is a comprehensive way we may not be doomed. And you are proposing to dismiss our incredible advantage and return back to the course of being doomed, anyway.