This is a search for the most powerful and general algorithm (what I call a cognitive architecture), where you do not put any optimization pressure on the algorithm having “world optimizing” ability.
When you use the agent, you use it in “sessions”, like LLMs, and those sessions are finite time and you clear state variables afterwards. No online training, training must be all offline.
For ongoing tasks, the agent must fill out a data structure that is human readable such that another agent can seamlessly “pick up” where the last ended session at.
The reason why this form of AGI is likely to beat the 2 forms you mention is :
(1) it’s much cheaper and faster to reach AGI and then ASI.
(2) it will outperform in ability the “human emulator”, and will be measurably safer on tasks due to no state buildup than the “world optimizer”. The world optimizer cannot be tested for safety because the machine is constantly accumulating state. You need to be able to test models where they are always in a known state, and you benchmark their reliability.
Here’s a third form of AGI:
https://www.lesswrong.com/posts/Aq82XqYhgqdPdPrBA/?commentId=Mvyq996KxiE4LR6ii
This is a search for the most powerful and general algorithm (what I call a cognitive architecture), where you do not put any optimization pressure on the algorithm having “world optimizing” ability.
When you use the agent, you use it in “sessions”, like LLMs, and those sessions are finite time and you clear state variables afterwards. No online training, training must be all offline.
For ongoing tasks, the agent must fill out a data structure that is human readable such that another agent can seamlessly “pick up” where the last ended session at.
The reason why this form of AGI is likely to beat the 2 forms you mention is :
(1) it’s much cheaper and faster to reach AGI and then ASI.
(2) it will outperform in ability the “human emulator”, and will be measurably safer on tasks due to no state buildup than the “world optimizer”. The world optimizer cannot be tested for safety because the machine is constantly accumulating state. You need to be able to test models where they are always in a known state, and you benchmark their reliability.
Many of the ideas were drawn from :
https://www.lesswrong.com/posts/HByDKLLdaWEcA2QQD/applying-superintelligence-without-collusion
https://www.lesswrong.com/posts/5hApNw5f7uG8RXxGS/the-open-agency-model