I fail to understand this option C is a viable path to superintelligence. In my model if you’re chaining lots of simple or “dumb” pieces together to get complex behavior, you need some “force” or optimization process going on to steer the whole into high-performance.
For example, individual neurons (both natural and artificial) are simple, and can be chained up together in complex behavior, but the complex behavior only arises when you train the system with some sort of reward/optimization signals.
Maybe I’m wrong here and for “slightly smart” components such as existing LLMs you can actually hook them up in large groups in a clever way, with further learning happening only at the prompt-level, etc, and the system scales up to superintelligence somehow.
Because this generates a lot of perplexity in my world-model, I mostly don’t know how to reason about these hypothetical agents. I’m afraid that such agents will be far removed from the “folk psychology” / interpretability of the component LLM (e.g maybe it queries LLMs a million times in a complicated runtime-defined network of information/prompt flows before giving an answer)? Maybe you can understand what each LLM is doing but not what the whole is doing in a meaningful way. Would love to be wrong!
It seems plausible to me that we can achieve improvements in the cognition of such agents the same way we improve human cognition, using various rationality techniques to organise thoughts in a more productive manner.
For example, instead of just asking LLM “Develop me a plan to achieve X” and simply going with it, We then promt the model to find possible failure modes in this plan, and then to find a way around these failure modes, alternative options and so on.
We may not get 10000 IQ intelligence, totally leaving all humans in the dust in ten years. And this is another good thing, a future where we try to make smarter and smarter LLM-based agents with clever chains of promt ingeneiring looks more like a slow take off, than a fast one. But I believe we would be able to achive human and a bit higther than human level AGI this way.
I fail to understand this option C is a viable path to superintelligence. In my model if you’re chaining lots of simple or “dumb” pieces together to get complex behavior, you need some “force” or optimization process going on to steer the whole into high-performance.
For example, individual neurons (both natural and artificial) are simple, and can be chained up together in complex behavior, but the complex behavior only arises when you train the system with some sort of reward/optimization signals.
Maybe I’m wrong here and for “slightly smart” components such as existing LLMs you can actually hook them up in large groups in a clever way, with further learning happening only at the prompt-level, etc, and the system scales up to superintelligence somehow.
Because this generates a lot of perplexity in my world-model, I mostly don’t know how to reason about these hypothetical agents. I’m afraid that such agents will be far removed from the “folk psychology” / interpretability of the component LLM (e.g maybe it queries LLMs a million times in a complicated runtime-defined network of information/prompt flows before giving an answer)? Maybe you can understand what each LLM is doing but not what the whole is doing in a meaningful way. Would love to be wrong!
It seems plausible to me that we can achieve improvements in the cognition of such agents the same way we improve human cognition, using various rationality techniques to organise thoughts in a more productive manner.
For example, instead of just asking LLM “Develop me a plan to achieve X” and simply going with it, We then promt the model to find possible failure modes in this plan, and then to find a way around these failure modes, alternative options and so on.
We may not get 10000 IQ intelligence, totally leaving all humans in the dust in ten years. And this is another good thing, a future where we try to make smarter and smarter LLM-based agents with clever chains of promt ingeneiring looks more like a slow take off, than a fast one. But I believe we would be able to achive human and a bit higther than human level AGI this way.