For my Ensemble General Intelligence model, I was mostly imagining #2 instead of #3.
I said of my ensemble general intelligence model:
It could also dynamically generate narrow optimisers on the fly for the problem sets.
General intelligence might be described as an algorithm for picking (a) narrow optimiser(s) to apply to a given problem set (given x examples from said set).
I did not intend to imply that the set of narrow optimisers the general optimiser is selecting from is represented within the agent. I was thinking of a rough mathematical model for how you can describe it.
That there exists a (potentially infinite) set of all possible narrow optimisers a general intelligence might generate/select from, and there exists a function mapping problems sets (given x examples of said set) to narrow optimisers does not imply that any such representation is stored internally in the agent, nor that the agent implements a look up table.
I equivocated between selection and generation. In practice I imagine generation, but the mathematics of selection are easier to reason about.
I imagine that trying to implement ensemble specialised is impractical in the real world because there are too many possible problem sets. I did not at all consider it a potential model of general intelligence.
I might add this clarification when next I’m on my laptop.
It seems to me that the qualm is not about #2 vs #3 as models for humans, but how easily transfer learning happens for the relevant models of general intelligence, and what progress among the class of general intelligence that manifests in our world looks like.
Currently, I think that it’s possible to improve the meta optimisation processes for generating object level optimisation processes, but this doesn’t imply that an improvement to a particular object level optimisation process will transfer across domains.
This is important because improving object level processes and improving meta level processes are different. And improving meta level processes mostly looks like learning a new domain quicker as opposed to improved accuracy in all extant domains. Predictive accuracy still doesn’t transfer across domains the way it would for a simple optimiser.
I can probably make this distinction clearer, elaborate on it more in the OP.
I’ll think on this issue more in the morning.
The section I’m least confident/knowledgeable about is the speculation around applicability of NFL theorems and exploitation of structure/regularity, so I’ll avoid discussing it.
I simply do not think it’s a discussion I can contribute meaningfully to.
Future me with better models of optimisation processes would be able to reason better around it.
For my Ensemble General Intelligence model, I was mostly imagining #2 instead of #3.
I said of my ensemble general intelligence model:
I did not intend to imply that the set of narrow optimisers the general optimiser is selecting from is represented within the agent. I was thinking of a rough mathematical model for how you can describe it.
That there exists a (potentially infinite) set of all possible narrow optimisers a general intelligence might generate/select from, and there exists a function mapping problems sets (given x examples of said set) to narrow optimisers does not imply that any such representation is stored internally in the agent, nor that the agent implements a look up table.
I equivocated between selection and generation. In practice I imagine generation, but the mathematics of selection are easier to reason about.
I imagine that trying to implement ensemble specialised is impractical in the real world because there are too many possible problem sets. I did not at all consider it a potential model of general intelligence.
I might add this clarification when next I’m on my laptop.
It seems to me that the qualm is not about #2 vs #3 as models for humans, but how easily transfer learning happens for the relevant models of general intelligence, and what progress among the class of general intelligence that manifests in our world looks like.
Currently, I think that it’s possible to improve the meta optimisation processes for generating object level optimisation processes, but this doesn’t imply that an improvement to a particular object level optimisation process will transfer across domains.
This is important because improving object level processes and improving meta level processes are different. And improving meta level processes mostly looks like learning a new domain quicker as opposed to improved accuracy in all extant domains. Predictive accuracy still doesn’t transfer across domains the way it would for a simple optimiser.
I can probably make this distinction clearer, elaborate on it more in the OP.
I’ll think on this issue more in the morning.
The section I’m least confident/knowledgeable about is the speculation around applicability of NFL theorems and exploitation of structure/regularity, so I’ll avoid discussing it.
I simply do not think it’s a discussion I can contribute meaningfully to.
Future me with better models of optimisation processes would be able to reason better around it.