I do think that if you get an AGI significantly past human intelligence in all respects, it would obviously tend to FOOM. I mean, I suspect that Eliezer fooms if you give an Eliezer the ability to backup, branch, and edit himself.
What improvements would you make to your brain that you would anticipate yielding greater intelligence? I can think of a few possible strategies:
Just adding a bunch of neurons everywhere. Make my brain bigger.
Study how very smart brains look, and try to make my brain look more like theirs.
For an AI, the first strategy is equivalent to adding more hardware and scaling. In that respect, the “recursive” part of recursive self-improvement doesn’t seem to add anything. We already know that we can get improvements by scaling hardware. There aren’t major avenues for this strategy to FOOM via a feedback loop.
The second strategy is interesting, though it might not work well for anyone “at the frontier” of intelligence.
My guess is that to do meaningful recursive self-improvement, your code must be fairly interpretable.
EY knows more neuroscience than me (I know very little) but here’s a 5-min brainstorm of ideas:
--For a fixed compute budget, spend more of it on neurons associated with higher-level thought (the neocortex?) and less of it on neurons associated with e.g. motor control or vision.
--Assuming we are an upload of some sort rather than a physical brain, tinker with the rules a bit so that e.g. neuron waste products get magically deleted instead of having to be pumped out, neurons never run out of energy/oxygen and need to rest, etc. Study situations where you are in “peak performance” or “flow” and then explore ways to make your brain enter those states at will.
--Use ML pruning techniques to cut away neurons that aren’t being useful, to get slightly crappier mini-Eliezers that cost 10% the compute. These can then automate away 90% of your cognition, saving you enough compute that you can either think a few times faster or have a few copies running in parallel.
--Build automated tools that search through your brain for circuits that are doing something pretty simple, like a giant OR gate or an oscillator, and then replace those circuits with small bits of code, thereby saving significant compute. If anything goes wrong, no worries, just revert to backup.
What improvements would you make to your brain that you would anticipate yielding greater intelligence? I can think of a few possible strategies:
Just adding a bunch of neurons everywhere. Make my brain bigger.
Study how very smart brains look, and try to make my brain look more like theirs.
For an AI, the first strategy is equivalent to adding more hardware and scaling. In that respect, the “recursive” part of recursive self-improvement doesn’t seem to add anything. We already know that we can get improvements by scaling hardware. There aren’t major avenues for this strategy to FOOM via a feedback loop.
The second strategy is interesting, though it might not work well for anyone “at the frontier” of intelligence.
My guess is that to do meaningful recursive self-improvement, your code must be fairly interpretable.
EY knows more neuroscience than me (I know very little) but here’s a 5-min brainstorm of ideas:
--For a fixed compute budget, spend more of it on neurons associated with higher-level thought (the neocortex?) and less of it on neurons associated with e.g. motor control or vision.
--Assuming we are an upload of some sort rather than a physical brain, tinker with the rules a bit so that e.g. neuron waste products get magically deleted instead of having to be pumped out, neurons never run out of energy/oxygen and need to rest, etc. Study situations where you are in “peak performance” or “flow” and then explore ways to make your brain enter those states at will.
--Use ML pruning techniques to cut away neurons that aren’t being useful, to get slightly crappier mini-Eliezers that cost 10% the compute. These can then automate away 90% of your cognition, saving you enough compute that you can either think a few times faster or have a few copies running in parallel.
--Build automated tools that search through your brain for circuits that are doing something pretty simple, like a giant OR gate or an oscillator, and then replace those circuits with small bits of code, thereby saving significant compute. If anything goes wrong, no worries, just revert to backup.
This was a fun exercise!