A lot of this post sounds like fake ignorance. If you just read over it, you might think the questions asked are genuinely unknown, but if you think for a bit, you can see we have quite a lot of evidence and can give a rough answer.
We simply don’t know if intelligence is instrumental or quickly hits diminishing returns.
Well humans are doing ok for themselves, it seems to have accelerating returns up to the level of a smart human. Whats more, intelligence gets more valuable with increasing scale, and with cheaper compute. When controlling a roomba, you are controling a few watts. An algorithm that took a 1kw computer cluster to run, and improved efficiency by 5% wouldn’t be worth it. But it would be worth it to control a power station. Whatsmore, the human brain seems a long way from the theoretical limits of compute. So as a lower bound, imagine what a team of smart humans could do running at 1000 times speed, and then imagine that cost you < 1 watt in energy.
Can intelligence be effectively applied to itself at all?
Yes. Hence the fields of psycology and AI research.
It seems that you need to apply a lot more energy to get a bit more complexity.
Doesn’t seem to match evolutionary record.
What evidence do we have that the payoff of intelligent, goal-oriented experimentation yields enormous advantages (enough to enable an intelligence explosion) over evolutionary discovery relative to its cost?
The way humans can easily do things that evolution never could. The fact that evolution is a really stupid algorithm, its generally much faster to make the problem space differentiable, then use gradient descent.
A lot of this post sounds like fake ignorance. If you just read over it, you might think the questions asked are genuinely unknown, but if you think for a bit, you can see we have quite a lot of evidence and can give a rough answer.
Well humans are doing ok for themselves, it seems to have accelerating returns up to the level of a smart human. Whats more, intelligence gets more valuable with increasing scale, and with cheaper compute. When controlling a roomba, you are controling a few watts. An algorithm that took a 1kw computer cluster to run, and improved efficiency by 5% wouldn’t be worth it. But it would be worth it to control a power station. Whatsmore, the human brain seems a long way from the theoretical limits of compute. So as a lower bound, imagine what a team of smart humans could do running at 1000 times speed, and then imagine that cost you < 1 watt in energy.
Yes. Hence the fields of psycology and AI research.
Doesn’t seem to match evolutionary record.
The way humans can easily do things that evolution never could. The fact that evolution is a really stupid algorithm, its generally much faster to make the problem space differentiable, then use gradient descent.