If you believe modern economies are incredibly inefficient coordination mechanisms thats a deeper disagreement beyond this post.
But in general my estimate for the intellectual work required to create an entirely new path to a much better compute substrate is something at least vaguely on the order of the amount of intellectual work accumulated into our current foundry tech.
That is not my estimate for the minimal amount of intelligence required to takeover the world in some sense—that would probably require less. But again this is focused on critiquing scenarios where a superintelligence (something greater than humanity in net intelligence) bootstraps from AGI rapidly.
If you believe modern economies are incredibly inefficient coordination mechanisms thats a deeper disagreement beyond this post.
Yep, seems plausibly like a relevant crux. Modern economies sure seem incredibly inefficient, especially when viewed through the lens of “how much is this system doing long-term planning and trying to improve its own intelligence”.
In many important tasks in the modern economy, it isn’t possible to replace on expert with any number of average humans. A large fraction of average humans aren’t experts.
A large fraction of human brains are stacking shelves or driving cars or playing computer games or relaxing etc. Given a list of important tasks in the computer supply chain, most humans, most of the time, are simply not making any attempt at all to solve them.
And of course a few percent of the modern economy is actively trying to blow each other up.
If you believe modern economies are incredibly inefficient coordination mechanisms thats a deeper disagreement beyond this post.
But in general my estimate for the intellectual work required to create an entirely new path to a much better compute substrate is something at least vaguely on the order of the amount of intellectual work accumulated into our current foundry tech.
That is not my estimate for the minimal amount of intelligence required to takeover the world in some sense—that would probably require less. But again this is focused on critiquing scenarios where a superintelligence (something greater than humanity in net intelligence) bootstraps from AGI rapidly.
Yep, seems plausibly like a relevant crux. Modern economies sure seem incredibly inefficient, especially when viewed through the lens of “how much is this system doing long-term planning and trying to improve its own intelligence”.
In many important tasks in the modern economy, it isn’t possible to replace on expert with any number of average humans. A large fraction of average humans aren’t experts.
A large fraction of human brains are stacking shelves or driving cars or playing computer games or relaxing etc. Given a list of important tasks in the computer supply chain, most humans, most of the time, are simply not making any attempt at all to solve them.
And of course a few percent of the modern economy is actively trying to blow each other up.