The first two points in (1) are plausible from what we know so far. I’d hardly put it at more than 90% that there’s no way around them, but let’s go with that for now. How do you get 1 EUR per 10^22 FLOPs as a fundamental physical limit? The link has conservative assumptions based on current human technology. My prior on those holding is negligible, somewhere below 1%.
But that aside, even the most pessimistic assumption in this post don’t imply that a singularity won’t happen.
We know that it is possible to train a 200 IQ equivalent intelligence for at most 3 MW-hr, energy that costs at most $30. We also know that once trained, it is possible for it to do at least the equivalent of a decade of thought by the most expert human that has ever lived for a similar cost. We have very good reason to expect that it could be capable of thinking very much faster than the equivalent human.
Those alone are sufficient for a superintelligence take-off that renders human economic activity (and possibly the human species) irrelevant.
Because 20-year-old people with 200 IQ exist, and their brains consume approximately 3 MW-hr by age 20. Therefore there are no fundamental physical limitations preventing this.
I think this calculation is invalid. A human is created from a seed worth 700 MB of information, encoded in the form of DNA. This was created in millions of years of evolution, compressing/worth a large (but finite) amount of information (energy). A relevant fraction of hardware and software is encoded in this information. Additional learning is done during 20 years worth 3 MWh. The fractional value of this learning part is unknown.
Initialising a starting state of 700 MB at 10^-21 J per bit operation costs about 6 picojoules.
Obtaining that starting state through evolution probably cost many exajoules, but that’s irrelevant to the central thesis of the post: fundamental physical limits based on the cost of energy required for the existence of various levels of intelligence.
If you really intended this post to hypothesize that the only way for AI to achieve high intelligence would be to emulate all of evolution in Earth’s history, then maybe you can put that in another post and invite discussion on it. My comment was in relation to what you actually wrote.
The first two points in (1) are plausible from what we know so far. I’d hardly put it at more than 90% that there’s no way around them, but let’s go with that for now. How do you get 1 EUR per 10^22 FLOPs as a fundamental physical limit? The link has conservative assumptions based on current human technology. My prior on those holding is negligible, somewhere below 1%.
But that aside, even the most pessimistic assumption in this post don’t imply that a singularity won’t happen.
We know that it is possible to train a 200 IQ equivalent intelligence for at most 3 MW-hr, energy that costs at most $30. We also know that once trained, it is possible for it to do at least the equivalent of a decade of thought by the most expert human that has ever lived for a similar cost. We have very good reason to expect that it could be capable of thinking very much faster than the equivalent human.
Those alone are sufficient for a superintelligence take-off that renders human economic activity (and possibly the human species) irrelevant.
How can we know that “it is possible to train a 200 IQ equivalent intelligence for at most 3 MW-hr”?
Because 20-year-old people with 200 IQ exist, and their brains consume approximately 3 MW-hr by age 20. Therefore there are no fundamental physical limitations preventing this.
I think this calculation is invalid. A human is created from a seed worth 700 MB of information, encoded in the form of DNA. This was created in millions of years of evolution, compressing/worth a large (but finite) amount of information (energy). A relevant fraction of hardware and software is encoded in this information. Additional learning is done during 20 years worth 3 MWh. The fractional value of this learning part is unknown.
Initialising a starting state of 700 MB at 10^-21 J per bit operation costs about 6 picojoules.
Obtaining that starting state through evolution probably cost many exajoules, but that’s irrelevant to the central thesis of the post: fundamental physical limits based on the cost of energy required for the existence of various levels of intelligence.
If you really intended this post to hypothesize that the only way for AI to achieve high intelligence would be to emulate all of evolution in Earth’s history, then maybe you can put that in another post and invite discussion on it. My comment was in relation to what you actually wrote.