Thanks for the in-depth answer. The engineer side of me gets leery whenever ‘straightforward real world scaling following a working theory’ is a premise, the likelihood of there being no significant technical obstacles at all, other than resources and energy, seems vanishingly low. A thousand and one factors could impede in realizing even the most perfect theory, much like other complex engineered systems. Possible surprises such as some dependence on the substrate, on the specific arrangement of hardware, on other emergent factors, on software factors, etc...
If there is a general theory of intelligence and it scales well, there are two possibilities. Either we are already in a hardware overhang, and we get an intelligence explosion even without recursive self improvement. Or the compute required is so great that it takes an expensive supercomputer to run, in which case it’ll be a slow takeoff. The probability that we have exactly human intelligence levels of compute seems low to me. Probably we either have way too much or way too little.
Thanks for the in-depth answer. The engineer side of me gets leery whenever ‘straightforward real world scaling following a working theory’ is a premise, the likelihood of there being no significant technical obstacles at all, other than resources and energy, seems vanishingly low. A thousand and one factors could impede in realizing even the most perfect theory, much like other complex engineered systems. Possible surprises such as some dependence on the substrate, on the specific arrangement of hardware, on other emergent factors, on software factors, etc...
If there is a general theory of intelligence and it scales well, there are two possibilities. Either we are already in a hardware overhang, and we get an intelligence explosion even without recursive self improvement. Or the compute required is so great that it takes an expensive supercomputer to run, in which case it’ll be a slow takeoff. The probability that we have exactly human intelligence levels of compute seems low to me. Probably we either have way too much or way too little.