I’m not sure I follow. Whether it’s the evolving configuration of atoms or bits, both can lead to new applications. The main difference to me seems that today it is typically harder to configure atoms than bits, but perhaps that’s just by our own design of the atoms underlying the bits? If some desired information system would require a specific atomic configuration, then you’d be hardware constrained again.
Let’s say that in order to build AGI we find out you actually need super power efficient computronium, and silicon can’t do that, you need carbon. Now it’s no longer a solved hardware problem, you are going to have to invest massively in carbon based computing. Paul and the rationalists are stuck waiting for the hardware engineers.
This feels related to Hanson’s recent article: https://www.overcomingbias.com/2021/04/to-beat-aliens-we-must-become-aliens.html
Which mentions that the greatest threat to successor ages are preceding ages that don’t like the prospect of ‘alienation’.
Some of the arguments mentioned here against technology point in that same direction.