What do you think about energy costs? Last I thought about this it seemed plausible to me that in ten years or so the atoms making up supercomputers will be <10% of the cost of training giant models, most of the cost being paying for the electricity and upkeep and rent.
Lifetime energy costs are already significant, but I don’t think the problem will get that skew this decade. IRDS’ predicted transistor scaling until ~2028 should prevent power density increasing by too much.
Longer-term this does become a greater concern. I can’t say I have particularly wise predictions here. There are ways to get more energy efficiency by spending more on lower-clocked hardware, or by using a larger memory:compute ratio, and there are also hardware architectures with plausible significant power advantages. There are even potential ways for energy to fall in price, like with solar PV or fusion, though I haven’t a good idea how far PV prices could fall, and for fusion it seems like a roll of the dice what the price will be.
It’s entirely possible energy does just become the dominant cost and none of those previous points matter, but it’s also an input we know we can scale up pretty much arbitrarily if we’re willing to spend the money. It’s also something that only starts to become a fundamental economic roadblock after a lot more scaling. For instance, the 100,000 wafer scale processor example requires a lot of power, but only about as much as largest PV installations that currently exist. You could then upgrade it to 2028 technology and stack memory on top of the wafers without changing power density by all that much.
This is likely a topic worth periodically revisiting as the issue gets closer.
Ok, thanks! I defer to your judgment on this, you clearly know way more than me. Oh well, there goes one of my hopes for the price of compute reaching a floor.
There are ways to get more energy efficiency by spending more on lower-clocked hardware, or by using a larger memory:compute ratio, and there are also hardware architectures with plausible significant power advantages.
As far as I understand, we’re only 3 orders of magnitude away from Landauer limit, which doesn’t leave a lot of room to squeeze efficiency out of. On the supply side, fusion doesn’t seem like a relevant factor before 2050 unless an alternative approach takes us by surprise. Solar PV efficiency is already on the OOM of 1, so any advances have to come from reduction in production and maintenance costs (which is plausible for all I know).
On the technology readiness level, I put reversible computing somewhere between von Neumann probes and warp drive. Definitely post-Singularity, likely impossible.
Irreversible is normal computing, the operation makes a state change which does not allow you to go backwards. Reversible computing is a lab curiosity at very small scale, using circuits which slide between states without dissipating energy and can slide the other way too. As Maxim says, it is far-out speculation whether we can really build computers that way.
Warp drive is more likely than not physically impossible, and even if possible would require insane energies, manipulating spacetime using exotic matter (which has never been produced) etc.
It is a true magitech.
Von Neumann Probes seem easier; they re probably physically possible but the sheer engineering for it to work seems very very difficult. In fact there are no credible plans or ideas to even build one.
Just having interstellar space travel is an immense task.
Doing thing with circuits seems comparatively more feasible.
I don’t expect a sustained Moore’s Law type improvement to efficiency here, just the possibility of a few technology jumps with modest but meaningful gains. A factor of 10 beyond CMOS would amount to an extension of a decade.
I probably have much shorter average fusion timelines than you, albeit also with high variance, and wouldn’t be hugely surprised if fusion ramped up commercial operations through the 2030s, nor would I be shocked if it didn’t. The new wave of fusion startups seem to have coherent justifications to me, as a layman.
I would be shocked if fusion provides >10% of electricity to any major economy in the 2030s, like cold-fusion-is-possible-level shocked. On the one hand, the technologies new fusion start-ups are working with are obviously much more plausible than cold fusion, on the other hand there are a LOT of likely ways for fusion to fail besides just technical problems, so my intuition tells me it’s a toss-up.
I don’t know nearly as much about solar PV so my confidence intervals there are much wider. I agree that if there was sufficient economic incentive, we could scale to incredible amounts of compute right now, crypto mining shows an empirical lower bound to that ability.
I agree with all of this. I wasn’t intending to imply fusion would lower global average prices in that timeframe. A massive supercomputer effort like I was describing could build its own plant locally if necessary.
Well done!
What do you think about energy costs? Last I thought about this it seemed plausible to me that in ten years or so the atoms making up supercomputers will be <10% of the cost of training giant models, most of the cost being paying for the electricity and upkeep and rent.
Lifetime energy costs are already significant, but I don’t think the problem will get that skew this decade. IRDS’ predicted transistor scaling until ~2028 should prevent power density increasing by too much.
Longer-term this does become a greater concern. I can’t say I have particularly wise predictions here. There are ways to get more energy efficiency by spending more on lower-clocked hardware, or by using a larger memory:compute ratio, and there are also hardware architectures with plausible significant power advantages. There are even potential ways for energy to fall in price, like with solar PV or fusion, though I haven’t a good idea how far PV prices could fall, and for fusion it seems like a roll of the dice what the price will be.
It’s entirely possible energy does just become the dominant cost and none of those previous points matter, but it’s also an input we know we can scale up pretty much arbitrarily if we’re willing to spend the money. It’s also something that only starts to become a fundamental economic roadblock after a lot more scaling. For instance, the 100,000 wafer scale processor example requires a lot of power, but only about as much as largest PV installations that currently exist. You could then upgrade it to 2028 technology and stack memory on top of the wafers without changing power density by all that much.
This is likely a topic worth periodically revisiting as the issue gets closer.
Ok, thanks! I defer to your judgment on this, you clearly know way more than me. Oh well, there goes one of my hopes for the price of compute reaching a floor.
As far as I understand, we’re only 3 orders of magnitude away from Landauer limit, which doesn’t leave a lot of room to squeeze efficiency out of. On the supply side, fusion doesn’t seem like a relevant factor before 2050 unless an alternative approach takes us by surprise. Solar PV efficiency is already on the OOM of 1, so any advances have to come from reduction in production and maintenance costs (which is plausible for all I know).
The Landauer limit constrains irreversible computing, not computing in general.
On the technology readiness level, I put reversible computing somewhere between von Neumann probes and warp drive. Definitely post-Singularity, likely impossible.
Knowing little about irreversible computing this nevertheless sound surprising to me. Why exactly is irreversible computing so hard?
EDIT ofc I meant reversible not irreversible computing here!
Irreversible is normal computing, the operation makes a state change which does not allow you to go backwards. Reversible computing is a lab curiosity at very small scale, using circuits which slide between states without dissipating energy and can slide the other way too. As Maxim says, it is far-out speculation whether we can really build computers that way.
Warp drive is more likely than not physically impossible, and even if possible would require insane energies, manipulating spacetime using exotic matter (which has never been produced) etc. It is a true magitech.
Von Neumann Probes seem easier; they re probably physically possible but the sheer engineering for it to work seems very very difficult. In fact there are no credible plans or ideas to even build one. Just having interstellar space travel is an immense task.
Doing thing with circuits seems comparatively more feasible.
Agreed. I had [this recent paper](https://ieeexplore.ieee.org/abstract/document/9325353) in mind when I raised the question.
I don’t expect a sustained Moore’s Law type improvement to efficiency here, just the possibility of a few technology jumps with modest but meaningful gains. A factor of 10 beyond CMOS would amount to an extension of a decade.
I probably have much shorter average fusion timelines than you, albeit also with high variance, and wouldn’t be hugely surprised if fusion ramped up commercial operations through the 2030s, nor would I be shocked if it didn’t. The new wave of fusion startups seem to have coherent justifications to me, as a layman.
I would be shocked if fusion provides >10% of electricity to any major economy in the 2030s, like cold-fusion-is-possible-level shocked. On the one hand, the technologies new fusion start-ups are working with are obviously much more plausible than cold fusion, on the other hand there are a LOT of likely ways for fusion to fail besides just technical problems, so my intuition tells me it’s a toss-up.
I don’t know nearly as much about solar PV so my confidence intervals there are much wider. I agree that if there was sufficient economic incentive, we could scale to incredible amounts of compute right now, crypto mining shows an empirical lower bound to that ability.
I agree with all of this. I wasn’t intending to imply fusion would lower global average prices in that timeframe. A massive supercomputer effort like I was describing could build its own plant locally if necessary.