I share the sense that many “AGI” forecasts are going to be very hard to arbitrate—at best they have a few years of slack one way or the other, and at worst they will be completely reinterpreted (I could easily see someone arguing for AGI today).
I try to give forecasts for “technological singularity” instead, which I think has a variety of mostly-equivalent operationalizations. (When asked to give a timeline to AI I often give a tongue-in-cheek operationalization of “capture >1% of sun’s energy.” This is obviously more appropriate if coupling timelines with the substantive prediction that that it will only make a few months of difference which crazy-ambitious technological milestone you choose—just as I think it only makes a few centuries of difference which milestone you use for forecasting the technological singularity starting from 10,000 BC.)
I’m fond of “x percent of sun’s energy used”-syle stuff because I would expect runaway superintelligence to probably go ahead and use that energy, and it has a decent shot at being resolvable.
But I think we need to be careful about assuming all the crazy-ambitious milestones end up only a few months from each other. You could have a situation where cosmic industrialization is explosively fast heading away from Earth, with every incentive to send out seed ships for a land-grab. But it’s plausible that could be going on despite here on Earth it’s much slower, if there are some big incumbents that maintain control and develop more slowly. I’m not super sure that’ll happen but it’s not obvious that all the big milestones happen within a few months of each other, if we assume local control is maintained and the runaway Foom goes elsewhere.
This is an example of why I think it does matter what milestone people pick, but it will often be for reasons that are very hard to foresee.
Yeah, I somewhat prefer “could capture >1% of the energy within a year if we wanted.” But if being more serious I think it’s better to just directly try to get at the spirit of “basically everything happens all at once,” or else do something more mundane like massive growth in energy capture (like 100%/year rather than “all the sun’s energy”).
I share the sense that many “AGI” forecasts are going to be very hard to arbitrate—at best they have a few years of slack one way or the other, and at worst they will be completely reinterpreted (I could easily see someone arguing for AGI today).
I try to give forecasts for “technological singularity” instead, which I think has a variety of mostly-equivalent operationalizations. (When asked to give a timeline to AI I often give a tongue-in-cheek operationalization of “capture >1% of sun’s energy.” This is obviously more appropriate if coupling timelines with the substantive prediction that that it will only make a few months of difference which crazy-ambitious technological milestone you choose—just as I think it only makes a few centuries of difference which milestone you use for forecasting the technological singularity starting from 10,000 BC.)
I’m fond of “x percent of sun’s energy used”-syle stuff because I would expect runaway superintelligence to probably go ahead and use that energy, and it has a decent shot at being resolvable.
But I think we need to be careful about assuming all the crazy-ambitious milestones end up only a few months from each other. You could have a situation where cosmic industrialization is explosively fast heading away from Earth, with every incentive to send out seed ships for a land-grab. But it’s plausible that could be going on despite here on Earth it’s much slower, if there are some big incumbents that maintain control and develop more slowly. I’m not super sure that’ll happen but it’s not obvious that all the big milestones happen within a few months of each other, if we assume local control is maintained and the runaway Foom goes elsewhere.
This is an example of why I think it does matter what milestone people pick, but it will often be for reasons that are very hard to foresee.
Yeah, I somewhat prefer “could capture >1% of the energy within a year if we wanted.” But if being more serious I think it’s better to just directly try to get at the spirit of “basically everything happens all at once,” or else do something more mundane like massive growth in energy capture (like 100%/year rather than “all the sun’s energy”).