If we just naively combine these two, we get two multiplying exponential growths for the total semiconductor compute over time, which again gives us exponential growth. This can be considered explosive in its own way but it is not a singularity.
Data storage: A quick search for global data storage lead me to this whitepaper where Fig. 1 on page 6 shows the “Annual Size of the Global Datasphere” and looks as if they just directly plotted an exponential function and did not directly use yearly empirical data. I am not sure whether the true data was just this close to an exponential or whether they did just decide to directly plot an exponential. An both cases, this speaks against super-exponential growth.
General remarks: I have the feeling that in technological singularity there are so many moving parts that even if we found a quantity which consistently shows power-law growth, we would still have a large uncertainty about the timing of singularity. For example, I assume that the transition from DNA as basically the only storage medium to human culture with written text was a significant game-changer. But the timing of this transition could have easily been different by a few centuries (certainly) or millennia (probably). Similarly I would expect that the first general artificial intelligence will be really relevant for a singularity, while it could easily be delayed by a few years /decades even if we know an expected date for a phase transition.
Possibly this is similar to supercooling of water, where it is the presence of ample condensation points which allows us to get the usual freezing temperature. If the liquid water is already beyond the standard freezing point, the first nucleus will decide whether the whole thing freezes.
Also, I could imagine “the amount of information that is usefully integrated in a decision-making agent” to be a quantity that is quite substrate-independent (DNA, people, humanity, AI models) so that one could try to find a trend over long time scales. But I expect that it will be super hard to define “amount of information” sufficiently well. At least “parameter counts in machine learning” (if we simply assume that a model will not integrate roughly as much data as it has parameters) does seem compatible with super-exponential growth.
Thank you for your research! First of all, I don’t expect the non-human parameter to give a clear power-law, since we need to add humans as well. Of course, close to singularity the impact of humans will be very small, but maybe we are not that close yet. Now for the details:
Compute: 1. Yes, Moore’s law was a quite steady exponential for quite a while, but we indeed should multiply it. 2. The graph shows just a five years period, and not the number of chips produced, but revenue. The five years period is too small for any conclusions, and I am not sure that fluctuations in revenue are not driven mainly by market price rather than by produced amount.
Data storage: Yes, I saw that one before, seems more like they just draw a nice picture rather than real data.
General remarks:
I agree with the point that AGI appearance can be sufficiently random. I can see two mechanisms that potentially may make it less random. First, we may need a lot of computational resources, data storage etc. to create it, and as a lab or company reaches the threshold, it happens easily with already existing algorithms. Second, we may need a lot of digitalized data to train AGI, so the transition again happens only as we have that much data. Lastly, notice that cthe reation of AGI is not a singularity in a mathematical sense yet. It will certainly accelerate our progress, but not to infinity, so if the data will predict for example singularity in 2030, it will likely mean AGI earlier than that.
How trustworthy would this prediction be? Depends on the amount of data and noise. If we have just 10-20 datapoints scattered all around the graph, so you can connect the dots in any way you like—not really. If, instead, we are lucky and the control parameter happened to be something easily measurable (something such that you can get just-in-time statistics, like the number of papers on arXiv right now, so we can get really a lot of data points) and the parameter continues to change as theory predicts—it would be a quite strong argument for the timeline.
It is not very likely that the control parameter will be that easily measurable and will obey power-law that good. I think it is a very high risk—very high gain project (very high gain, because if the prediction will be very clear it will be possible to persuade more people that the problem is important).
I recently stumbled upon this paper “The World’s Technological Capacity to Store, Communicate, and Compute Information”, which has some neat overviews regarding data storage, broadcasting and compute trends: From a quick look at the Figures my impression is that compute and storage look very much like ‘just’ exponential, while there is a super-exponential figure (Fig. 4) for the total communication bandwidth (1986-2007)[1]
General
I can see two mechanisms that potentially may make it less random.
That makes sense. Now that I think about this, I could well imagine that something like “scale is all you need” is sufficiently true that randomness doesn’t shift the expected date by a large amount.
if the data will predict for example singularity in 2030, it will likely mean AGI earlier than that.
Good point! I think that the time span around and before the first AGI will be most relevant to us as it probably provides the largest possibility to steer the outcome to something good, but this indeed is not the date we found get in a power-law singularity. This feels quite related to the discussion around biological anchors for estimating the necessary compute for transformative AI and the conclusions one can draw from this. I feel that if one thinks that these are informative, even ‘just’ the exponential compute trends provide rather strong bounds (at least compared to having biological time-scales or such as reference).
t is not very likely that the control parameter will be that easily measurable and will obey power-law that good. I think it is a very high risk—very high gain project (very high gain, because if the prediction will be very clear it will be possible to persuade more people that the problem is important).
Regarding persuading people: I am not sure whether such a trend would make such a large psychological difference compared to the things that we already have: All Possible Views About Humanity’s Future Are Wild. But it would still be a noteworthy finding in any case
Quick hand-wavy estimate whether the trend continued in the last 15 years: If I just assume ‘trend continues’ to mean ‘doubling time halves every 7 years with a factor x40 from 2000 to 2007’ (this isn’t power law, but much easier for me to think about and hopefully close enough in this parameter range) we’d have to find an increase in global bandwidth by (a factor of x40 is roughly 1.5 OOM and we get this factor twice in the first 7 years and 4 times in the second step which makes) roughly 9 orders of magnitude from 2007 to 2021. I did not try to find current numbers, but 9 OOM sound unlikely to me. At least my data usage did increase by maybe 3 to 5 OOM, but 9 OOM just seems too high. Thus, I anecdotally conclude that this trend very probably slowed down in the last 15 years compared to ‘doubling time halves every 7 years’. But to be fair, I would not have correctly predicted the 1986-2007 trend either—so this shouldn’t be taken too seriously.
Compute: A very simple attempt at estimating (non-biological) computing power:
A version of Moore’s law applies to the cost of computation power. And Moore’s law held true quite steadily, so we can assume an exponential growth,
The figure in this article on the growth of the semiconductor industry shows significant oscillations in the growth rate in the last 25 years, but seems totally compatible with exponential growth plus a lot of noise.
If we just naively combine these two, we get two multiplying exponential growths for the total semiconductor compute over time, which again gives us exponential growth. This can be considered explosive in its own way but it is not a singularity.
Data storage: A quick search for global data storage lead me to this whitepaper where Fig. 1 on page 6 shows the “Annual Size of the Global Datasphere” and looks as if they just directly plotted an exponential function and did not directly use yearly empirical data. I am not sure whether the true data was just this close to an exponential or whether they did just decide to directly plot an exponential. An both cases, this speaks against super-exponential growth.
General remarks: I have the feeling that in technological singularity there are so many moving parts that even if we found a quantity which consistently shows power-law growth, we would still have a large uncertainty about the timing of singularity. For example, I assume that the transition from DNA as basically the only storage medium to human culture with written text was a significant game-changer. But the timing of this transition could have easily been different by a few centuries (certainly) or millennia (probably). Similarly I would expect that the first general artificial intelligence will be really relevant for a singularity, while it could easily be delayed by a few years /decades even if we know an expected date for a phase transition.
Possibly this is similar to supercooling of water, where it is the presence of ample condensation points which allows us to get the usual freezing temperature. If the liquid water is already beyond the standard freezing point, the first nucleus will decide whether the whole thing freezes.
Also, I could imagine “the amount of information that is usefully integrated in a decision-making agent” to be a quantity that is quite substrate-independent (DNA, people, humanity, AI models) so that one could try to find a trend over long time scales. But I expect that it will be super hard to define “amount of information” sufficiently well. At least “parameter counts in machine learning” (if we simply assume that a model will
notintegrate roughly as much data as it has parameters) does seem compatible with super-exponential growth.Thank you for your research! First of all, I don’t expect the non-human parameter to give a clear power-law, since we need to add humans as well. Of course, close to singularity the impact of humans will be very small, but maybe we are not that close yet. Now for the details:
Compute:
1. Yes, Moore’s law was a quite steady exponential for quite a while, but we indeed should multiply it.
2. The graph shows just a five years period, and not the number of chips produced, but revenue. The five years period is too small for any conclusions, and I am not sure that fluctuations in revenue are not driven mainly by market price rather than by produced amount.
Data storage:
Yes, I saw that one before, seems more like they just draw a nice picture rather than real data.
General remarks:
I agree with the point that AGI appearance can be sufficiently random. I can see two mechanisms that potentially may make it less random. First, we may need a lot of computational resources, data storage etc. to create it, and as a lab or company reaches the threshold, it happens easily with already existing algorithms. Second, we may need a lot of digitalized data to train AGI, so the transition again happens only as we have that much data.
Lastly, notice that cthe reation of AGI is not a singularity in a mathematical sense yet. It will certainly accelerate our progress, but not to infinity, so if the data will predict for example singularity in 2030, it will likely mean AGI earlier than that.
How trustworthy would this prediction be? Depends on the amount of data and noise. If we have just 10-20 datapoints scattered all around the graph, so you can connect the dots in any way you like—not really. If, instead, we are lucky and the control parameter happened to be something easily measurable (something such that you can get just-in-time statistics, like the number of papers on arXiv right now, so we can get really a lot of data points) and the parameter continues to change as theory predicts—it would be a quite strong argument for the timeline.
It is not very likely that the control parameter will be that easily measurable and will obey power-law that good. I think it is a very high risk—very high gain project (very high gain, because if the prediction will be very clear it will be possible to persuade more people that the problem is important).
Trends of different quantities:
Generally, I agree with your points :)
I recently stumbled upon this paper “The World’s Technological Capacity to Store, Communicate, and Compute Information”, which has some neat overviews regarding data storage, broadcasting and compute trends:
From a quick look at the Figures my impression is that compute and storage look very much like ‘just’ exponential, while there is a super-exponential figure (Fig. 4) for the total communication bandwidth (1986-2007)[1]
General
That makes sense. Now that I think about this, I could well imagine that something like “scale is all you need” is sufficiently true that randomness doesn’t shift the expected date by a large amount.
Good point! I think that the time span around and before the first AGI will be most relevant to us as it probably provides the largest possibility to steer the outcome to something good, but this indeed is not the date we found get in a power-law singularity.
This feels quite related to the discussion around biological anchors for estimating the necessary compute for transformative AI and the conclusions one can draw from this. I feel that if one thinks that these are informative, even ‘just’ the exponential compute trends provide rather strong bounds (at least compared to having biological time-scales or such as reference).
Regarding persuading people: I am not sure whether such a trend would make such a large psychological difference compared to the things that we already have: All Possible Views About Humanity’s Future Are Wild. But it would still be a noteworthy finding in any case
Quick hand-wavy estimate whether the trend continued in the last 15 years: If I just assume ‘trend continues’ to mean ‘doubling time halves every 7 years with a factor x40 from 2000 to 2007’ (this isn’t power law, but much easier for me to think about and hopefully close enough in this parameter range) we’d have to find an increase in global bandwidth by (a factor of x40 is roughly 1.5 OOM and we get this factor twice in the first 7 years and 4 times in the second step which makes) roughly 9 orders of magnitude from 2007 to 2021. I did not try to find current numbers, but 9 OOM sound unlikely to me. At least my data usage did increase by maybe 3 to 5 OOM, but 9 OOM just seems too high. Thus, I anecdotally conclude that this trend very probably slowed down in the last 15 years compared to ‘doubling time halves every 7 years’. But to be fair, I would not have correctly predicted the 1986-2007 trend either—so this shouldn’t be taken too seriously.