Here I think I share your interpretation of Yudkowsky; I just disagree with Yudkowsky. I agree on the second part; the model overestimates median TAI arrival time. But I disagree on the first part—I think that having a probability distribution over when to expect TAI / AGI / AI-PONR etc. is pretty important/decision-relevant, e.g. for advising people on whether to go to grad school, or for deciding what sort of research project to undertake. (Perhaps Yudkowsky agrees with this much.)
Hum, I would say Yudkowsky seems to agree with the value of a probability distribution for timelines.
So to me it seems “obvious” that my view of optimization is only strong enough to produce loose, qualitative conclusions, and that it can only be matched to its retrodiction of history, or wielded to pro- duce future predictions, on the level of qualitative physics.
“Things should speed up here,” I could maybe say. But not “The doubling time of this exponential should be cut in half.”
I aspire to a deeper understanding of intelligence than this, mind you. But I’m not sure that even perfect Bayesian enlightenment would let me predict quantitatively how long it will take an AI to solve various problems in advance of it solving them. That might just rest on features of an unexplored solution space which I can’t guess in advance, even though I understand the process that searches.
On the other hand, my interpretation of Yudkowsky strongly disagree with the second part of your paragraph:
And I think that Ajeya’s framework is the best framework I know of for getting that distribution. I think any reasonable distribution should be formed by Ajeya’s framework, or some more complicated model that builds off of it (adding more bells and whistles such as e.g. a data-availability constraint or a probability-of-paradigm-shift mechanic.). Insofar as Yudkowsky was arguing against this, and saying that we need to throw out the whole model and start from scratch with a different model, I was not convinced. (Though maybe I need to reread the post and/or your steelman summary)
So my interpretation of the text is that Yudkowsky says that you need to know how compute will be transformed into AGI to estimate the timelines (then you can plug your estimates for the compute), and that the default of any approach which relies on biological analogies for that part will be sprouting nonsense, because evolution and biology optimize in fundamentally different ways than human researchers do.
For each of the three examples, he goes into more detail about the way this is instantiated. My understanding of his criticism of Ajeya’s model is that he disagrees that just current deep learning algorithms are actually a recipe for turning compute into AGI, and so saying “we keep to current deep learning and estimated the required compute” doesn’t make sense and doesn’t solve the question of how to turn compute into AGI. (Note that his might be the place where you or someone defending Ajeya’s model want to disagree with Yudkowsky. I’m just pointing that this is a more productive place to debate him because that might actually make him change his mind — or change your mind if he convinces you)
The more general argument (the reason why “the trick” doesn’t work) is that if you actually have a way of transforming compute into AGI, that means you know how to build AGI. And if you do, you’re very, very close to the end of the timeline.
I guess I would say: Ajeya’s framework/model can incorporate this objection; this isn’t a “get rid of the whole framework” objection but rather a “tweak the model in the following way” objection.
Like, I agree that it would be bad if everyone who used Ajeya’s model had to put 100% of their probability mass into the six bio anchors she chose. That’s super misleading/biasing/ignores loads of other possible ways AGI might happen. But I don’t think of this as a necessary part of Ajeya’s model; when I use it, I throw out the six bio anchors and just directly input my probability distribution over OOMs of compute. My distribution is informed by the bio anchors, of course, but that’s not the only thing that informs it.
First, I want to clarify that I feel we’re going into a more interesting place, where there’s a better chance that you might find a point that invalidates Yudkowsky’s argument, and can thus convince him of the value of the model.
But it’s also important to realize that IMO, Yudkowsky is not just saying that biological anchors are bad. The more general problem (which is also developed in this post) is that predicting the Future is really hard. In his own model of AGI timelines, the factor that is basically impossible to predict until you can make AGI is the “how much resources are needed to build AGI”.
So saying “let’s just throw away the biological anchors” doesn’t evade the general counterargument that to predict timelines at all, you need to find information on “how much resources are needed to build AGI”, and that is incredibly hard. If you or Ajeya can argue for actual evidence in that last question, then yeah, I expect Yudkowsky would possibly update on the validity of the timeline estimates.
But at the moment, in this thread, I see no argument like that.
Hum, I would say Yudkowsky seems to agree with the value of a probability distribution for timelines.
(Quoting The Weak Inside View (2008) from the AI FOOM Debate)
On the other hand, my interpretation of Yudkowsky strongly disagree with the second part of your paragraph:
So my interpretation of the text is that Yudkowsky says that you need to know how compute will be transformed into AGI to estimate the timelines (then you can plug your estimates for the compute), and that the default of any approach which relies on biological analogies for that part will be sprouting nonsense, because evolution and biology optimize in fundamentally different ways than human researchers do.
For each of the three examples, he goes into more detail about the way this is instantiated. My understanding of his criticism of Ajeya’s model is that he disagrees that just current deep learning algorithms are actually a recipe for turning compute into AGI, and so saying “we keep to current deep learning and estimated the required compute” doesn’t make sense and doesn’t solve the question of how to turn compute into AGI. (Note that his might be the place where you or someone defending Ajeya’s model want to disagree with Yudkowsky. I’m just pointing that this is a more productive place to debate him because that might actually make him change his mind — or change your mind if he convinces you)
The more general argument (the reason why “the trick” doesn’t work) is that if you actually have a way of transforming compute into AGI, that means you know how to build AGI. And if you do, you’re very, very close to the end of the timeline.
I guess I would say: Ajeya’s framework/model can incorporate this objection; this isn’t a “get rid of the whole framework” objection but rather a “tweak the model in the following way” objection.
Like, I agree that it would be bad if everyone who used Ajeya’s model had to put 100% of their probability mass into the six bio anchors she chose. That’s super misleading/biasing/ignores loads of other possible ways AGI might happen. But I don’t think of this as a necessary part of Ajeya’s model; when I use it, I throw out the six bio anchors and just directly input my probability distribution over OOMs of compute. My distribution is informed by the bio anchors, of course, but that’s not the only thing that informs it.
First, I want to clarify that I feel we’re going into a more interesting place, where there’s a better chance that you might find a point that invalidates Yudkowsky’s argument, and can thus convince him of the value of the model.
But it’s also important to realize that IMO, Yudkowsky is not just saying that biological anchors are bad. The more general problem (which is also developed in this post) is that predicting the Future is really hard. In his own model of AGI timelines, the factor that is basically impossible to predict until you can make AGI is the “how much resources are needed to build AGI”.
So saying “let’s just throw away the biological anchors” doesn’t evade the general counterargument that to predict timelines at all, you need to find information on “how much resources are needed to build AGI”, and that is incredibly hard. If you or Ajeya can argue for actual evidence in that last question, then yeah, I expect Yudkowsky would possibly update on the validity of the timeline estimates.
But at the moment, in this thread, I see no argument like that.