Eliezer’s main point of his ~20k words isn’t really what I want to defend, but I will say a few words about how I would constructively interpret it. I think Eliezer’s main claim is that he has an intuitive system-1 model of the inhomogeneous Poisson process that emits working AGI systems, and that this model isn’t informed by the compute equivalents of biological anchors, and that his lack of being informed by that isn’t a mistake. I’m not sure if he’s actually making the stronger claim that anyone whose model is informed by the biological anchors is making a mistake, but if so, I don’t agree. My own model is somewhat informed by biological anchors; it’s more informed by what the TAI report calls “subjective impressiveness extrapolation”, extrapolations on benchmark performance, and some vague sense of other point processes that emit AI winters and major breakthroughs. Someone who has total Knightean uncertainty and no intuitive models of how AGI comes about would surely do well to adopt OpenPhil’s distribution as a prior.
Eliezer’s main point of his ~20k words isn’t really what I want to defend, but I will say a few words about how I would constructively interpret it. I think Eliezer’s main claim is that he has an intuitive system-1 model of the inhomogeneous Poisson process that emits working AGI systems, and that this model isn’t informed by the compute equivalents of biological anchors, and that his lack of being informed by that isn’t a mistake. I’m not sure if he’s actually making the stronger claim that anyone whose model is informed by the biological anchors is making a mistake, but if so, I don’t agree. My own model is somewhat informed by biological anchors; it’s more informed by what the TAI report calls “subjective impressiveness extrapolation”, extrapolations on benchmark performance, and some vague sense of other point processes that emit AI winters and major breakthroughs. Someone who has total Knightean uncertainty and no intuitive models of how AGI comes about would surely do well to adopt OpenPhil’s distribution as a prior.
(I’m not sure whether your summary captures Eliezer’s view, but strong-upvoted for what strikes me as a reasonable attempt.)