(My comment is quite critical, but I want to make it clear that I think doing this exercise is great and important, despite my disagreement with the result of the exercise ;) )
So, for having done the same exercise, I feel that you go far too meta here. And that by doing so, you’re losing most of the actual valuable meta insights of the post. I’m not necessarily saying that your interpretation doesn’t fit what Yudkowsky says, but if the goal is to distill where Yudkowsky is coming from in this specific post, I feel like this comment fails.
The “trick that never works”, in general form, is to go looking in epistemology-space for some grounding in objective reality, which will systematically tend to lead you into these illusory traps.
AFAIU, Yudkowsky is not at all arguing against searching for grounding in reality, he’s arguing for a very specific grounding in reality that I’ve been calling deep knowledge in my post interpreting him on the topic. He’s arguing that there are ways to go beyond the agnosticism of Science (which is very similar to the agnosticism of the outside view and reference class forecasting) between hypotheses that haven’t been falsified yet, and let you move towards the true answer despite the search space being far too large to tractably explore. (See that section in particular of my post, where I go into a lot of details about Yudkowsky’s writing on that in the Sequences).
I also feel like your interpretation conflates the errors that Humbali makes and the ones Simulated-OpenPhil makes, but they’re different in my understanding:
Humbali keeps on criticising Yudkowsky’s confidence, and is the representative of the bad uses of the outside view and reference class forecasting. Which is why a lot of the answers to Humbali focus on deep knowledge (which Yudkowsky refer to here with the extended metaphor of the rails), where it comes from, and why it lets you discard some hypotheses (which is the whole point)
Simulated-OpenPhil mostly defend their own approach and the fact that you can use biological anchors to reason about timelines if you do it carefully. The answer Yudkowsky gives is IMO that they don’t have/give a way of linking the path of evolution in search space and the path of human research in search space, and as such more work and more uncertainty handling on evolution and the other biological anchors don’t give you more information about AGI timelines. The only thing you get out of evolution and biological anchors is the few bits that Yudkowsky already integrates in his model (like that humans will need less optimization power because they’re smarter than evolution), which are not enough to predict timelines.
If I had to state it (and I will probably go into more detail in the post I’m currently writing), my interpretation is that the trick that never works is “using a biological analogy that isn’t closely connected to how human research is optimizing for AGI”. So the way of making a “perpetual motion machine” would be to explain why the specific path of evolution (or other anchors) is related to the path of human optimization, and derive stuff from this.
(My comment is quite critical, but I want to make it clear that I think doing this exercise is great and important, despite my disagreement with the result of the exercise ;) )
So, for having done the same exercise, I feel that you go far too meta here. And that by doing so, you’re losing most of the actual valuable meta insights of the post. I’m not necessarily saying that your interpretation doesn’t fit what Yudkowsky says, but if the goal is to distill where Yudkowsky is coming from in this specific post, I feel like this comment fails.
AFAIU, Yudkowsky is not at all arguing against searching for grounding in reality, he’s arguing for a very specific grounding in reality that I’ve been calling deep knowledge in my post interpreting him on the topic. He’s arguing that there are ways to go beyond the agnosticism of Science (which is very similar to the agnosticism of the outside view and reference class forecasting) between hypotheses that haven’t been falsified yet, and let you move towards the true answer despite the search space being far too large to tractably explore. (See that section in particular of my post, where I go into a lot of details about Yudkowsky’s writing on that in the Sequences).
I also feel like your interpretation conflates the errors that Humbali makes and the ones Simulated-OpenPhil makes, but they’re different in my understanding:
Humbali keeps on criticising Yudkowsky’s confidence, and is the representative of the bad uses of the outside view and reference class forecasting. Which is why a lot of the answers to Humbali focus on deep knowledge (which Yudkowsky refer to here with the extended metaphor of the rails), where it comes from, and why it lets you discard some hypotheses (which is the whole point)
Simulated-OpenPhil mostly defend their own approach and the fact that you can use biological anchors to reason about timelines if you do it carefully. The answer Yudkowsky gives is IMO that they don’t have/give a way of linking the path of evolution in search space and the path of human research in search space, and as such more work and more uncertainty handling on evolution and the other biological anchors don’t give you more information about AGI timelines. The only thing you get out of evolution and biological anchors is the few bits that Yudkowsky already integrates in his model (like that humans will need less optimization power because they’re smarter than evolution), which are not enough to predict timelines.
If I had to state it (and I will probably go into more detail in the post I’m currently writing), my interpretation is that the trick that never works is “using a biological analogy that isn’t closely connected to how human research is optimizing for AGI”. So the way of making a “perpetual motion machine” would be to explain why the specific path of evolution (or other anchors) is related to the path of human optimization, and derive stuff from this.