A summary of the Hanson-Yudkowsky FOOM debate

[Kaj_Sotala]

In late spring this year, Luke tasked me with writing a summary and analysis of the Hanson-Yudkowsky FOOM debate, with the intention of having it eventually published in somewhere. Due to other priorities, this project was put on hold for the time being. Because it doesn’t look like it will be finished in the near future, and because Curiouskid asked to see it, we thought that we might as well share the thing.

I have reorganized the debate, presenting it by topic rather than in chronological order: I start by providing some brief conceptual background that’s useful for understanding Eliezer’s optimization power argument, after which I present his argument. Robin’s various objections follow, after which there is a summary of Robin’s view of how the Singularity will be like, together with Eliezer’s objections to that view. Hopefully, this should make the debate easier to follow. This summary also incorporates material from the 90-minute live debate on the topic that they had in 2011. The full table of contents:

1. Introduction

2. Overview

3. The optimization power argument

   1. Conceptual background

   2. The argument: Yudkowsky

   3. Recursive self-improvement

   4. Hard takeoff

   5. Questioning optimization power: the question of abstractions

   6. Questioning optimization power: the historical record

   7. Questioning optimization power: the UberTool question

4. Hanson’s Singularity scenario

   1. Architecture vs. content, sharing of information

   2. Modularity of knowledge

   3. Local or global singularity?

5. Wrap-up

6. Conclusions

7. References

Here’s the link to the current draft, any feedback is welcomed. Feel free to comment if you know of useful references, if you think I’ve misinterpreted something that was said, or if you think there’s any other problem. I’d also be curious to hear to what extent people think that this outline is easier to follow than the original debate, or whether it’s just as confusing.