Questions about the standard-university-textbook from the future that tells us how to build an AGI. I’ll take answers on any of these!
Where is ML in this textbook? Is it under a section called “god-forsaken approaches” or does it play a key role? Follow-up: Where is logical induction?
If running superintelligent AGIs didn’t kill you and death was cancelled in general, how long would it take you to write the textbook?
Is there anything else you can share about this textbook? Do you know any of the other chapter names?
I don’t think there is an “AGI textbook” any more than there is an “industrialization textbook.” There are lots of books about general principles and useful kinds of machines. That said, if I had to make wild guesses about roughly what that future understanding would look like:
There is a recognizable concept of “learning” meaning something like “search for policies that perform well in past or simulated situations.” That plays a large role, comparably important to planning or Bayesian inference. Logical induction is likely an elaboration of Bayesian inference that receives relatively little airtime except in specialized discussions.
This one is tougher given that I don’t know what “the textbook” is. And I guess in the story all other humans are magically disappeared? If I was stuck with a single AWS cluster from 2022 and given unlimited time, I’d wildly guess that it would take me something between 1e4 and 1e8 years to create an autopoetic AI that obsoleted my own contributions (mostly because serial time is extremely valuable and I have a lot of compute). Writing the textbook does not seem like very much work after having done the deed?
I’d roughly guess big sections on learning, inference, planning, alignment, and clever algorithms for all of the above. I’d guess maybe 50% of content is smart versions of stuff we know now and 50% is stuff we didn’t figure out at the time, but it depends a lot on how you define this textbook.
I’m mostly going to answer assuming that there’s not some incredibly different paradigm (i.e. something as different from ML as ML is from expert systems). I do think the probability of “incredibly different paradigm” is low.
I’m also going to answer about the textbook at, idk, the point at which GDP doubles every 8 years. (To avoid talking about the post-Singularity textbook that explains how to build a superintelligence with clearly understood “intelligence algorithms” that can run easily on one of today’s laptops, which I know very little about.)
I think I roughly agree with Paul if you are talking about the textbook that tells us how to build the best systems for the tasks that we want to do. (Analogy: today’s textbook for self-driving cars.) That being said, I think that much of the improvement over time will be driven by improvements specifically in ML. (Analogy: today’s textbook for deep learning.) So we can talk about that textbook as well.
It’s a textbook that’s entirely about “finding good programs through a large, efficient search with a stringent goal”, which we currently call ML. The content may be primarily some new approach for achieving this, with neural nets being a historical footnote, or it might be entirely about neural nets (though presumably with new architectures or other changes from today). Logical induction doesn’t appear in the textbook.
Jeez, who knows. If I intuitively query my brain here, it mostly doesn’t have an answer; a thousand vs. million vs. billion years don’t really change my intuitive predictions about what I’d get done. So we can instead back it out from other estimates. Given timelines of 10^1 − 10^2 years, and, idk, ~10^6 humans working on the problem near the end, seems like I’m implicitly predicting ~10^7 human-years of effort in our actual world. Then you have to adjust for a ton of factors, e.g. my quality relative to the average, the importance of serial thinking time, the benefit that real-world humans get from AI products that I won’t get, the difficulty of exploration in thought-space by 1 person vs. 10^6 people, etc. Maybe I end up at ~10^5 years as a median estimate with wide uncertainty (especially on the right tail).
Jeez, who knows. Probably chapters / sections on how to define search spaces of programs (currently, “architectures”), efficient search algorithms within those spaces (currently, “gradient descent” and “loss functions”), how to set a stringent goal (currently, “what dataset to use”).
Where is ML in this textbook? Is it under a section called “god-forsaken approaches” or does it play a key role? Follow-up: Where is logical induction?
Key role, but most current ML is in the “applied” section, where the “theory” section instead explains the principles by which neural nets (or future architectures) work on the inside. Logical induction is a sidebar at some point explaining the theoretical ideal we’re working towards, like I assume AIXI is in some textbooks.
Is there anything else you can share about this textbook? Do you know any of the other chapter names?
Questions about the standard-university-textbook from the future that tells us how to build an AGI. I’ll take answers on any of these!
Where is ML in this textbook? Is it under a section called “god-forsaken approaches” or does it play a key role? Follow-up: Where is logical induction?
If running superintelligent AGIs didn’t kill you and death was cancelled in general, how long would it take you to write the textbook?
Is there anything else you can share about this textbook? Do you know any of the other chapter names?
I’m going to try and write a table of contents for the textbook, just because it seems like a fun exercise.
Epistemic status: unbridled speculation
Volume I: Foundation
Preface [mentioning, ofc, the infamous incident of 2041]
Chapter 0: Introduction
Part I: Statistical Learning Theory
Chapter 1: Offline Learning [VC theory and Watanabe’s singular learning theory are both special cases of what’s in this chapter]
Chapter 2: Online Learning [infra-Bayesianism is introduced here, Garrabrant induction too]
Chapter 3: Reinforcement Learning
Chapter 4: Lifelong Learning [this chapter deals with traps, unobservable rewards and long-term planning]
Part II: Computational Learning Theory
Chapter 5: Algebraic Classes [the theory of SVMs is a special case of what’s explained here]
Chapter 6: Circuits [learning various class of circuits]
Chapter 7: Neural Networks
Chapter 8: ???
Chapter 9: Reflective Learning [some version of Turing reinforcement learning comes here]
Part III: Universal Priors
Chapter 10: Solomonoff’s Prior [including regret analysis using algorithmic statistics]
Chapter 11: Bounded Simplicity Priors
Chapter 12: ??? [might involve: causality, time hierarchies, logical languages, noise-tolerant computation...]
Chapter 13: Physicalism and the Bridge Transform
Chapter 14: Intelligence Measures
Part IV: Multi-Agent Systems
Chapter 15: Impatient Games
Chapter 16: Population Games
Chapter 17: Space Bounds and Superrationality
Chapter 18: Language [cheap talk, agreement theorems...]
Part V: Alignment Protocols
Chapter 19: Quantilization
Chapter 20: Malign Capacity Bounds [about confidence thresholds and consensus algorithms]
Chapter 21: Value Learning [using the intelligence measures from chapter 14]
Chapter 22: ??? [debate and/or some version of IDA might be here, or not]
Chapter 23: ???
Volume II: Algorithms [about efficient algorithms for practical models of computation, and various trade-offs.]
???
Volume III: Applications
???
I don’t think there is an “AGI textbook” any more than there is an “industrialization textbook.” There are lots of books about general principles and useful kinds of machines. That said, if I had to make wild guesses about roughly what that future understanding would look like:
There is a recognizable concept of “learning” meaning something like “search for policies that perform well in past or simulated situations.” That plays a large role, comparably important to planning or Bayesian inference. Logical induction is likely an elaboration of Bayesian inference that receives relatively little airtime except in specialized discussions.
This one is tougher given that I don’t know what “the textbook” is. And I guess in the story all other humans are magically disappeared? If I was stuck with a single AWS cluster from 2022 and given unlimited time, I’d wildly guess that it would take me something between 1e4 and 1e8 years to create an autopoetic AI that obsoleted my own contributions (mostly because serial time is extremely valuable and I have a lot of compute). Writing the textbook does not seem like very much work after having done the deed?
I’d roughly guess big sections on learning, inference, planning, alignment, and clever algorithms for all of the above. I’d guess maybe 50% of content is smart versions of stuff we know now and 50% is stuff we didn’t figure out at the time, but it depends a lot on how you define this textbook.
I’m mostly going to answer assuming that there’s not some incredibly different paradigm (i.e. something as different from ML as ML is from expert systems). I do think the probability of “incredibly different paradigm” is low.
I’m also going to answer about the textbook at, idk, the point at which GDP doubles every 8 years. (To avoid talking about the post-Singularity textbook that explains how to build a superintelligence with clearly understood “intelligence algorithms” that can run easily on one of today’s laptops, which I know very little about.)
I think I roughly agree with Paul if you are talking about the textbook that tells us how to build the best systems for the tasks that we want to do. (Analogy: today’s textbook for self-driving cars.) That being said, I think that much of the improvement over time will be driven by improvements specifically in ML. (Analogy: today’s textbook for deep learning.) So we can talk about that textbook as well.
It’s a textbook that’s entirely about “finding good programs through a large, efficient search with a stringent goal”, which we currently call ML. The content may be primarily some new approach for achieving this, with neural nets being a historical footnote, or it might be entirely about neural nets (though presumably with new architectures or other changes from today). Logical induction doesn’t appear in the textbook.
Jeez, who knows. If I intuitively query my brain here, it mostly doesn’t have an answer; a thousand vs. million vs. billion years don’t really change my intuitive predictions about what I’d get done. So we can instead back it out from other estimates. Given timelines of 10^1 − 10^2 years, and, idk, ~10^6 humans working on the problem near the end, seems like I’m implicitly predicting ~10^7 human-years of effort in our actual world. Then you have to adjust for a ton of factors, e.g. my quality relative to the average, the importance of serial thinking time, the benefit that real-world humans get from AI products that I won’t get, the difficulty of exploration in thought-space by 1 person vs. 10^6 people, etc. Maybe I end up at ~10^5 years as a median estimate with wide uncertainty (especially on the right tail).
Jeez, who knows. Probably chapters / sections on how to define search spaces of programs (currently, “architectures”), efficient search algorithms within those spaces (currently, “gradient descent” and “loss functions”), how to set a stringent goal (currently, “what dataset to use”).
Where is ML in this textbook? Is it under a section called “god-forsaken approaches” or does it play a key role? Follow-up: Where is logical induction?
Key role, but most current ML is in the “applied” section, where the “theory” section instead explains the principles by which neural nets (or future architectures) work on the inside. Logical induction is a sidebar at some point explaining the theoretical ideal we’re working towards, like I assume AIXI is in some textbooks.
Is there anything else you can share about this textbook? Do you know any of the other chapter names?
Planning, Abstraction, Reasoning, Self-awareness.