Relatedly, Eliezer saying “Robin was wrong for doubting RSI; if other crazy stuff will happen before RSI then he’s just even more wrong” seems wrong.
Eliezer’s argument for localized Foom (and for localized RSI in particular) wasn’t ‘no cool tech will happen prior to AGI; therefore AGI will produce a localized Foom’. If it were, then it would indeed be bizarre to cite an example of pre-AGI cool tech (AlphaGo Zero) and say ‘aha, evidence for localized Foom’.
Rather, Eliezer’s argument for localized Foom and localized RSI was:
It’s not hard to improve on human brains.
You can improve on human brains with relatively simple algorithms; you don’t need a huge library of crucial proprietary components that are scattered all over the economy and need to be carefully accumulated and assembled.
The important dimensions for improvement aren’t just ‘how fast or high-fidelity is the system’s process of learning human culture?’.
General intelligence isn’t just a bunch of heterogeneous domain-specific narrow modules glued together.
Insofar as general intelligence decomposes into parts/modules, these modules work a lot better as one brain than as separate heterogeneous AIs scattered around the world. (See Permitted Possibilities, & Locality.)
I.e.:
Localized Foom isn’t blocked by humans being near a cognitive ceiling in general.
Localized Foom isn’t blocked by “there’s no algorithmic progress on AI” or “there’s no simple, generally applicable algorithmic progress on AI”.
Localized Foom isn’t blocked by “humans are only amazing because we can accumulate culture; and humans already cross that threshold, so it won’t be that big of a deal if something else crosses the exact same threshold; and since AI will be dependent on painstakingly accumulated human culture in the same way we are, it won’t be able to suddenly pull ahead”.
Localized Foom isn’t blocked by “getting an AI that’s par-human at one narrow domain or task won’t mean you have an AI that’s par-human at anything else”.
Localized Foom isn’t blocked by “there’s no special advantage to doing the cognition inside a brain, vs. doing it in distributed fashion across many different AIs in the world that work very differently”.
AlphaGo and its successors were indeed evidence for these claims, to the extent you can get evidence for them by looking at performance on board games.
Insofar as Robin thinks ems come before AI, impressive AI progress is also evidence for Eliezer’s view over Robin’s; but this wasn’t the focus of the Foom debate or of Eliezer’s follow-up. This would be much more of a crux if Robin endorsed ‘AGI quickly gets you localized Foom, but AGI doesn’t happen until after ems’; but I don’t think he endorses a story like that. (Though he does endorse ‘AGI doesn’t happen until after ems’, to the extent ‘AGI’ makes sense as a category in Robin’s ontology.)
AlphaGo and its successors are also evidence that progress often surprises people and comes in spurts: there weren’t a ton of people loudly saying ‘if a major AGI group tries hard in the next 1-4 years, we’ll immediately blast past the human range of Go ability even though AI has currently never beaten a Go professional’ one, two, or four years before AlphaGo. But this is more directly relevant to the Paul-Eliezer disagreement than the Robin-Eliezer one, and it’s weaker evidence insofar as Go isn’t economically important.
[...] When I remarked upon how it sure looked to me like humans had an architectural improvement over chimpanzees that counted for a lot, Hanson replied that this seemed to him like a one-time gain from allowing the cultural accumulation of knowledge.
I emphasize how all the mighty human edifice of Go knowledge, the joseki and tactics developed over centuries of play, the experts teaching children from an early age, was entirely discarded by AlphaGo Zero with a subsequent performance improvement. These mighty edifices of human knowledge, as I understand the Hansonian thesis, are supposed to be the bulwark against rapid gains in AI capability across multiple domains at once. I said, “Human intelligence is crap and our accumulated skills are crap,” and this appears to have been borne out.
Similarly, single research labs like DeepMind are not supposed to pull far ahead of the general ecology, because adapting AI to any particular domain is supposed to require lots of components developed all over the place by a market ecology that makes those components available to other companies. AlphaGo Zero is much simpler than that. To the extent that nobody else can run out and build AlphaGo Zero, it’s either because Google has Tensor Processing Units that aren’t generally available, or because DeepMind has a silo of expertise for being able to actually make use of existing ideas like ResNets, or both.
Sheer speed of capability gain should also be highlighted here. Most of my argument for FOOM in the Yudkowsky-Hanson debate was about self-improvement and what happens when an optimization loop is folded in on itself. Though it wasn’t necessary to my argument, the fact that Go play went from “nobody has come close to winning against a professional” to “so strongly superhuman they’re not really bothering any more” over two years just because that’s what happens when you improve and simplify the architecture, says you don’t even need self-improvement to get things that look like FOOM.
Yes, Go is a closed system allowing for self-play. It still took humans centuries to learn how to play it. Perhaps the new Hansonian bulwark against rapid capability gain can be that the environment has lots of empirical bits that are supposed to be very hard to learn, even in the limit of AI thoughts fast enough to blow past centuries of human-style learning in 3 days; and that humans have learned these vital bits over centuries of cultural accumulation of knowledge, even though we know that humans take centuries to do 3 days of AI learning when humans have all the empirical bits they need; and that AIs cannot absorb this knowledge very quickly using “architecture”, even though humans learn it from each other using architecture. If so, then let’s write down this new world-wrecking assumption (that is, the world ends if the assumption is false) and be on the lookout for further evidence that this assumption might perhaps be wrong.
AlphaGo clearly isn’t a general AI. There’s obviously stuff humans do that make us much more general than AlphaGo, and AlphaGo obviously doesn’t do that. However, if even with the human special sauce we’re to expect AGI capabilities to be slow, domain-specific, and requiring feed-in from a big market ecology, then the situation we see without human-equivalent generality special sauce should not look like this.
To put it another way, I put a lot of emphasis in my debate on recursive self-improvement and the remarkable jump in generality across the change from primate intelligence to human intelligence. It doesn’t mean we can’t get info about speed of capability gains without self-improvement. It doesn’t mean we can’t get info about the importance and generality of algorithms without the general intelligence trick. The debate can start to settle for fast capability gains before we even get to what I saw as the good parts; I wouldn’t have predicted AlphaGo and lost money betting against the speed of its capability gains, because reality held a more extreme position than I did on the Yudkowsky-Hanson spectrum.
I think it’s good to go back to this specific quote and think about how it compares to AGI progress.
A difference I think Paul has mentioned before is that Go was not a competitive industry and competitive industries will have smaller capability jumps. Assuming this is true, I also wonder whether the secret sauce for AGI will be within the main competitive target of the AGI industry.
The thing the industry is calling AGI and targeting may end up being a specific style of shallow deployable intelligence when “real” AGI is a different style of “deeper” intelligence (with, say, less economic value at partial stages and therefore relatively unpursued). This would allow a huge jump like AlphaGo in AGI even in a competitive industry targeting AGI.
Both possibilities seem plausible to me and I’d like to hear arguments either way.
Eliezer’s argument for localized Foom (and for localized RSI in particular) wasn’t ‘no cool tech will happen prior to AGI; therefore AGI will produce a localized Foom’. If it were, then it would indeed be bizarre to cite an example of pre-AGI cool tech (AlphaGo Zero) and say ‘aha, evidence for localized Foom’.
Rather, Eliezer’s argument for localized Foom and localized RSI was:
It’s not hard to improve on human brains.
You can improve on human brains with relatively simple algorithms; you don’t need a huge library of crucial proprietary components that are scattered all over the economy and need to be carefully accumulated and assembled.
The important dimensions for improvement aren’t just ‘how fast or high-fidelity is the system’s process of learning human culture?’.
General intelligence isn’t just a bunch of heterogeneous domain-specific narrow modules glued together.
Insofar as general intelligence decomposes into parts/modules, these modules work a lot better as one brain than as separate heterogeneous AIs scattered around the world. (See Permitted Possibilities, & Locality.)
I.e.:
Localized Foom isn’t blocked by humans being near a cognitive ceiling in general.
Localized Foom isn’t blocked by “there’s no algorithmic progress on AI” or “there’s no simple, generally applicable algorithmic progress on AI”.
Localized Foom isn’t blocked by “humans are only amazing because we can accumulate culture; and humans already cross that threshold, so it won’t be that big of a deal if something else crosses the exact same threshold; and since AI will be dependent on painstakingly accumulated human culture in the same way we are, it won’t be able to suddenly pull ahead”.
Localized Foom isn’t blocked by “getting an AI that’s par-human at one narrow domain or task won’t mean you have an AI that’s par-human at anything else”.
Localized Foom isn’t blocked by “there’s no special advantage to doing the cognition inside a brain, vs. doing it in distributed fashion across many different AIs in the world that work very differently”.
AlphaGo and its successors were indeed evidence for these claims, to the extent you can get evidence for them by looking at performance on board games.
Insofar as Robin thinks ems come before AI, impressive AI progress is also evidence for Eliezer’s view over Robin’s; but this wasn’t the focus of the Foom debate or of Eliezer’s follow-up. This would be much more of a crux if Robin endorsed ‘AGI quickly gets you localized Foom, but AGI doesn’t happen until after ems’; but I don’t think he endorses a story like that. (Though he does endorse ‘AGI doesn’t happen until after ems’, to the extent ‘AGI’ makes sense as a category in Robin’s ontology.)
AlphaGo and its successors are also evidence that progress often surprises people and comes in spurts: there weren’t a ton of people loudly saying ‘if a major AGI group tries hard in the next 1-4 years, we’ll immediately blast past the human range of Go ability even though AI has currently never beaten a Go professional’ one, two, or four years before AlphaGo. But this is more directly relevant to the Paul-Eliezer disagreement than the Robin-Eliezer one, and it’s weaker evidence insofar as Go isn’t economically important.
Quoting Eliezer’s AlphaGo Zero and the Foom Debate:
I think it’s good to go back to this specific quote and think about how it compares to AGI progress.
A difference I think Paul has mentioned before is that Go was not a competitive industry and competitive industries will have smaller capability jumps. Assuming this is true, I also wonder whether the secret sauce for AGI will be within the main competitive target of the AGI industry.
The thing the industry is calling AGI and targeting may end up being a specific style of shallow deployable intelligence when “real” AGI is a different style of “deeper” intelligence (with, say, less economic value at partial stages and therefore relatively unpursued). This would allow a huge jump like AlphaGo in AGI even in a competitive industry targeting AGI.
Both possibilities seem plausible to me and I’d like to hear arguments either way.