It would be both surprising news, and immensely bad news, to learn that only a tiny group of people could (or should) work on such a problem—that would mean applying vastly less parallel “compute” to the problem, relative to what is theoretically available, and that when the problem is forbiddingly difficult to begin with.
I have substantial probability on an even worse state: there’s *multiple* people or groups of people, *each* of which is *separately* necessary for AGI to go well. Like, metaphorically, your liver, heart, and brain would each be justified in having a “rarity narrative”. In other words, yes, the parallel compute is necessary—there’s lots of data and ideas and thinking that has to happen—but there’s a continuum of how fungible the compute is relative to the problems that need to be solved, and there’s plenty of stuff at the “not very fungible but very important” end. Blood is fungible (though you definitely need it), but you can’t just lose a heart valve, or your hippocampus, and be fine.
I didn’t mention it in the comment, but having a larger pool of researchers is not only useful for doing “ordinary” work in parallel—it also increases the rate at which your research community discovers and accumulates outlier-level, irreplaceable genius figures of the Euler/Gauss kind.
If there are some such figures already in the community, great, but there are presumably others yet to be discovered. That their impact is currently potential, not actual, does not make its sacrifice any less damaging.
Yep. (And I’m happy this overall discussion is happening, partly because, assuming rarity narratives are part of what leads to all this destructive psychic stuff as you described, then if a research community wants to work with people about whom rarity narratives would actually be somewhat *true*, the research community has as an important subgoal to figure out how to have true rarity narratives in a non-harmful way.)
I have substantial probability on an even worse state: there’s *multiple* people or groups of people, *each* of which is *separately* necessary for AGI to go well. Like, metaphorically, your liver, heart, and brain would each be justified in having a “rarity narrative”. In other words, yes, the parallel compute is necessary—there’s lots of data and ideas and thinking that has to happen—but there’s a continuum of how fungible the compute is relative to the problems that need to be solved, and there’s plenty of stuff at the “not very fungible but very important” end. Blood is fungible (though you definitely need it), but you can’t just lose a heart valve, or your hippocampus, and be fine.
I didn’t mention it in the comment, but having a larger pool of researchers is not only useful for doing “ordinary” work in parallel—it also increases the rate at which your research community discovers and accumulates outlier-level, irreplaceable genius figures of the Euler/Gauss kind.
If there are some such figures already in the community, great, but there are presumably others yet to be discovered. That their impact is currently potential, not actual, does not make its sacrifice any less damaging.
Yep. (And I’m happy this overall discussion is happening, partly because, assuming rarity narratives are part of what leads to all this destructive psychic stuff as you described, then if a research community wants to work with people about whom rarity narratives would actually be somewhat *true*, the research community has as an important subgoal to figure out how to have true rarity narratives in a non-harmful way.)