I admit, I think this is kind of a crux, but let me get down to this statement:
I want to flag this as an assumption that isn’t obvious. If this were true for the problems we care about, we could solve them by employing a lot of humans.
One big difference between a human-level AI and a real human is coordination costs: Even without advanced decision theories like FDT/UDT/LDT, the ability to have millions of copies of an AI makes it possible for them to all have similar values, and divergences between them are more controllable in a virtual environment than a physical environment.
But my more substantive claim is that lots of how progress is made in the real world is because population growth allows for more complicated economies, more ability to specialize without losing essential skills, and just simply more data to deal with reality, and alignment, including strong alignment is not different here.
Indeed, I’d argue that a lot more alignment progress happened in the 2022-2024 period than the 2005-2015 period, and while I don’t credit it all to population growth of alignment researchers, I do think a reasonably significant amount of the progress happened because we got more people into alignment.
Intelligence/IQ is always good, but not a dealbreaker as long as you can substitute it with a larger population.
See these quotes from Carl Shulman here for why:
Yeah. In science the association with things like scientific output, prizes, things like that, there’s a strong correlation and it seems like an exponential effect. It’s not a binary drop-off. There would be levels at which people cannot learn the relevant fields, they can’t keep the skills in mind faster than they forget them. It’s not a divide where there’s Einstein and the group that is 10 times as populous as that just can’t do it. Or the group that’s 100 times as populous as that suddenly can’t do it. The ability to do the things earlier with less evidence and such falls off at a faster rate in Mathematics and theoretical Physics and such than in most fields.
Yes, people would have discovered general relativity just from the overwhelming data and other people would have done it after Einstein.
Intelligence/IQ is always good, but not a dealbreaker as long as you can substitute it with a larger population.
IMO this is pretty obviously wrong. There are some kinds of problem solving that scales poorly with population, just as there are some computations that scale poorly with parallelisation.
When I said “problems we care about”, I was referring to a cluster of problems that very strongly appear to not scale well with population. Maybe this is an intuitive picture of the cluster of problems I’m referring to.
When I said “problems we care about”, I was referring to a cluster of problems that very strongly appear to not scale well with population. Maybe this is an intuitive picture of the cluster of problems I’m referring to.
I think the problem identified here is in large part a demand problem, in that lots of AI people only wanted AI capabilities, and didn’t care for AI interpretability at all, so once the scaling happened, a lot of the focus went purely to AI scaling.
(Which is an interesting example of Goodhart’s law in action, perhaps.)
IMO this is pretty obviously wrong. There are some kinds of problem solving that scales poorly with population, just as there are some computations that scale poorly with parallelisation.
I definitely agree that there exist such problems where the scaling with population is pretty bad, but I’ll give 2 responses here:
The differences between a human level AI and an actual human are the ability to coordinate and share ontologies better between millions of instances, so the common problems that arise when trying to factorize out problems are greatly reduced.
I think that while there are serial bottlenecks to lots of problem solving in the real world such that it prevents hyperfast outcomes, I don’t think that serial bottlenecks are the dominating factor, because the stuff that is parallelizable like good execution is often far more valuable than the inherently serial computations like deep/original ideas.
I admit, I think this is kind of a crux, but let me get down to this statement:
One big difference between a human-level AI and a real human is coordination costs: Even without advanced decision theories like FDT/UDT/LDT, the ability to have millions of copies of an AI makes it possible for them to all have similar values, and divergences between them are more controllable in a virtual environment than a physical environment.
But my more substantive claim is that lots of how progress is made in the real world is because population growth allows for more complicated economies, more ability to specialize without losing essential skills, and just simply more data to deal with reality, and alignment, including strong alignment is not different here.
Indeed, I’d argue that a lot more alignment progress happened in the 2022-2024 period than the 2005-2015 period, and while I don’t credit it all to population growth of alignment researchers, I do think a reasonably significant amount of the progress happened because we got more people into alignment.
Intelligence/IQ is always good, but not a dealbreaker as long as you can substitute it with a larger population.
See these quotes from Carl Shulman here for why:
The link for these quotes is here below:
https://www.lesswrong.com/posts/BdPjLDG3PBjZLd5QY/carl-shulman-on-dwarkesh-podcast-june-2023#Can_we_detect_deception_
IMO this is pretty obviously wrong. There are some kinds of problem solving that scales poorly with population, just as there are some computations that scale poorly with parallelisation.
E.g. project euler problems.
When I said “problems we care about”, I was referring to a cluster of problems that very strongly appear to not scale well with population. Maybe this is an intuitive picture of the cluster of problems I’m referring to.
On this:
I think the problem identified here is in large part a demand problem, in that lots of AI people only wanted AI capabilities, and didn’t care for AI interpretability at all, so once the scaling happened, a lot of the focus went purely to AI scaling.
(Which is an interesting example of Goodhart’s law in action, perhaps.)
See here:
https://www.lesswrong.com/posts/gXinMpNJcXXgSTEpn/ai-craftsmanship#Qm8Kg7PjZoPTyxrr6
I definitely agree that there exist such problems where the scaling with population is pretty bad, but I’ll give 2 responses here:
The differences between a human level AI and an actual human are the ability to coordinate and share ontologies better between millions of instances, so the common problems that arise when trying to factorize out problems are greatly reduced.
I think that while there are serial bottlenecks to lots of problem solving in the real world such that it prevents hyperfast outcomes, I don’t think that serial bottlenecks are the dominating factor, because the stuff that is parallelizable like good execution is often far more valuable than the inherently serial computations like deep/original ideas.