I couldn’t really follow along with my own probabilities because things started wild from the get-go. You say we need to “invent algorithms for transformative AI,” when in fact we already have algorithms that are in-principle general, they’re just orders of magnitude too inefficient, but we’re making gradual algorithmic progress all the time. Checking the pdf, I remain confused about your picture of the world here. Do you think I’m drastically overstating the generality of current ML and the gradualness of algorithmic improvement, such that currently we are totally lacking the ability to build AGI, but after some future discovery (recognizable on its own merits and not some context-dependent “last straw”) we will suddenly be able to?
And your second question is also weird! I don’t really understand the epistemic state of the AI researchers in this hypothetical. They’re supposed to have built something that’s AGI, it just learns slower than humans. How did they get confidence in this fact? I think this question is well-posed enough that I could give a probability for it, except that I’m still confused about how to conditionalize on the first question.
The rest of the questions make plenty of sense, no complaints there.
In terms of the logical structure, I’d point out that inference costs staying low, producing chips, and producing lots of robots are all definitely things that could be routes to transformative AI, but they’re not necessary. The big alternate path missing here is quality. An AI that generates high-quality designs or plans might not have a human equivalent, in which case “what’s the equivalent cost at $25 per human hour” is a wrong question. Producing chips and producing robots could also happen or not happen in any combination and the world could still be transformed by high-quality AI decision-making.
I’m curious and I wonder if I’m missing something that’s obvious to others: What are the algorithms we already have for AGI? What makes you confident they will work before seeing any demonstration of AGI?
So, the maximally impractical but also maximally theoretically rigorous answer here is AIXI-tl.
An almost as impractical answer would be Markov chain Monte Carlo search for well-performing huge neural nets on some objective.
I say MCMC search because I’m confident that there’s some big neural nets that are good at navigating the real world, but any specific efficient training method we know of right now could fail to scale up reliably. Instability being the main problem, rather than getting stuck in local optima.
Dumb but thorough hyperparameter search and RL on a huge neural net should also work. Here we’re adding a few parts of “I am confident in this because of empirical data abut the historical success of scaling up neural nets trained with SGD” to arguments that still mostly rest on “I am confident because of mathematical reasoning about what it means to get a good score at an objective.”
Gotcha. I guess there’s a blurry line between program search and training. Somehow training feels reasonable to me, but something like searching over all possible programs feels unreasonable to me. I suppose the output of such a program search is what I might mean by an algorithm for AGI.
Hyperparameter search and RL on a huge neural net feels wildly underspecified to me. Like, what would be its inputs and outputs, even?
Since I’m fine with saying things that are wildly inefficient, almost any input/output that’s sufficient to reward modeling of the real world (rather than e.g. just playing the abstract game of chess) is sufficient. A present-day example might be self-driving car planning algorithms (though I don’t think any major companies actually use end to end NN planning).
Right, but what inputs and outputs would be sufficient to reward modeling of the real world? I think that might take some exploration and experimentation, and my 60% forecast is the odds of such inquiries succeeding by 2043.
Even with infinite compute, I think it’s quite difficult to build something that generalizes well without overfitting.
what inputs and outputs would be sufficient to reward modeling of the real world?
This is an interesting question but I think it’s not actually relevant. Like, it’s really interesting to think about a thermostat—something who’s only inputs are a thermometer and a clock, and only output is a switch hooked to a heater. Given arbitrarily large computing power and arbitrary amounts of on-distribution training data, will RL ever learn all about the outside world just from temperature patterns? Will it ever learn to deliberately affect the humans around it by turning the heater on and off? Or is it stuck being a dumb thermostat, a local optimum enforced not by the limits of computation but by the structure of the problem it faces?
But people are just going to build AIs attached to video cameras, or screens read by humans, or robot cars, or the internet, which are enough information flow by orders of magnitude, so it’s not super important where the precise boundary is.
Right, I’m not interested in minimum sufficiency. I’m just interested in the straightforward question of what data pipes would we even plug into the algorithm that would result in AGI. Sounds like you think a bunch of cameras and computers would work? To me, it feels like an empirical problem that will take years of research.
Thanks, this was interesting.
I couldn’t really follow along with my own probabilities because things started wild from the get-go. You say we need to “invent algorithms for transformative AI,” when in fact we already have algorithms that are in-principle general, they’re just orders of magnitude too inefficient, but we’re making gradual algorithmic progress all the time. Checking the pdf, I remain confused about your picture of the world here. Do you think I’m drastically overstating the generality of current ML and the gradualness of algorithmic improvement, such that currently we are totally lacking the ability to build AGI, but after some future discovery (recognizable on its own merits and not some context-dependent “last straw”) we will suddenly be able to?
And your second question is also weird! I don’t really understand the epistemic state of the AI researchers in this hypothetical. They’re supposed to have built something that’s AGI, it just learns slower than humans. How did they get confidence in this fact? I think this question is well-posed enough that I could give a probability for it, except that I’m still confused about how to conditionalize on the first question.
The rest of the questions make plenty of sense, no complaints there.
In terms of the logical structure, I’d point out that inference costs staying low, producing chips, and producing lots of robots are all definitely things that could be routes to transformative AI, but they’re not necessary. The big alternate path missing here is quality. An AI that generates high-quality designs or plans might not have a human equivalent, in which case “what’s the equivalent cost at $25 per human hour” is a wrong question. Producing chips and producing robots could also happen or not happen in any combination and the world could still be transformed by high-quality AI decision-making.
I’m curious and I wonder if I’m missing something that’s obvious to others: What are the algorithms we already have for AGI? What makes you confident they will work before seeing any demonstration of AGI?
So, the maximally impractical but also maximally theoretically rigorous answer here is AIXI-tl.
An almost as impractical answer would be Markov chain Monte Carlo search for well-performing huge neural nets on some objective.
I say MCMC search because I’m confident that there’s some big neural nets that are good at navigating the real world, but any specific efficient training method we know of right now could fail to scale up reliably. Instability being the main problem, rather than getting stuck in local optima.
Dumb but thorough hyperparameter search and RL on a huge neural net should also work. Here we’re adding a few parts of “I am confident in this because of empirical data abut the historical success of scaling up neural nets trained with SGD” to arguments that still mostly rest on “I am confident because of mathematical reasoning about what it means to get a good score at an objective.”
Gotcha. I guess there’s a blurry line between program search and training. Somehow training feels reasonable to me, but something like searching over all possible programs feels unreasonable to me. I suppose the output of such a program search is what I might mean by an algorithm for AGI.
Hyperparameter search and RL on a huge neural net feels wildly underspecified to me. Like, what would be its inputs and outputs, even?
Since I’m fine with saying things that are wildly inefficient, almost any input/output that’s sufficient to reward modeling of the real world (rather than e.g. just playing the abstract game of chess) is sufficient. A present-day example might be self-driving car planning algorithms (though I don’t think any major companies actually use end to end NN planning).
Right, but what inputs and outputs would be sufficient to reward modeling of the real world? I think that might take some exploration and experimentation, and my 60% forecast is the odds of such inquiries succeeding by 2043.
Even with infinite compute, I think it’s quite difficult to build something that generalizes well without overfitting.
This is an interesting question but I think it’s not actually relevant. Like, it’s really interesting to think about a thermostat—something who’s only inputs are a thermometer and a clock, and only output is a switch hooked to a heater. Given arbitrarily large computing power and arbitrary amounts of on-distribution training data, will RL ever learn all about the outside world just from temperature patterns? Will it ever learn to deliberately affect the humans around it by turning the heater on and off? Or is it stuck being a dumb thermostat, a local optimum enforced not by the limits of computation but by the structure of the problem it faces?
But people are just going to build AIs attached to video cameras, or screens read by humans, or robot cars, or the internet, which are enough information flow by orders of magnitude, so it’s not super important where the precise boundary is.
Right, I’m not interested in minimum sufficiency. I’m just interested in the straightforward question of what data pipes would we even plug into the algorithm that would result in AGI. Sounds like you think a bunch of cameras and computers would work? To me, it feels like an empirical problem that will take years of research.