Maybe you could write a full post about your views. I’d very much like to read good criticism of singularitarism, but so far your objections aren’t very strong.
The core assumptions in this comment, for example, seem to be not really visible. I’m guessing the idea is something like it’d be really, really hard to do an AI that can do everything a human does, and trying to leave real-world problem-solving to subhuman AIs won’t work.
But no-one’s talking about going after problems in the physical world with a glorified optimizing compiler, so why do you bring up this as the main example? The starting for a lot of current AGI thinking, as far as I’ve understood, is to make an AI with the ability to learn and some means to interact with the world. This AI is then expected to learn to act in the world like humans learn when they grow from newborns to adults.
So is there some kind of basic difference in understanding here, when I’m thinking of AIs as learning semi-autonomous agents, and you’re thinking them as, I guess, some kind of pre-programmed unchanging procedures for doing specific things?
Yes, basically my claim is that an AI of the sort you’re talking about is a job for the world over timescales of generations, not for a single team over timescales of years or decades; it’s hard to prove a negative, and you are right that the comments I’ve been making here don’t—can’t—strongly justify that claim. I’ll think about whether I can put together my reasoning into a full post.
Your position is one that most people assign some probability mass to. However, I get the impression that you’re extremely (over)confident in it. So I look forward to hearing your case.
Ok, thanks. As far as I see, this is the most important core objection then.
There’s actually a second big unknown too before getting into full singularitarism, whether this kind of human-equivalent AI could easily boost itself to strongly superhuman levels with any sort of ease.
But the question of just how difficult it is to build the learning baby AI is really important, and I don’t have any good ideas on how to estimate it except from stuff that can be figured out from biology. The human genome gives us the number of bits that keeps passing through evolution and the general initial complexity for humans, but it’s big enough that without a very good design sense trying to navigate that kind of design space would indeed take generations. Brains and learning have been evolving for a very long time, indicating that the machinery may be very elaborate to get right. Compared to this, symbolic language seems to have popped up very quickly in evolution, which gives reason to believe that once there’s a robust nonverbal cognitive architecture, adding symbolic cognition capabilities isn’t nearly as hard as getting the basic architecture together.
It might also be that the selective pressure in favor of increased intelligence increased suddenly, most likely as a result of competition among humans.
Once a singleton AI becomes marginally smarter than the smartest human, how are we to distinguish between further advances in intelligence as opposed to, say, an increase in it’s ability to impress us with high-tech parlor tricks? Would there be competition between AIs, and if so, over what?
Maybe you could write a full post about your views. I’d very much like to read good criticism of singularitarism, but so far your objections aren’t very strong.
The core assumptions in this comment, for example, seem to be not really visible. I’m guessing the idea is something like it’d be really, really hard to do an AI that can do everything a human does, and trying to leave real-world problem-solving to subhuman AIs won’t work.
But no-one’s talking about going after problems in the physical world with a glorified optimizing compiler, so why do you bring up this as the main example? The starting for a lot of current AGI thinking, as far as I’ve understood, is to make an AI with the ability to learn and some means to interact with the world. This AI is then expected to learn to act in the world like humans learn when they grow from newborns to adults.
So is there some kind of basic difference in understanding here, when I’m thinking of AIs as learning semi-autonomous agents, and you’re thinking them as, I guess, some kind of pre-programmed unchanging procedures for doing specific things?
Yes, basically my claim is that an AI of the sort you’re talking about is a job for the world over timescales of generations, not for a single team over timescales of years or decades; it’s hard to prove a negative, and you are right that the comments I’ve been making here don’t—can’t—strongly justify that claim. I’ll think about whether I can put together my reasoning into a full post.
Your position is one that most people assign some probability mass to. However, I get the impression that you’re extremely (over)confident in it. So I look forward to hearing your case.
Ok, thanks. As far as I see, this is the most important core objection then.
There’s actually a second big unknown too before getting into full singularitarism, whether this kind of human-equivalent AI could easily boost itself to strongly superhuman levels with any sort of ease.
But the question of just how difficult it is to build the learning baby AI is really important, and I don’t have any good ideas on how to estimate it except from stuff that can be figured out from biology. The human genome gives us the number of bits that keeps passing through evolution and the general initial complexity for humans, but it’s big enough that without a very good design sense trying to navigate that kind of design space would indeed take generations. Brains and learning have been evolving for a very long time, indicating that the machinery may be very elaborate to get right. Compared to this, symbolic language seems to have popped up very quickly in evolution, which gives reason to believe that once there’s a robust nonverbal cognitive architecture, adding symbolic cognition capabilities isn’t nearly as hard as getting the basic architecture together.
It might also be that the selective pressure in favor of increased intelligence increased suddenly, most likely as a result of competition among humans.
Once a singleton AI becomes marginally smarter than the smartest human, how are we to distinguish between further advances in intelligence as opposed to, say, an increase in it’s ability to impress us with high-tech parlor tricks? Would there be competition between AIs, and if so, over what?