Much of intelligent behavior consists of search space problems, which tend to parallelize well. At the bare minimum, it ought to be able to run more copies of itself as its access to hardware increases, which is still pretty scary. I do suspect that there’s a logarithmic component to intelligence, as at some point you’ve already sampled the future outcome space thoroughly enough that most of the new bits of prediction you’re getting back are redundant—but the point of diminishing returns could be very, very high.
What about manipulators? I havent, as far as I know, seen much analysis of manipulation capabilities (and counter-manipulation) on Less Wrong. Mostly there is the AI-box issue (a really freaking big deal, I agree) and then it seems to be considered here that the AI will quickly invent super-nanotech, will not be able to be impeded in its progress, and will become godlike very quickly. I’ve seen some arguments for this, but never a really good analysis, and it’s the remaining reason I am a bit skeptical of the power of FOOM.
The way I think about it, you can set lower bounds on the abilities of an AI by thinking of it as an economic agent. Now, at some point, that abstraction becomes pretty meaningless, but in the early days, a powerful, bootstrapping optimization agent could still incorporate, hire or persuade people to do things for it, make rapid innovations in various fields, have machines made of various types, and generally wind up running the place fairly quickly, even if the problem of bootstrapping versatile nanomachines from current technology turns out to be time-consuming for a superintelligence. I would imagine that nanotech would be where it’d go in the longer run, but that might take time—I don’t know, I don’t know enough about the subject. But even without strong Drexlerian nanotechnology, it’s still possible to get an awful lot done.
Much of intelligent behavior consists of search space problems, which tend to parallelize well. At the bare minimum, it ought to be able to run more copies of itself as its access to hardware increases, which is still pretty scary. I do suspect that there’s a logarithmic component to intelligence, as at some point you’ve already sampled the future outcome space thoroughly enough that most of the new bits of prediction you’re getting back are redundant—but the point of diminishing returns could be very, very high.
What about manipulators? I havent, as far as I know, seen much analysis of manipulation capabilities (and counter-manipulation) on Less Wrong. Mostly there is the AI-box issue (a really freaking big deal, I agree) and then it seems to be considered here that the AI will quickly invent super-nanotech, will not be able to be impeded in its progress, and will become godlike very quickly. I’ve seen some arguments for this, but never a really good analysis, and it’s the remaining reason I am a bit skeptical of the power of FOOM.
The way I think about it, you can set lower bounds on the abilities of an AI by thinking of it as an economic agent. Now, at some point, that abstraction becomes pretty meaningless, but in the early days, a powerful, bootstrapping optimization agent could still incorporate, hire or persuade people to do things for it, make rapid innovations in various fields, have machines made of various types, and generally wind up running the place fairly quickly, even if the problem of bootstrapping versatile nanomachines from current technology turns out to be time-consuming for a superintelligence. I would imagine that nanotech would be where it’d go in the longer run, but that might take time—I don’t know, I don’t know enough about the subject. But even without strong Drexlerian nanotechnology, it’s still possible to get an awful lot done.
That much I do totally agree.