I don’t quite follow you on the intelligence explosion issue. For instance, why does a strong argument against the intelligence explosion hypothesis need to show that a feedback loop is unlikely? Couldn’t we believe that it is likely, but not likely to be very rapid for a while? For instance, there is probably a feedback loop in intelligence already, where humans with better thoughts and equipment are effectively smarter, and can then devise better thoughts and equipment. But this has been true for a while, and is a fairly slow process (at least for now, relative to our ability to deal with things).
Yeah, upon rereading that response, I think I created a few non sequiturs in revision. I’m not even 100% sure what I meant by some bits. I think the arguments that now seem confusing were me was saying that by putting an intelligence feedback loop in the reference class of “feedback loops in general” and then using that to forecast low impact, the thing that is doing most of the work is simply how low impact most stuff is.
A nuclear bomb (or a raindrop forming, or tipping back a little too far in your chair) can be modeled as a feedback loop through several orders of magnitude of power output, and then eventually that model breaks down and the explosion dissipates, and the world might be a little scarred and radioactive, but it is overall not much different. But if your AI increased by several orders of magnitude in intelligence (let’s just pretend that’s meaningful for a second), I would expect that to be a much bigger deal, just because the thing that’s increasing is different. That is, I was thinking that the implicit model used by the reference class argument from the original link seems to predict local advantages in AI, but predict *against* those local advantages being important to the world at large, which I think is putting the most weight on the weakest link.
Part of this picture I had comes from what I’m imagining as prototypical reference class members—note that I only imagined self-sustaining feedback, not “subcritical” feedback. In retrospect, this seems to be begging the question somewhat—subcritical feedback speeds up progress, but doesn’t necessarily concentrate it, unless there is some specific threshold effect for getting that feedback. Another feature of my prototypes was that they’re out-of-equilibrium rather than in-equilibrium (an example of feedback in equilibrium is global warming, where there’s lots of feedback effects but they’re more or less canceling each other out), but this seems justified.
I would agree that one can imagine some kind of feedback loop in “effective smartness” of humans, but I am not sure how natural it is to divorce this from the economic / technological revolution that has radically reshaped our planet, since so much of our effective smartness enhancement is also economy / technology. But this is ye olde reference class ping pong.
Thanks for your thoughts!
I don’t quite follow you on the intelligence explosion issue. For instance, why does a strong argument against the intelligence explosion hypothesis need to show that a feedback loop is unlikely? Couldn’t we believe that it is likely, but not likely to be very rapid for a while? For instance, there is probably a feedback loop in intelligence already, where humans with better thoughts and equipment are effectively smarter, and can then devise better thoughts and equipment. But this has been true for a while, and is a fairly slow process (at least for now, relative to our ability to deal with things).
Yeah, upon rereading that response, I think I created a few non sequiturs in revision. I’m not even 100% sure what I meant by some bits. I think the arguments that now seem confusing were me was saying that by putting an intelligence feedback loop in the reference class of “feedback loops in general” and then using that to forecast low impact, the thing that is doing most of the work is simply how low impact most stuff is.
A nuclear bomb (or a raindrop forming, or tipping back a little too far in your chair) can be modeled as a feedback loop through several orders of magnitude of power output, and then eventually that model breaks down and the explosion dissipates, and the world might be a little scarred and radioactive, but it is overall not much different. But if your AI increased by several orders of magnitude in intelligence (let’s just pretend that’s meaningful for a second), I would expect that to be a much bigger deal, just because the thing that’s increasing is different. That is, I was thinking that the implicit model used by the reference class argument from the original link seems to predict local advantages in AI, but predict *against* those local advantages being important to the world at large, which I think is putting the most weight on the weakest link.
Part of this picture I had comes from what I’m imagining as prototypical reference class members—note that I only imagined self-sustaining feedback, not “subcritical” feedback. In retrospect, this seems to be begging the question somewhat—subcritical feedback speeds up progress, but doesn’t necessarily concentrate it, unless there is some specific threshold effect for getting that feedback. Another feature of my prototypes was that they’re out-of-equilibrium rather than in-equilibrium (an example of feedback in equilibrium is global warming, where there’s lots of feedback effects but they’re more or less canceling each other out), but this seems justified.
I would agree that one can imagine some kind of feedback loop in “effective smartness” of humans, but I am not sure how natural it is to divorce this from the economic / technological revolution that has radically reshaped our planet, since so much of our effective smartness enhancement is also economy / technology. But this is ye olde reference class ping pong.