Isn’t it almost certain that super-optimizing AI will result in unintended consequences? I think it’s almost certain that super-optimizing AI will have to deal with their own unintended consequences. Isn’t the expectation of encountering intelligence so advanced, that it’s perfect and infallible essentially the expectation of encountering God?
Isn’t the expectation of encountering intelligence so advanced, that it’s perfect and infallible essentially the expectation of encountering God?
Which god? If by “God” you mean “something essentially perfect and infallible,” then yes. If by “God” you mean “that entity that killed a bunch of Egyptian kids” or “that entity that’s responsible for lightning” or “that guy that annoyed the Roman empire 2 millennia ago,” then no.
Also, essentially infallible to us isn’t necessarily essentially infallible to it (though I suspect that any attempt at AGI will have enough hacks and shortcuts that we can see faults too).
Which god? If by “God” you mean “something essentially perfect and infallible,” then yes.
That one. Big man in sky invented by shepherds does’t interest me much. Just because I’m a better optimizer of resources in certain contexts than an amoeba doesn’t make me perfect and infallible. Just because X is orders of magnitude a better optimizer than Y doesn’t make X perfect and infallible. Just because X can rapidly optimize itself doesn’t make it infallible either. Yet when people talk about the post-singularity super-optimizers, they seem to be talking about some sort of Sci-Fi God.
Y’know, I’m not really sure where that idea comes from. The optimization power of even a moderately transhuman AI would be quite incredible, but I’ve never seen a convincing argument that intelligence scales with optimization power (though the argument that optimization power scales with intelligence seems sound).
What’s unintended consequences? An imperfect ability to predict the future? Read strictly, any finite entity’s ability to predict the future is going to be imperfect.
Reason: Cockroaches and the behavior of humans. We can and do kill individuals and specific groups of individuals. We can’t kill all of them, however. If humans can get into space, the lightspeed barrier might let far-flung tribes of “human fundamentalists,” to borrow a term from Charles Stross, to survive, though individuals would often be killed and would never stand a chance in a direct conflict with a super AI.
In itself that doesn’t seem to be relevant evidence. “There exist species that humans cannot eradicate without major coordinated effort”. It doesn’t follow that either the same would hold for far more powerful AIs, nor that we should model AI-human relationship on humans-cockroaches rather than humans-kittens or humans-smallpox.
If humans can get into space, the lightspeed barrier might let far-flung tribes of “human fundamentalists,” to borrow a term from Charles Stross, to survive
It’s easy to imagine specific scenarios, especially when generalizing from fictional evidence. In fact we don’t have evidence sufficient to even raise any scenario as concrete as yours to the level of awareness.
I could as easily reply that AI that wanted to kill fleeing humans could do so by powerful enough directed lasers, which will overtake any STL ship. But this is a contrived scenario. There really is no reason to discuss it specifically. (For one thing, there’s still no evidence human space colonization or even solar system colonization will happen anytime soon. And unlike AI it’s not going to happen suddenly, without lots of advanced notice.)
It’s easy to imagine specific scenarios, especially when generalizing from fictional evidence. In fact we don’t have evidence sufficient to even raise any scenario as concrete as yours to the level of awareness.
…
I could as easily reply that AI that wanted to kill fleeing humans could do so by powerful enough directed lasers, which will overtake any STL ship. But this is a contrived scenario. There really is no reason to discuss it specifically.
A summary of your points is that: while conceivable, there’s no reason to think it’s at all likely. Ok. How about, “Because it’s fun to think about?”
Actually, lasers might not be practical against maneuverable targets because of the diffraction limit and the lightspeed limit. In order to focus a laser at very great distances, one would need very large lenses. (Perhaps planet sized, depending on distance and frequency.) Targets could respond by moving out of the beam, and the lightspeed limit would preclude immediate retargeting. Compensating for this by making the beam wider would be very expensive.
Regarding lasers: I could list things the attackers might do to succeed. But I don’t want to discuss it because we’d be speculating on practically zero evidence. I’ll merely say that I would rather that my hopes for the future do not depend on a failure of imagination on part of an enemy superintelligent AI.
You’re assuming that there’s always an answer for the more intelligent actor. Only happens that way in the movies. Sometimes you get the bear, and sometimes the bear gets you.
Sometimes one can pin their hopes on the laws of physics in the face of a more intelligent foe.
Isn’t it almost certain that super-optimizing AI will result in unintended consequences? I think it’s almost certain that super-optimizing AI will have to deal with their own unintended consequences. Isn’t the expectation of encountering intelligence so advanced, that it’s perfect and infallible essentially the expectation of encountering God?
Which god? If by “God” you mean “something essentially perfect and infallible,” then yes. If by “God” you mean “that entity that killed a bunch of Egyptian kids” or “that entity that’s responsible for lightning” or “that guy that annoyed the Roman empire 2 millennia ago,” then no.
Also, essentially infallible to us isn’t necessarily essentially infallible to it (though I suspect that any attempt at AGI will have enough hacks and shortcuts that we can see faults too).
That one. Big man in sky invented by shepherds does’t interest me much. Just because I’m a better optimizer of resources in certain contexts than an amoeba doesn’t make me perfect and infallible. Just because X is orders of magnitude a better optimizer than Y doesn’t make X perfect and infallible. Just because X can rapidly optimize itself doesn’t make it infallible either. Yet when people talk about the post-singularity super-optimizers, they seem to be talking about some sort of Sci-Fi God.
Y’know, I’m not really sure where that idea comes from. The optimization power of even a moderately transhuman AI would be quite incredible, but I’ve never seen a convincing argument that intelligence scales with optimization power (though the argument that optimization power scales with intelligence seems sound).
“optimization power” is more-or-less equivalent to “intelligence”, in local parlance. Do you have a different definition of intelligence in mind?
One that doesn’t classify evolution as intelligent.
So the nonapples theory of intelligence, then?
More generally, a theory that requires modeling of the future for something to be intelligent.
What’s unintended consequences? An imperfect ability to predict the future? Read strictly, any finite entity’s ability to predict the future is going to be imperfect.
What if the AI are advanced over us as we are over cockroaches, and the superintelligent AI find us just as annoying, disgusting, and hard to kill?
What reason is there to expect such a thing?
(Not to mention that, proverbs notwithstanding, humans can and do kill cockroaches easily; I wouldn’t want the tables to be reversed.)
Reason: Cockroaches and the behavior of humans. We can and do kill individuals and specific groups of individuals. We can’t kill all of them, however. If humans can get into space, the lightspeed barrier might let far-flung tribes of “human fundamentalists,” to borrow a term from Charles Stross, to survive, though individuals would often be killed and would never stand a chance in a direct conflict with a super AI.
In itself that doesn’t seem to be relevant evidence. “There exist species that humans cannot eradicate without major coordinated effort”. It doesn’t follow that either the same would hold for far more powerful AIs, nor that we should model AI-human relationship on humans-cockroaches rather than humans-kittens or humans-smallpox.
It’s easy to imagine specific scenarios, especially when generalizing from fictional evidence. In fact we don’t have evidence sufficient to even raise any scenario as concrete as yours to the level of awareness.
I could as easily reply that AI that wanted to kill fleeing humans could do so by powerful enough directed lasers, which will overtake any STL ship. But this is a contrived scenario. There really is no reason to discuss it specifically. (For one thing, there’s still no evidence human space colonization or even solar system colonization will happen anytime soon. And unlike AI it’s not going to happen suddenly, without lots of advanced notice.)
A summary of your points is that: while conceivable, there’s no reason to think it’s at all likely. Ok. How about, “Because it’s fun to think about?”
Actually, lasers might not be practical against maneuverable targets because of the diffraction limit and the lightspeed limit. In order to focus a laser at very great distances, one would need very large lenses. (Perhaps planet sized, depending on distance and frequency.) Targets could respond by moving out of the beam, and the lightspeed limit would preclude immediate retargeting. Compensating for this by making the beam wider would be very expensive.
Regarding lasers: I could list things the attackers might do to succeed. But I don’t want to discuss it because we’d be speculating on practically zero evidence. I’ll merely say that I would rather that my hopes for the future do not depend on a failure of imagination on part of an enemy superintelligent AI.
You’re assuming that there’s always an answer for the more intelligent actor. Only happens that way in the movies. Sometimes you get the bear, and sometimes the bear gets you.
Sometimes one can pin their hopes on the laws of physics in the face of a more intelligent foe.
It’s more fun to me to think about pleasant extremely improbable futures than unpleasant ones. To each their own.
There’s lots of scope for great adventure stories in dystopian futures.