Unless its utility function has a maximum, we are at risk. Observing Mandelbrot fractals is probably enhanced by having all the atoms of a galaxy playing the role of pixels.
Cruel physics, cruel physics. There is speed of light delay, that’s thing, and I’m not maniacal about mandelbox (its a 3d fractal) anyway, I won’t want to wipe out interesting stuff in the galaxy for minor gain in the resolution. And if i can circumvent speed of light, all bets are off WRT what kind of resources i would need (or if i would need any, maybe i get infinite computing power in finite space and time)
But consider, as Ben Goertzel mentioned, that nobody is trying to build a random AI.
How’s about generating human brain (in crude emulation of developmental biology)? It’s pretty darn random.
My argument is that, the AI whose only goal is helping humans, if bugged, has the only goal that is messing with humans. The AI that just represents humans in a special way is not this scary, albeit still is, to some extent.
Consider this seed AI: evolution. Comes up with mankind, that tries to talk with outside (god) without even knowing that outside exists, has endangered species list. Of course, if we are sufficiently resource bound, we are going to eat up all other forms of life, but we’d be resource bound because we are too stupid to find a way to go to space, and we clearly would rather not exerminate all other lifeforms.
This example ought to entirely invalidate this notion that ‘almost all’ AIs in AI design space are going to eat you. We have 1 example: evolution going FOOM via evolving human brain, and it cares about wildlife somewhat, yes we do immense damage to environment, but we would not if we could avoid it , even at some expense. If you have 1 example probe into random AI space, and it’s not all this bad, you seriously should not go around telling how you’re extremely sure it is just blind luck et cetera.
Add some anthropics… humans are indeed a FOOMing intelligence relative to the evolutionary timescale, but it’s no use declaring that “we’ve got one example of a random intelligence, and look, its humans::goal_system is remarkably similar to our own goal_system, therefore the next random try will also be similar”...
I’m also pretty sure that evolution would hate us if it had such a concept: instead of our intended design goal of “go and multiply”, we came up with stupid goals that make no sense, like love, happyness, etc.
So what? The AI can come up with Foo, Bar, Baz that we never thought it would.
The point is that we got entirely unexpected goal system (starting from evolution as a seed optimizer), with which we got greenpeace seriously risking their lives trying to sink japanese whaling ship, complete with international treaties against whaling. It is okay the AI won’t have love, happyness, etc. but why exactly should i be so extremely sure the foo, bar, and baz won’t make it assign some nonzero utility to mankind? Why we assume the AI will have the goal system of a bacteria?
Why should i be so sure as to approve of stepping into a clearly marked, obvious minefield of “AIs that want to mess with mankind”?
edit: To clarify, here we have AI’s weird random goal systems being reduced to, approximately, a real number: how much it values other complex dynamical systems vs less complex stuff. We value complex systems, and don’t like to disrupt them, even if we don’t understand anything. And most amazingly, the original process (evolution) looks like a good example of, if anything, an unfriendly AI attempt that wouldn’t give a slightest damn. We still do disrupt complex systems, when the resources are a serious enough bottleneck, but we’re making progress at not doing it and trading off some of the efficiency to avoid breaking things.
Not disrupting complex systems doesn’t seem to be an universal human value to me (just as Greenpeace is not our universal value system, either). But you’re right, it’s probably not a good approach to treat an AI as just another grey goo.
The problem is that it will be still us who will create that AI, so it will end up having values related to us. It would be a deliberate effort at our part to try to build something that isn’t a member of the FAI-like sphere you wrote about (in which I agree with pangel’s comment). For example, by ordering it to leave us alone and try to build stuff out of Jupiter instead. But then… what’s the point? If this AI was to prevent any further AI development on Earth… that would be a nice case of “ugly just-not-friendly-enough AI messing with humanity”, but if it wasn’t, then we could still end up converting the planet to paperclips by another AI developed later.
We have international treaties to this sense. The greenpeace just assigns it particularly high value, comparing to the rest who assign much smaller value. Still, if we had fewer resource and R&D limitations we would be able to preserve animals much better, as the value of animals as animals would stay the same while the cost of alternative ways of acquiring the resources would be lower.
With regards to the effort to build something that’s not a member of the FAI-like sphere, that’s where the majority of real effort to build the AI lies today. Look at the real projects that use techniques which have known practical spinoffs (neural networks), and have the computing power. Blue brain. The FAI effort is a microscopic, neglected fraction of AI effort.
Also, the prevention of paperclippers doesn’t strike me as particularly bad scenario. The smarter AI doesn’t need to use clumsy bureaucracy style mechanisms of forbidding all AI development.
Cruel physics, cruel physics. There is speed of light delay, that’s thing, and I’m not maniacal about mandelbox (its a 3d fractal) anyway, I won’t want to wipe out interesting stuff in the galaxy for minor gain in the resolution. And if i can circumvent speed of light, all bets are off WRT what kind of resources i would need (or if i would need any, maybe i get infinite computing power in finite space and time)
How’s about generating human brain (in crude emulation of developmental biology)? It’s pretty darn random.
My argument is that, the AI whose only goal is helping humans, if bugged, has the only goal that is messing with humans. The AI that just represents humans in a special way is not this scary, albeit still is, to some extent.
Consider this seed AI: evolution. Comes up with mankind, that tries to talk with outside (god) without even knowing that outside exists, has endangered species list. Of course, if we are sufficiently resource bound, we are going to eat up all other forms of life, but we’d be resource bound because we are too stupid to find a way to go to space, and we clearly would rather not exerminate all other lifeforms.
This example ought to entirely invalidate this notion that ‘almost all’ AIs in AI design space are going to eat you. We have 1 example: evolution going FOOM via evolving human brain, and it cares about wildlife somewhat, yes we do immense damage to environment, but we would not if we could avoid it , even at some expense. If you have 1 example probe into random AI space, and it’s not all this bad, you seriously should not go around telling how you’re extremely sure it is just blind luck et cetera.
Add some anthropics… humans are indeed a FOOMing intelligence relative to the evolutionary timescale, but it’s no use declaring that “we’ve got one example of a random intelligence, and look, its humans::goal_system is remarkably similar to our own goal_system, therefore the next random try will also be similar”...
I’m also pretty sure that evolution would hate us if it had such a concept: instead of our intended design goal of “go and multiply”, we came up with stupid goals that make no sense, like love, happyness, etc.
So what? The AI can come up with Foo, Bar, Baz that we never thought it would.
The point is that we got entirely unexpected goal system (starting from evolution as a seed optimizer), with which we got greenpeace seriously risking their lives trying to sink japanese whaling ship, complete with international treaties against whaling. It is okay the AI won’t have love, happyness, etc. but why exactly should i be so extremely sure the foo, bar, and baz won’t make it assign some nonzero utility to mankind? Why we assume the AI will have the goal system of a bacteria?
Why should i be so sure as to approve of stepping into a clearly marked, obvious minefield of “AIs that want to mess with mankind”?
edit: To clarify, here we have AI’s weird random goal systems being reduced to, approximately, a real number: how much it values other complex dynamical systems vs less complex stuff. We value complex systems, and don’t like to disrupt them, even if we don’t understand anything. And most amazingly, the original process (evolution) looks like a good example of, if anything, an unfriendly AI attempt that wouldn’t give a slightest damn. We still do disrupt complex systems, when the resources are a serious enough bottleneck, but we’re making progress at not doing it and trading off some of the efficiency to avoid breaking things.
Not disrupting complex systems doesn’t seem to be an universal human value to me (just as Greenpeace is not our universal value system, either). But you’re right, it’s probably not a good approach to treat an AI as just another grey goo.
The problem is that it will be still us who will create that AI, so it will end up having values related to us. It would be a deliberate effort at our part to try to build something that isn’t a member of the FAI-like sphere you wrote about (in which I agree with pangel’s comment). For example, by ordering it to leave us alone and try to build stuff out of Jupiter instead. But then… what’s the point? If this AI was to prevent any further AI development on Earth… that would be a nice case of “ugly just-not-friendly-enough AI messing with humanity”, but if it wasn’t, then we could still end up converting the planet to paperclips by another AI developed later.
We have international treaties to this sense. The greenpeace just assigns it particularly high value, comparing to the rest who assign much smaller value. Still, if we had fewer resource and R&D limitations we would be able to preserve animals much better, as the value of animals as animals would stay the same while the cost of alternative ways of acquiring the resources would be lower.
With regards to the effort to build something that’s not a member of the FAI-like sphere, that’s where the majority of real effort to build the AI lies today. Look at the real projects that use techniques which have known practical spinoffs (neural networks), and have the computing power. Blue brain. The FAI effort is a microscopic, neglected fraction of AI effort.
Also, the prevention of paperclippers doesn’t strike me as particularly bad scenario. The smarter AI doesn’t need to use clumsy bureaucracy style mechanisms of forbidding all AI development.