Well, the crux of the issue is that the random AIs may be more likely to leave us alone than near-misses at FAI.
Random AIs will very nearly all kill us. That is, the overwhelming majority of random AIs do stuff. Doing stuff takes resources. We are resources. We are the resources that are like… right near where it was created and are most necessary to bootstrap it’s way to the rest of the universe.
For the majority of AIs we are terminally irrelevant but our termination is slightly instrumentally useful.
You’re making a giant number of implicit ill-founded assumptions here that must all be true. Read my section on the AI space in general.
Firstly, you assume that the ‘stuff’ is unbounded. Needs not be true. I for one thing want to figure out how universe works, out of pure curiosity. That may likely be a very bounded goal right here. I also like to watch nature, or things like mandelbox fractal, which is unbounded but also preserves the nature. Those are valid examples of goals. The AI crowd, when warned not to anthropomorphize, switches to animalomorphization, or worse yet, bacteriomorphization where the AI is just a smarter gray goo, doing the goo thing intelligently. No. The human goal system can be the lower bound on the complexity of the goal system of super human AI. edit: and on top of that, all the lower biological imperatives like desire to reproduce sexually, we tend to satisfy in very unintended ways, from porn to birth control. If i were an upload i would get rid of much of those distracting nonsense goals.
Secondly, you assume that achieving of ‘stuff’ is raw resource-bound, rather than e.g. structuring the resources—bound. So that we’ll be worth less than the atoms we are made of. Which needs never happen.
In this you have a sub-assumption that the AI can only do stuff the gray goo way, and won’t ever discover anything cleverer (like quantum computing, which grows much more rapidly with size) which it would e.g. want to keep crammed together because of light speed lag. The “ai is going to eat us all” is just another of those priveledged baseless guesses what an entity way smarter than you would do. The near-FAI is the only thing with which we are pretty sure it won’t leave us alone.
Unless its utility function has a maximum, we are at risk. Observing Mandelbrot fractals is probably enhanced by having all the atoms of a galaxy playing the role of pixels.
Would you agree that unless the utility function of a random AI has a (rather low) maximum, and barring the discovery of infinite matter/energy sources, its immediate neighbourhood is likely to get repurposed?
I must say that at least I finally understand why you think botched FAIs are more risky than others.
But consider, as Ben Goertzel mentioned, that nobody is trying to build a random AI. Whatever achieves AGI-level is likely to have a built-in representation for humans and to have a tendency to interact with them. Check to see if I actually understood you correctly: does the previous sentence make it more probable that any future AGI is likely to be destructive?
Unless its utility function has a maximum, we are at risk. Observing Mandelbrot fractals is probably enhanced by having all the atoms of a galaxy playing the role of pixels.
Cruel physics, cruel physics. There is speed of light delay, that’s thing, and I’m not maniacal about mandelbox (its a 3d fractal) anyway, I won’t want to wipe out interesting stuff in the galaxy for minor gain in the resolution. And if i can circumvent speed of light, all bets are off WRT what kind of resources i would need (or if i would need any, maybe i get infinite computing power in finite space and time)
But consider, as Ben Goertzel mentioned, that nobody is trying to build a random AI.
How’s about generating human brain (in crude emulation of developmental biology)? It’s pretty darn random.
My argument is that, the AI whose only goal is helping humans, if bugged, has the only goal that is messing with humans. The AI that just represents humans in a special way is not this scary, albeit still is, to some extent.
Consider this seed AI: evolution. Comes up with mankind, that tries to talk with outside (god) without even knowing that outside exists, has endangered species list. Of course, if we are sufficiently resource bound, we are going to eat up all other forms of life, but we’d be resource bound because we are too stupid to find a way to go to space, and we clearly would rather not exerminate all other lifeforms.
This example ought to entirely invalidate this notion that ‘almost all’ AIs in AI design space are going to eat you. We have 1 example: evolution going FOOM via evolving human brain, and it cares about wildlife somewhat, yes we do immense damage to environment, but we would not if we could avoid it , even at some expense. If you have 1 example probe into random AI space, and it’s not all this bad, you seriously should not go around telling how you’re extremely sure it is just blind luck et cetera.
Add some anthropics… humans are indeed a FOOMing intelligence relative to the evolutionary timescale, but it’s no use declaring that “we’ve got one example of a random intelligence, and look, its humans::goal_system is remarkably similar to our own goal_system, therefore the next random try will also be similar”...
I’m also pretty sure that evolution would hate us if it had such a concept: instead of our intended design goal of “go and multiply”, we came up with stupid goals that make no sense, like love, happyness, etc.
So what? The AI can come up with Foo, Bar, Baz that we never thought it would.
The point is that we got entirely unexpected goal system (starting from evolution as a seed optimizer), with which we got greenpeace seriously risking their lives trying to sink japanese whaling ship, complete with international treaties against whaling. It is okay the AI won’t have love, happyness, etc. but why exactly should i be so extremely sure the foo, bar, and baz won’t make it assign some nonzero utility to mankind? Why we assume the AI will have the goal system of a bacteria?
Why should i be so sure as to approve of stepping into a clearly marked, obvious minefield of “AIs that want to mess with mankind”?
edit: To clarify, here we have AI’s weird random goal systems being reduced to, approximately, a real number: how much it values other complex dynamical systems vs less complex stuff. We value complex systems, and don’t like to disrupt them, even if we don’t understand anything. And most amazingly, the original process (evolution) looks like a good example of, if anything, an unfriendly AI attempt that wouldn’t give a slightest damn. We still do disrupt complex systems, when the resources are a serious enough bottleneck, but we’re making progress at not doing it and trading off some of the efficiency to avoid breaking things.
Not disrupting complex systems doesn’t seem to be an universal human value to me (just as Greenpeace is not our universal value system, either). But you’re right, it’s probably not a good approach to treat an AI as just another grey goo.
The problem is that it will be still us who will create that AI, so it will end up having values related to us. It would be a deliberate effort at our part to try to build something that isn’t a member of the FAI-like sphere you wrote about (in which I agree with pangel’s comment). For example, by ordering it to leave us alone and try to build stuff out of Jupiter instead. But then… what’s the point? If this AI was to prevent any further AI development on Earth… that would be a nice case of “ugly just-not-friendly-enough AI messing with humanity”, but if it wasn’t, then we could still end up converting the planet to paperclips by another AI developed later.
We have international treaties to this sense. The greenpeace just assigns it particularly high value, comparing to the rest who assign much smaller value. Still, if we had fewer resource and R&D limitations we would be able to preserve animals much better, as the value of animals as animals would stay the same while the cost of alternative ways of acquiring the resources would be lower.
With regards to the effort to build something that’s not a member of the FAI-like sphere, that’s where the majority of real effort to build the AI lies today. Look at the real projects that use techniques which have known practical spinoffs (neural networks), and have the computing power. Blue brain. The FAI effort is a microscopic, neglected fraction of AI effort.
Also, the prevention of paperclippers doesn’t strike me as particularly bad scenario. The smarter AI doesn’t need to use clumsy bureaucracy style mechanisms of forbidding all AI development.
You’re making a giant number of implicit ill-founded assumptions here that must all be true
I don’t accept that I make or are required to make any of the assumptions that you declare that I make. Allow me to emphasize just how slight a convenience it has to be for an indifferent entity to exterminate humanity. Very, very slight.
I’ll bow out of this conversation. It isn’t worth having it in a hidden draft.
What ever. That is the problem with human language, simplest statements have a zillion possible unfounded assumptions that are not even well defined nor is the maker of statement even aware of them (or would admit making them, because he didn’t, because he just manipulated symbols).
Take “i think therefore i am”. innocent phrase, something that entirely boxed in blind symbolic ai should be able to think, right? No. Wrong. The “I” is only a meaningful symbol when there’s non-i to separate from i, the “think” when you can do something other than thinking, that you need to separate from thought, via symbol ‘think’; therefore implies the statements where it does not follow, and I am refers to the notion that non-i might exist without I existing. Yet if you say something like this, are you ‘making’ those assumptions? You can say no—they come in pre-made, and aren’t being processed.
Random AIs will very nearly all kill us. That is, the overwhelming majority of random AIs do stuff. Doing stuff takes resources. We are resources. We are the resources that are like… right near where it was created and are most necessary to bootstrap it’s way to the rest of the universe.
For the majority of AIs we are terminally irrelevant but our termination is slightly instrumentally useful.
You’re making a giant number of implicit ill-founded assumptions here that must all be true. Read my section on the AI space in general.
Firstly, you assume that the ‘stuff’ is unbounded. Needs not be true. I for one thing want to figure out how universe works, out of pure curiosity. That may likely be a very bounded goal right here. I also like to watch nature, or things like mandelbox fractal, which is unbounded but also preserves the nature. Those are valid examples of goals. The AI crowd, when warned not to anthropomorphize, switches to animalomorphization, or worse yet, bacteriomorphization where the AI is just a smarter gray goo, doing the goo thing intelligently. No. The human goal system can be the lower bound on the complexity of the goal system of super human AI. edit: and on top of that, all the lower biological imperatives like desire to reproduce sexually, we tend to satisfy in very unintended ways, from porn to birth control. If i were an upload i would get rid of much of those distracting nonsense goals.
Secondly, you assume that achieving of ‘stuff’ is raw resource-bound, rather than e.g. structuring the resources—bound. So that we’ll be worth less than the atoms we are made of. Which needs never happen.
In this you have a sub-assumption that the AI can only do stuff the gray goo way, and won’t ever discover anything cleverer (like quantum computing, which grows much more rapidly with size) which it would e.g. want to keep crammed together because of light speed lag. The “ai is going to eat us all” is just another of those priveledged baseless guesses what an entity way smarter than you would do. The near-FAI is the only thing with which we are pretty sure it won’t leave us alone.
Unless its utility function has a maximum, we are at risk. Observing Mandelbrot fractals is probably enhanced by having all the atoms of a galaxy playing the role of pixels.
Would you agree that unless the utility function of a random AI has a (rather low) maximum, and barring the discovery of infinite matter/energy sources, its immediate neighbourhood is likely to get repurposed?
I must say that at least I finally understand why you think botched FAIs are more risky than others.
But consider, as Ben Goertzel mentioned, that nobody is trying to build a random AI. Whatever achieves AGI-level is likely to have a built-in representation for humans and to have a tendency to interact with them. Check to see if I actually understood you correctly: does the previous sentence make it more probable that any future AGI is likely to be destructive?
Cruel physics, cruel physics. There is speed of light delay, that’s thing, and I’m not maniacal about mandelbox (its a 3d fractal) anyway, I won’t want to wipe out interesting stuff in the galaxy for minor gain in the resolution. And if i can circumvent speed of light, all bets are off WRT what kind of resources i would need (or if i would need any, maybe i get infinite computing power in finite space and time)
How’s about generating human brain (in crude emulation of developmental biology)? It’s pretty darn random.
My argument is that, the AI whose only goal is helping humans, if bugged, has the only goal that is messing with humans. The AI that just represents humans in a special way is not this scary, albeit still is, to some extent.
Consider this seed AI: evolution. Comes up with mankind, that tries to talk with outside (god) without even knowing that outside exists, has endangered species list. Of course, if we are sufficiently resource bound, we are going to eat up all other forms of life, but we’d be resource bound because we are too stupid to find a way to go to space, and we clearly would rather not exerminate all other lifeforms.
This example ought to entirely invalidate this notion that ‘almost all’ AIs in AI design space are going to eat you. We have 1 example: evolution going FOOM via evolving human brain, and it cares about wildlife somewhat, yes we do immense damage to environment, but we would not if we could avoid it , even at some expense. If you have 1 example probe into random AI space, and it’s not all this bad, you seriously should not go around telling how you’re extremely sure it is just blind luck et cetera.
Add some anthropics… humans are indeed a FOOMing intelligence relative to the evolutionary timescale, but it’s no use declaring that “we’ve got one example of a random intelligence, and look, its humans::goal_system is remarkably similar to our own goal_system, therefore the next random try will also be similar”...
I’m also pretty sure that evolution would hate us if it had such a concept: instead of our intended design goal of “go and multiply”, we came up with stupid goals that make no sense, like love, happyness, etc.
So what? The AI can come up with Foo, Bar, Baz that we never thought it would.
The point is that we got entirely unexpected goal system (starting from evolution as a seed optimizer), with which we got greenpeace seriously risking their lives trying to sink japanese whaling ship, complete with international treaties against whaling. It is okay the AI won’t have love, happyness, etc. but why exactly should i be so extremely sure the foo, bar, and baz won’t make it assign some nonzero utility to mankind? Why we assume the AI will have the goal system of a bacteria?
Why should i be so sure as to approve of stepping into a clearly marked, obvious minefield of “AIs that want to mess with mankind”?
edit: To clarify, here we have AI’s weird random goal systems being reduced to, approximately, a real number: how much it values other complex dynamical systems vs less complex stuff. We value complex systems, and don’t like to disrupt them, even if we don’t understand anything. And most amazingly, the original process (evolution) looks like a good example of, if anything, an unfriendly AI attempt that wouldn’t give a slightest damn. We still do disrupt complex systems, when the resources are a serious enough bottleneck, but we’re making progress at not doing it and trading off some of the efficiency to avoid breaking things.
Not disrupting complex systems doesn’t seem to be an universal human value to me (just as Greenpeace is not our universal value system, either). But you’re right, it’s probably not a good approach to treat an AI as just another grey goo.
The problem is that it will be still us who will create that AI, so it will end up having values related to us. It would be a deliberate effort at our part to try to build something that isn’t a member of the FAI-like sphere you wrote about (in which I agree with pangel’s comment). For example, by ordering it to leave us alone and try to build stuff out of Jupiter instead. But then… what’s the point? If this AI was to prevent any further AI development on Earth… that would be a nice case of “ugly just-not-friendly-enough AI messing with humanity”, but if it wasn’t, then we could still end up converting the planet to paperclips by another AI developed later.
We have international treaties to this sense. The greenpeace just assigns it particularly high value, comparing to the rest who assign much smaller value. Still, if we had fewer resource and R&D limitations we would be able to preserve animals much better, as the value of animals as animals would stay the same while the cost of alternative ways of acquiring the resources would be lower.
With regards to the effort to build something that’s not a member of the FAI-like sphere, that’s where the majority of real effort to build the AI lies today. Look at the real projects that use techniques which have known practical spinoffs (neural networks), and have the computing power. Blue brain. The FAI effort is a microscopic, neglected fraction of AI effort.
Also, the prevention of paperclippers doesn’t strike me as particularly bad scenario. The smarter AI doesn’t need to use clumsy bureaucracy style mechanisms of forbidding all AI development.
I don’t accept that I make or are required to make any of the assumptions that you declare that I make. Allow me to emphasize just how slight a convenience it has to be for an indifferent entity to exterminate humanity. Very, very slight.
I’ll bow out of this conversation. It isn’t worth having it in a hidden draft.
What ever. That is the problem with human language, simplest statements have a zillion possible unfounded assumptions that are not even well defined nor is the maker of statement even aware of them (or would admit making them, because he didn’t, because he just manipulated symbols).
Take “i think therefore i am”. innocent phrase, something that entirely boxed in blind symbolic ai should be able to think, right? No. Wrong. The “I” is only a meaningful symbol when there’s non-i to separate from i, the “think” when you can do something other than thinking, that you need to separate from thought, via symbol ‘think’; therefore implies the statements where it does not follow, and I am refers to the notion that non-i might exist without I existing. Yet if you say something like this, are you ‘making’ those assumptions? You can say no—they come in pre-made, and aren’t being processed.