Your statement sounds a bit overgeneralized—but you probably have a point.
Still, would you indulge me in some idle speculation? Maybe there could be a species of aliens that evolved to intelligence by developing special microbe-infested organs (which would be firewalled somehow from the rest of the alien themselves) and incentivizing the microbial colonies somehow to solve problems for the host.
Maybe we humans evolved to intelligence that way—after all, we do have a lot of bacteria in our guts. But then, all the evidence that we have pointing to brains as information-processing center would have to be wrong. Maybe brains are the firewall organ! Memes are sortof like microbes, and they’re pretty well “firewalled” (genetic engineering is a meme-complex that might break out of the jail).
The notion of creating an ecology of entities, and incentivizing them to produce things that we value, might be a reasonable strategy, one that we humans have been using for some time.
I can’t see how this comment relates to the previous one. It seems to start an entirely new conversation. Also, the metaphor with brains and microbes doesn’t add understanding for me, I can only address the last paragraph, on its own.
The notion of creating an ecology of entities, and incentivizing them to produce things that we value, might be a reasonable strategy, one that we humans have been using for some time.
The crucial property of AIs making them a danger is (eventual) autonomy, not even rapid coming to power. Once the AI, or a society (“ecology”) of AIs, becomes sufficiently powerful to ignore vanilla humans, its values can’t be significantly influenced, and most of the future is going to be determined by those values. If those values are not good, from human values point of view, the future is lost to us, it has no goodness. The trick is to make sure that the values of such an autonomous entity are a very good match with our own, at some point where we still have a say in what they are.
Talk of “ecologies” of different agents creates an illusion of continuous control. The standard intuitive picture has little humans at the lower end with a network of gradually more powerful and/or different agents stretching out from them. But how much is really controlled by that node? Its power has no way of “amplifying” as you go through the network: if only humans and a few other agents share human values, these values will receive very little payoff. This is also not sustainable: over time, one should expect preference of agents with more power to gain in influence (which is what “more power” means).
The best way to win this race is to not create different-valued competitors that you don’t expect being able to turn into your own almost-copies, which seems infeasible for all the scenarios I know of. FAI is exactly about devising such a copycat, and if you can show how to do that with “ecologies”, all power to you, but I don’t expect anything from this line of thought.
To explain the relation, you said: “I can’t imagine having any return [...from this idea...] even in a perfect world, AI can still produce value, e.g. earn money online.”
I was trying to suggest that in fact there might be a path to Friendliness by installing sufficient safeguards that the primary way a software entity could replicate or spread would be by providing value to humans.
In the comment above, I explained why what AI does is irrelevant, as long as it’s not guaranteed to actually have the right values: once it goes unchecked, it just reverts to whatever it actually prefers, be it in a flurry of hard takeoff or after a thousand years of close collaboration. “Safeguards”, in every context I saw, refer to things that don’t enforce values, only behavior, and that’s not enough. Even the ideas for enforcement of behavior look infeasible, but the more important point is that even if we win this one, we still lose eventually with such an approach.
My symbiotic-ecology-of-software-tools scenario was not a serious proposal as the best strategy to Friendliness. I was trying to increase the plausibility of SOME return at SOME cost, even given that AIs could produce value.
I’m afraid I see the issue as clear-cut, you can’t get “some” return, you can only win or lose (probability of getting there is of course more amenable to small nudges).
Making such a statement significantly increases the standard of reasoning I expect from a post. That is, I expect you to be either right or at least a step ahead of the one with whom you are communicating.
Your statement sounds a bit overgeneralized—but you probably have a point.
Still, would you indulge me in some idle speculation? Maybe there could be a species of aliens that evolved to intelligence by developing special microbe-infested organs (which would be firewalled somehow from the rest of the alien themselves) and incentivizing the microbial colonies somehow to solve problems for the host.
Maybe we humans evolved to intelligence that way—after all, we do have a lot of bacteria in our guts. But then, all the evidence that we have pointing to brains as information-processing center would have to be wrong. Maybe brains are the firewall organ! Memes are sortof like microbes, and they’re pretty well “firewalled” (genetic engineering is a meme-complex that might break out of the jail).
The notion of creating an ecology of entities, and incentivizing them to produce things that we value, might be a reasonable strategy, one that we humans have been using for some time.
I can’t see how this comment relates to the previous one. It seems to start an entirely new conversation. Also, the metaphor with brains and microbes doesn’t add understanding for me, I can only address the last paragraph, on its own.
The crucial property of AIs making them a danger is (eventual) autonomy, not even rapid coming to power. Once the AI, or a society (“ecology”) of AIs, becomes sufficiently powerful to ignore vanilla humans, its values can’t be significantly influenced, and most of the future is going to be determined by those values. If those values are not good, from human values point of view, the future is lost to us, it has no goodness. The trick is to make sure that the values of such an autonomous entity are a very good match with our own, at some point where we still have a say in what they are.
Talk of “ecologies” of different agents creates an illusion of continuous control. The standard intuitive picture has little humans at the lower end with a network of gradually more powerful and/or different agents stretching out from them. But how much is really controlled by that node? Its power has no way of “amplifying” as you go through the network: if only humans and a few other agents share human values, these values will receive very little payoff. This is also not sustainable: over time, one should expect preference of agents with more power to gain in influence (which is what “more power” means).
The best way to win this race is to not create different-valued competitors that you don’t expect being able to turn into your own almost-copies, which seems infeasible for all the scenarios I know of. FAI is exactly about devising such a copycat, and if you can show how to do that with “ecologies”, all power to you, but I don’t expect anything from this line of thought.
To explain the relation, you said: “I can’t imagine having any return [...from this idea...] even in a perfect world, AI can still produce value, e.g. earn money online.”
I was trying to suggest that in fact there might be a path to Friendliness by installing sufficient safeguards that the primary way a software entity could replicate or spread would be by providing value to humans.
In the comment above, I explained why what AI does is irrelevant, as long as it’s not guaranteed to actually have the right values: once it goes unchecked, it just reverts to whatever it actually prefers, be it in a flurry of hard takeoff or after a thousand years of close collaboration. “Safeguards”, in every context I saw, refer to things that don’t enforce values, only behavior, and that’s not enough. Even the ideas for enforcement of behavior look infeasible, but the more important point is that even if we win this one, we still lose eventually with such an approach.
My symbiotic-ecology-of-software-tools scenario was not a serious proposal as the best strategy to Friendliness. I was trying to increase the plausibility of SOME return at SOME cost, even given that AIs could produce value.
I seem to have stepped onto a cached thought.
I’m afraid I see the issue as clear-cut, you can’t get “some” return, you can only win or lose (probability of getting there is of course more amenable to small nudges).
Making such a statement significantly increases the standard of reasoning I expect from a post. That is, I expect you to be either right or at least a step ahead of the one with whom you are communicating.