For example if the FAI scenario is much less likely (a priori) than a Clippy scenario, then there’s no reason for Clippy to make strong concessions.
But if a “paperclips” maximizer, as opposed to “tables”, “cars”, or “alien sex toys” maximizer, is just one of many unfriendly maximizers, then maximizing “human values” is just one of many unlikely outcomes. In other words, you can’t just say that unfriendly AIs are more likely than friendly AIs when it comes to cooperation. Since the opposition between a paperclip maximizer and an “alien sex toy” maximizer is the same as the opposition between the former and an alien or human friendly AI. Since all of them want to maximize their opposing values. And even if there turns out to be a subset of values shared by some AIs, other groups could cooperate to outweigh their leverage.
But since there is an exponentially huge set of random maximisers, the probability of each individual one is infinitesimal. OTOH, human values have a high probability density in mindspace because people are actually working towards it.
human values have a high probability density in mindspace because people are actually working towards it
Depends on how high probability density have humans (and alien life forms so similar to humans that they share our values) in mindspace. Maybe very low. Maybe a society ruled by intelligent ants according to their values would make us very unhappy… and on a cosmic scale, ants are our cousins; alien life should be much more different.
I don’t understand what point you’re trying to make. My point was that cooperative game theory doesn’t magically guarantee a UFAI will treat us nicely. It might work but only if there is a sufficiently substantial Everett branch with a FAI. The probability of that branch probably strongly depends on effort invested into FAI research.
If the probability of FAI (and friendly uploads etc) is near zero, then we’re doomed either way. But even though I believe the probability of provably friendly AI coming first is <50% , its definitely not 10^-11!
But if a “paperclips” maximizer, as opposed to “tables”, “cars”, or “alien sex toys” maximizer, is just one of many unfriendly maximizers, then maximizing “human values” is just one of many unlikely outcomes. In other words, you can’t just say that unfriendly AIs are more likely than friendly AIs when it comes to cooperation. Since the opposition between a paperclip maximizer and an “alien sex toy” maximizer is the same as the opposition between the former and an alien or human friendly AI. Since all of them want to maximize their opposing values. And even if there turns out to be a subset of values shared by some AIs, other groups could cooperate to outweigh their leverage.
But since there is an exponentially huge set of random maximisers, the probability of each individual one is infinitesimal. OTOH, human values have a high probability density in mindspace because people are actually working towards it.
Depends on how high probability density have humans (and alien life forms so similar to humans that they share our values) in mindspace. Maybe very low. Maybe a society ruled by intelligent ants according to their values would make us very unhappy… and on a cosmic scale, ants are our cousins; alien life should be much more different.
I don’t understand what point you’re trying to make. My point was that cooperative game theory doesn’t magically guarantee a UFAI will treat us nicely. It might work but only if there is a sufficiently substantial Everett branch with a FAI. The probability of that branch probably strongly depends on effort invested into FAI research.
If the probability of FAI (and friendly uploads etc) is near zero, then we’re doomed either way. But even though I believe the probability of provably friendly AI coming first is <50% , its definitely not 10^-11!
Fair enough, but there’s still an enormous incentive to work on FAI.
Of course, I was not trying to suggest otherwise.