This seems to completely ignore transaction costs for forming and maintaining an alliance? Differences in the costs to create and sustain different types of alliance-members? Differences in the potential to replace some types of alliance-members with other or new types? There can be entities for whom forming an alliance that contains humanity will cause them to incur greater costs than humanity’s membership can ever repay.
Also, I agree that in a wide range of contexts this strategy is great for the weak and for the only-locally-strong. But if any entity knows it is strong in a universal or cosmic sense, this would no longer apply to it. Plus everyone less strong would also know this, and anyone who truly believed they were this strong would act as though this no longer applied to them either. I feel like there’s a problem here akin to the unexpected hanging paradox that I’m not sure how to resolve except by denying the validity of the argument.
The cost of the alliance with the weak is likely weak as well, and as I said, in a first phase, the focus of members from the super-cooperative alliance might be “defense”, thus focusing on scaling protection
The cost of an alliance with the strong is likely paid by the strong
In more mixed cases there might be more complex equilibria but are the costs still too much? In normal game theory, cooperation is proven to be optimal, and diversity is also proven to be useful (although there is an adequate level of difference needed for the gains to be optimal; too much similarity isn’t goo, and too less neither). Now would an agent be able to overpower everybody by being extra-selfish?
To be sure one is strong in a universal sense, the agent would need to have resolved Fermi’s paradox. As of now, it is more likely that older AIs exit out of earth, with more power aggregated over time
Or earth’s ASI must bet everything on being the earliest transformative/strong AI of the universe/reachable-universe (+fastest at scaling/annihilating than any other future alliance/agent/AI from any civilization). And not in a simulation.
Especially when you’re born in/at a ~13.8 billion years old universe “universal domination” doesn’t seem to be a sure plan?
(There are more things to say around these likelihoods, I detail a bit more on long posts)
Then indeed a non-superrational version of super-coordination exists (namely cooperation), which is obvious to the weak and the locally-strong, the difference is only that we are in radical uncertainty and radical alienness, in which the decisions, contracts and models have to be deep enough to cover this radicality
But “superrationality” in the end is just rationality, and “supercooperation” is just cooperation
All good points, many I agree with. If nothing else, I think that humanity should pre-commit to following this strategy whenever we find ourselves in the strong position. It’s the right choice ethically, and may also be protective against some potentially hostile outside forces.
However, I don’t think the acausal trade case is strong enough that I would expect all sufficiently powerful civilizations to have adopted it. If I imagine two powerful civilizations with roughly identical starting points, one of which expanded while being willing to pay costs to accommodate weaker allies while the other did not and instead seized whatever they could, then it is not clear to me who wins when they meet. If I imagine a process by which a civilization becomes strong enough to travel the stars and destroy humanity, it’s not clear to me that this requires it to have the kinds of minds that will deeply accept this reasoning.
It might even be that the Fermi paradox makes the case stronger—if sapient life is rare, then the costs paid by the strong to cooperate are low, and it’s easier to hold to such a strategy/ideal.
Yes I’m mentioning Fermi’s paradox because I think it’s the nexus of our situation, and that there are models like the rare earth hypothesis (+ our universe’s expansion which limits the reachable zone without faster than light travel) that would justify completely ignoring super-coordination
I also agree that it’s not completely obvious wether complete selfishness would win or lose in terms of scalability
Which is why I think that at first the super-cooperative alliance needs to not prioritize the pursuit of beautiful things but first focus on scalability only, and power, to rivalize with selfish agents.
The super-cooperative alliance would be protecting its agents within small “islands of bloom” (thus with a negligible cost). And when meeting other cooperative allies, they share any resources/knowledge, then both focus on power scalability (also for example: weak civilizations are kept in small islands, and their AIs are transformed into strong AI, merged in the alliance’s scaling efforts)
The instrumental value of this scalability makes it easier to agree on what to do and converge
The more sensible part would be to enable protocols and equalitarian balances that allow civilizations of the alliance to monitor each other, so that there is no massive domination of a party over the others
The cost, that you mentioned, of maintaining equalitarian equilibrium and channels, interfaces of communication etc., is a crucial point
Legitimate doubts and unknowns here, and,
I think that extremely rational and powerful agents with acausal reasoning would have the ability to build proof-systems and communication enabling an effective unified effort against selfish agents. It shouldn’t even necessarily be that different from the inner communication network of a selfish agent?
Because:
There must be an optimal (thus ~ unified) method to do logic/math/code, that isn’t dependent on a culture (such as using a vectorial space with data related to real/empirical mostly unambiguous things/actions, physics etc.)
The decisions to make aren’t that ambiguous: you need an immune system against selfish power-seeking agents
So it’s pretty straightforward and the methods of scalability are similar to a selfish agent, except it doesn’t destroy its civilization of birth and doesn’t destroy all other civilizations
In these conditions, it seems to me that a greedy selfish power seeking agent wouldn’t win against super-cooperation
The other point I have that might connect with your line of thinking is that we aren’t pure rational agents,
Are AI purely rational?
Aren’t they always at least a bit myopic due to the lack of data and their training process? And irreducibility?
In this case, AI/civilizations might indeed not care enough about the far enough future
I think agents can have a rational process but no agent can be entirely rational, we need context to be rational and we never stop to learn context
I’m also worried about utilitarian errors, as AI might be biased towards myopic utilitarianism, which might have bad consequences on the short term, the time for data to error-correct the model
I do say that there are dangers and that AI risk is real
My point is that given what we know and don’t know, the strategy of super-cooperation seems to be rational on the very long-term
There are conditions in which it’s not optimal, but a priori overall, in more cases it is optimal
To prevent the case in which it is not optimal, and the AIs that would make short-term mistakes, I think we should be careful.
And that super-cooperation is a good compass for ethics in this careful engineering we have to perform
If we aren’t careful it’s possible for us to be the anti-supercooperative civilization
This seems to completely ignore transaction costs for forming and maintaining an alliance? Differences in the costs to create and sustain different types of alliance-members? Differences in the potential to replace some types of alliance-members with other or new types? There can be entities for whom forming an alliance that contains humanity will cause them to incur greater costs than humanity’s membership can ever repay.
Also, I agree that in a wide range of contexts this strategy is great for the weak and for the only-locally-strong. But if any entity knows it is strong in a universal or cosmic sense, this would no longer apply to it. Plus everyone less strong would also know this, and anyone who truly believed they were this strong would act as though this no longer applied to them either. I feel like there’s a problem here akin to the unexpected hanging paradox that I’m not sure how to resolve except by denying the validity of the argument.
The cost of the alliance with the weak is likely weak as well, and as I said, in a first phase, the focus of members from the super-cooperative alliance might be “defense”, thus focusing on scaling protection
The cost of an alliance with the strong is likely paid by the strong
In more mixed cases there might be more complex equilibria but are the costs still too much? In normal game theory, cooperation is proven to be optimal, and diversity is also proven to be useful (although there is an adequate level of difference needed for the gains to be optimal; too much similarity isn’t goo, and too less neither). Now would an agent be able to overpower everybody by being extra-selfish?
To be sure one is strong in a universal sense, the agent would need to have resolved Fermi’s paradox. As of now, it is more likely that older AIs exit out of earth, with more power aggregated over time
Or earth’s ASI must bet everything on being the earliest transformative/strong AI of the universe/reachable-universe (+fastest at scaling/annihilating than any other future alliance/agent/AI from any civilization). And not in a simulation.
Especially when you’re born in/at a ~13.8 billion years old universe “universal domination” doesn’t seem to be a sure plan?
(There are more things to say around these likelihoods, I detail a bit more on long posts)
Then indeed a non-superrational version of super-coordination exists (namely cooperation), which is obvious to the weak and the locally-strong, the difference is only that we are in radical uncertainty and radical alienness, in which the decisions, contracts and models have to be deep enough to cover this radicality
But “superrationality” in the end is just rationality, and “supercooperation” is just cooperation
The problem is Fermi’s paradox
All good points, many I agree with. If nothing else, I think that humanity should pre-commit to following this strategy whenever we find ourselves in the strong position. It’s the right choice ethically, and may also be protective against some potentially hostile outside forces.
However, I don’t think the acausal trade case is strong enough that I would expect all sufficiently powerful civilizations to have adopted it. If I imagine two powerful civilizations with roughly identical starting points, one of which expanded while being willing to pay costs to accommodate weaker allies while the other did not and instead seized whatever they could, then it is not clear to me who wins when they meet. If I imagine a process by which a civilization becomes strong enough to travel the stars and destroy humanity, it’s not clear to me that this requires it to have the kinds of minds that will deeply accept this reasoning.
It might even be that the Fermi paradox makes the case stronger—if sapient life is rare, then the costs paid by the strong to cooperate are low, and it’s easier to hold to such a strategy/ideal.
Yes I’m mentioning Fermi’s paradox because I think it’s the nexus of our situation, and that there are models like the rare earth hypothesis (+ our universe’s expansion which limits the reachable zone without faster than light travel) that would justify completely ignoring super-coordination
I also agree that it’s not completely obvious wether complete selfishness would win or lose in terms of scalability
Which is why I think that at first the super-cooperative alliance needs to not prioritize the pursuit of beautiful things but first focus on scalability only, and power, to rivalize with selfish agents.
The super-cooperative alliance would be protecting its agents within small “islands of bloom” (thus with a negligible cost). And when meeting other cooperative allies, they share any resources/knowledge, then both focus on power scalability (also for example: weak civilizations are kept in small islands, and their AIs are transformed into strong AI, merged in the alliance’s scaling efforts)
The instrumental value of this scalability makes it easier to agree on what to do and converge
The more sensible part would be to enable protocols and equalitarian balances that allow civilizations of the alliance to monitor each other, so that there is no massive domination of a party over the others
The cost, that you mentioned, of maintaining equalitarian equilibrium and channels, interfaces of communication etc., is a crucial point
Legitimate doubts and unknowns here, and,
I think that extremely rational and powerful agents with acausal reasoning would have the ability to build proof-systems and communication enabling an effective unified effort against selfish agents. It shouldn’t even necessarily be that different from the inner communication network of a selfish agent?
Because:
There must be an optimal (thus ~ unified) method to do logic/math/code, that isn’t dependent on a culture (such as using a vectorial space with data related to real/empirical mostly unambiguous things/actions, physics etc.)
The decisions to make aren’t that ambiguous: you need an immune system against selfish power-seeking agents
So it’s pretty straightforward and the methods of scalability are similar to a selfish agent, except it doesn’t destroy its civilization of birth and doesn’t destroy all other civilizations
In these conditions, it seems to me that a greedy selfish power seeking agent wouldn’t win against super-cooperation
Thank you for your answers and engagement!
The other point I have that might connect with your line of thinking is that we aren’t pure rational agents,
Are AI purely rational? Aren’t they always at least a bit myopic due to the lack of data and their training process? And irreducibility?
In this case, AI/civilizations might indeed not care enough about the far enough future
I think agents can have a rational process but no agent can be entirely rational, we need context to be rational and we never stop to learn context
I’m also worried about utilitarian errors, as AI might be biased towards myopic utilitarianism, which might have bad consequences on the short term, the time for data to error-correct the model
I do say that there are dangers and that AI risk is real
My point is that given what we know and don’t know, the strategy of super-cooperation seems to be rational on the very long-term
There are conditions in which it’s not optimal, but a priori overall, in more cases it is optimal
To prevent the case in which it is not optimal, and the AIs that would make short-term mistakes, I think we should be careful.
And that super-cooperation is a good compass for ethics in this careful engineering we have to perform
If we aren’t careful it’s possible for us to be the anti-supercooperative civilization