By this token, everyone is neutral, no one is friendly, unless I am literally their top priority in the whole world, and they mine (that doesn’t sound like simple “friendship”… more like some kind of eternal fated bond between soulmates).
Can you explain your reasoning here? How does a bias towards or against imply a ‘top priority’?
Well, you’re the one who’s saying “the aliens wouldn’t be friendly if they know you’re biased towards your own side”. A bias means I prioritize my race over the aliens. This is normal and pretty expected; the aliens, too, will surely prioritize their own race over humans, if push came to shove. That’s no barrier to friendship. The ability to cooperate is fundamentally dependent on circumstances. The only case in which I will be absolutely sure that someone would never, ever turn on me, no matter how dire the circumstances, is if I am their top priority. Bias means you have a hierarchy of values, and some are higher than others; so “well-being of your family” is higher than “well-being of an equivalent number of total strangers”, and “well-being of humanity” may be higher than “well-being of the sentient octopuses of Rigel-4”. But the world usually isn’t made of binary trolley problems, and agents that are willing to be friendly and to put each other at a reasonably high (but not necessarily top) position in their value hierarchies have plenty of occasions to establish fruitful collaboration by throwing some other, less important value under the bus.
A relentless maximizer however is a fundamentally selfish kind of agent. A maximizer can never truly compromise because it does not have a range of acceptable states—it has only ONE all-important value that defines a single acceptable target state, and all its actions are in service of achieving that state. It can perform friendship only as long as it serves its goal, and will backstab you the next moment even if it was not in existential danger, merely because it has to advance towards its goal. I may care for the well-being of my wife, but I am not a Wife-Well-Being Maximizer. I would not for example stab someone to steal a pair of earrings that she would like if only I could get away with it; I still value a stranger’s life far more than my wife’s marginal enjoyment from a new piece of jewellery. A maximizer instead only cares about the goal, and everything else is at best instrumental, which makes it fundamentally unreliable (unless YOUR well-being happens to be the goal it cares about maximizing, and even then, I’d consider it a risky agent to have around).
Can you explain your reasoning here? How does a bias towards or against imply a ‘top priority’?
Well, you’re the one who’s saying “the aliens wouldn’t be friendly if they know you’re biased towards your own side”. A bias means I prioritize my race over the aliens. This is normal and pretty expected; the aliens, too, will surely prioritize their own race over humans, if push came to shove. That’s no barrier to friendship. The ability to cooperate is fundamentally dependent on circumstances. The only case in which I will be absolutely sure that someone would never, ever turn on me, no matter how dire the circumstances, is if I am their top priority. Bias means you have a hierarchy of values, and some are higher than others; so “well-being of your family” is higher than “well-being of an equivalent number of total strangers”, and “well-being of humanity” may be higher than “well-being of the sentient octopuses of Rigel-4”. But the world usually isn’t made of binary trolley problems, and agents that are willing to be friendly and to put each other at a reasonably high (but not necessarily top) position in their value hierarchies have plenty of occasions to establish fruitful collaboration by throwing some other, less important value under the bus.
A relentless maximizer however is a fundamentally selfish kind of agent. A maximizer can never truly compromise because it does not have a range of acceptable states—it has only ONE all-important value that defines a single acceptable target state, and all its actions are in service of achieving that state. It can perform friendship only as long as it serves its goal, and will backstab you the next moment even if it was not in existential danger, merely because it has to advance towards its goal. I may care for the well-being of my wife, but I am not a Wife-Well-Being Maximizer. I would not for example stab someone to steal a pair of earrings that she would like if only I could get away with it; I still value a stranger’s life far more than my wife’s marginal enjoyment from a new piece of jewellery. A maximizer instead only cares about the goal, and everything else is at best instrumental, which makes it fundamentally unreliable (unless YOUR well-being happens to be the goal it cares about maximizing, and even then, I’d consider it a risky agent to have around).