I don’t think the Benjamin Jesty case fully covers the space of ‘directly getting what you want’.
That seems like a case of ‘intelligent, specialized solution to a problem, which could then be solved with minimal resources.’
There’s a different class of power in the category of ‘directly getting what you want’ which looks less like a clever solution, and more like being in control of a system of physical objects.
This could be as simple as a child holding a lollipop, or a chipmunk with cheeks full of seeds.
Or it can be a more complicated system of physical control, with social implications. For instance, being armed with a powerful gun while facing an enraged charging grizzly bear. It doesn’t take a unique or clever or skillful action for the armed human to prevail in that case. And without a gun, there is little hope of the human prevailing.
Another case is a blacksmith with some iron ore and coal and a workshop. So long as nothing interferes, then it seems reasonable to expect that the blacksmith could solve a wide variety of different problems which needed a specific but simple iron tool. The blacksmith has some affordance in this case, and is more bottlenecked on supplies, energy, and time than on intellect or knowledge.
I can imagine a variety of future scenarios that don’t much look like the AI being in a dominance hierarchy with humanity, or look like it trading with humanity or other AIs.
For instance: if an AI were building a series of sea-floor factories, using material resources that it harvested itself. Some world governments might threaten the AI, tell it to stop using those physical resources. The AI might reply: “if you attack me, I will respond in a hostile manner. Just leave me alone.” If a government did attack the AI, the AI might release a deadly bioweapon which wiped out >99% of humanity in a month. That seems less like some kind of dominance conflict between human groups, and more like a human spraying poison on an inconvenient wasp nest.
Similarly, if an AI were industrializing and replicating in the asteroid belt, and some world governments told it to stop, but the AI just said No or said nothing at all. What would the governments do? Would they attack it? Send a rocket with a nuke? Fire a powerful laser? If the AI were sufficiently smart and competent, it would likely survive and counterattack with overwhelming force. For example, by directing a large asteroid at the Earth, and firing lasers at any rockets launched from Earth.
Or perhaps the AI would be secretly building subterranean factories, spreading through the crust of the Earth. We might not even notice until a whole continent started noticeably heating up from all the fission and fusion going on underground powering the AI factories.
If ants were smart enough to trade with, would I accept a deal from the local ants in order to allow them to have access to my kitchen trashcan? Maybe, but the price I would demand would be higher than I expect any ant colony (even a clever one) to be able to pay. If they invaded my kitchen anyway, I’d threaten them. If they continued, I’d destroy their colony (assuming that ants clever enough to bargain with weren’t a rare and precious occurrence). This would be even more the case, and even easier for me to do, if the ants moved and thought at 1/100th speed of a normal ant.
I don’t think it requires assuming that the AI is super-intelligent for any of these cases. Even current models know plenty of biology to develop a bioweapon capable of wiping out most of humanity, if they also had sufficient agency, planning, motivation, robotic control, and access to equipment / materials. Similarly, directing a big rock and some lasers at Earth doesn’t require superintelligence if you already have an industrial base in the asteroid belt.
Except for the child and the blacksmith, all of these seem like dominance conflicts to me. The blacksmith plausibly becomes a dominance conflict too once you consider how he ended up with the resources and what tasks he’s likely to face. You contrast these with conflicts between human groups, but I’d compare to e.g. a conflict between a drunk middle-aged loner who is looking for a brawl vs two young policemen and a bar owner.
I think we’re using different concepts of ‘dominance’ here. I usually think of ‘dominance’ as a social relationship between a strong party and a submissive party, a hierarchy. A relationship between a ruler and the ruled, or an abuser and abused. I don’t think that a human driving a bulldozer which destroys an anthill without the human even noticing that the anthill existed is the same sort of relationship. I think we need some word other than ‘dominant’ to describe the human wiping out the ants in an instant without sparing them a thought. It doesn’t particularly seem like a conflict even. The human in a bulldozer didn’t perceive themselves to be in a conflict, the ants weren’t powerful enough to register as an opponent or obstacle at all.
Thinking a bit more about this, I might group types of power into:
Power through relating: Social/economic/government/negotiating/threatening, reshaping the social world and the behavior of others
Power through understanding: having intellect and knowledge affordances, being able to solve clever puzzles in the world to achieve aims
Power through control: having physical affordances that allow for taking potent actions, reshaping the physical world
They all bleed together at the edges and are somewhat fungible in various ways, but I think it makes sense to talk of clusters despite their fuzzy edges.
Human psychology, mainly. “Dominance”-in-the-human-intuitive-sense was in the original post mainly because I think that’s how most humans intuitively understand “power”, despite (I claimed) not being particularly natural for more-powerful agents. So I’d expect humans to be confused insofar as they try to apply those dominance-in-the-human-intuitive-sense intuitions to more powerful agents.
And like, sure, one could use a notion of “dominance” which is general enough to encompass all forms of conflict, but at that point we can just talk about “conflict” and the like without the word “dominance”; using the word “dominance” for that is unnecessarily confusing, because most humans’ intuitive notion of “dominance” is narrower.
The post seems to me to be about notions of power, and the affordances of intelligent agents. I think this is a relevant kind of power to keep in mind.
I don’t think the Benjamin Jesty case fully covers the space of ‘directly getting what you want’.
That seems like a case of ‘intelligent, specialized solution to a problem, which could then be solved with minimal resources.’
There’s a different class of power in the category of ‘directly getting what you want’ which looks less like a clever solution, and more like being in control of a system of physical objects.
This could be as simple as a child holding a lollipop, or a chipmunk with cheeks full of seeds.
Or it can be a more complicated system of physical control, with social implications. For instance, being armed with a powerful gun while facing an enraged charging grizzly bear. It doesn’t take a unique or clever or skillful action for the armed human to prevail in that case. And without a gun, there is little hope of the human prevailing.
Another case is a blacksmith with some iron ore and coal and a workshop. So long as nothing interferes, then it seems reasonable to expect that the blacksmith could solve a wide variety of different problems which needed a specific but simple iron tool. The blacksmith has some affordance in this case, and is more bottlenecked on supplies, energy, and time than on intellect or knowledge.
I discuss this here: https://www.lesswrong.com/posts/uPi2YppTEnzKG3nXD/nathan-helm-burger-s-shortform?commentId=ZsDimsgkBNrfRps9r
I can imagine a variety of future scenarios that don’t much look like the AI being in a dominance hierarchy with humanity, or look like it trading with humanity or other AIs.
For instance: if an AI were building a series of sea-floor factories, using material resources that it harvested itself. Some world governments might threaten the AI, tell it to stop using those physical resources. The AI might reply: “if you attack me, I will respond in a hostile manner. Just leave me alone.” If a government did attack the AI, the AI might release a deadly bioweapon which wiped out >99% of humanity in a month. That seems less like some kind of dominance conflict between human groups, and more like a human spraying poison on an inconvenient wasp nest.
Similarly, if an AI were industrializing and replicating in the asteroid belt, and some world governments told it to stop, but the AI just said No or said nothing at all. What would the governments do? Would they attack it? Send a rocket with a nuke? Fire a powerful laser? If the AI were sufficiently smart and competent, it would likely survive and counterattack with overwhelming force. For example, by directing a large asteroid at the Earth, and firing lasers at any rockets launched from Earth.
Or perhaps the AI would be secretly building subterranean factories, spreading through the crust of the Earth. We might not even notice until a whole continent started noticeably heating up from all the fission and fusion going on underground powering the AI factories.
If ants were smart enough to trade with, would I accept a deal from the local ants in order to allow them to have access to my kitchen trashcan? Maybe, but the price I would demand would be higher than I expect any ant colony (even a clever one) to be able to pay. If they invaded my kitchen anyway, I’d threaten them. If they continued, I’d destroy their colony (assuming that ants clever enough to bargain with weren’t a rare and precious occurrence). This would be even more the case, and even easier for me to do, if the ants moved and thought at 1/100th speed of a normal ant.
I don’t think it requires assuming that the AI is super-intelligent for any of these cases. Even current models know plenty of biology to develop a bioweapon capable of wiping out most of humanity, if they also had sufficient agency, planning, motivation, robotic control, and access to equipment / materials. Similarly, directing a big rock and some lasers at Earth doesn’t require superintelligence if you already have an industrial base in the asteroid belt.
Except for the child and the blacksmith, all of these seem like dominance conflicts to me. The blacksmith plausibly becomes a dominance conflict too once you consider how he ended up with the resources and what tasks he’s likely to face. You contrast these with conflicts between human groups, but I’d compare to e.g. a conflict between a drunk middle-aged loner who is looking for a brawl vs two young policemen and a bar owner.
I think we’re using different concepts of ‘dominance’ here. I usually think of ‘dominance’ as a social relationship between a strong party and a submissive party, a hierarchy. A relationship between a ruler and the ruled, or an abuser and abused. I don’t think that a human driving a bulldozer which destroys an anthill without the human even noticing that the anthill existed is the same sort of relationship. I think we need some word other than ‘dominant’ to describe the human wiping out the ants in an instant without sparing them a thought. It doesn’t particularly seem like a conflict even. The human in a bulldozer didn’t perceive themselves to be in a conflict, the ants weren’t powerful enough to register as an opponent or obstacle at all.
What phenomenon are you modelling where this distinction is relevant?
Thinking a bit more about this, I might group types of power into:
Power through relating: Social/economic/government/negotiating/threatening, reshaping the social world and the behavior of others
Power through understanding: having intellect and knowledge affordances, being able to solve clever puzzles in the world to achieve aims
Power through control: having physical affordances that allow for taking potent actions, reshaping the physical world
They all bleed together at the edges and are somewhat fungible in various ways, but I think it makes sense to talk of clusters despite their fuzzy edges.
Human psychology, mainly. “Dominance”-in-the-human-intuitive-sense was in the original post mainly because I think that’s how most humans intuitively understand “power”, despite (I claimed) not being particularly natural for more-powerful agents. So I’d expect humans to be confused insofar as they try to apply those dominance-in-the-human-intuitive-sense intuitions to more powerful agents.
And like, sure, one could use a notion of “dominance” which is general enough to encompass all forms of conflict, but at that point we can just talk about “conflict” and the like without the word “dominance”; using the word “dominance” for that is unnecessarily confusing, because most humans’ intuitive notion of “dominance” is narrower.
Ah. I would say human psychology is too epiphenomenal so I’m mainly modelling things that shape (dis)equillibria in complex ecologies.
The post seems to me to be about notions of power, and the affordances of intelligent agents. I think this is a relevant kind of power to keep in mind.