Putting the entire failure to trade on the ability to communicate seems to understate the issue. Most if not all of the things listed that they ‘could’ do, are things which they could theoretically do with their physical capacities, but not with their cognitive abilities or ability to coordinate within themselves to accomplish a task.
In general, they aren’t able to act with the level of intentionality required to be helpful to us except in cases where those things we want are almost exactly the things they have evolved to do (like bees making honey, as mentioned in another comment).
The ‘failure to communicate’ is therefore in fact a failure to be able to think and act at the required level of flexibility and abstraction, and that seems more likely to carry over to our relations with some theoretical, super advanced AI or civilisation.
Maybe one useful thought experiment is whether we could train a dog-level intelligence to do most of these tasks if it had the actuators of an ant colony, given our good understanding of dog training (~= “communication”) and the fact that dogs still lack a bunch of key cognitive abilities humans have (so dog-human relations are somewhat analogous to human-AI relations).
(Also, ant colonies in aggregate do pretty complex things, so maybe they’re not that far off from dogs? But I’m mostly just thinking of Douglas Hofstadter’s “Aunt Hillary” here :)
My guess is that for a lot of Katja’s proposed trades, you’d only need the ants to have a moderate level of understanding, something like “dog level” or “pretty dumb AI system level”. (e.g. “do thing X in situations where you get inputs Y that were associated with thing-we-actually-care-about Z during the training session we gave you”.)
The ‘failure to communicate’ is therefore in fact a failure to be able to think and act at the required level of flexibility and abstraction, and that seems more likely to carry over to our relations with some theoretical, super advanced AI or civilisation.
Definitely true that you’re a more valuable trade partner if you’re smarter. But there are some particularly useful intelligence/comms thresholds that we meet and ants don’t—e.g. the “dog level”, plus some self-awareness stuff, plus not-awful world models in some domains.
Meta: the dog analogy ignores the distinction between training and trading. I’m eliding this here bc it’s hard to know what an ant colony’s “considered opinion” / “reflective endorsement” would mean, let alone an ant’s. but ofc this matters a lot for AGI-human interactions. Consider an AGi that keeps humans around on a “human preserve” out of sentiment, but only cares about certain features of humanity and genetically modifies others out of existence (analogous to training out certain behaviors or engaging in selective breeding), or tortures / brainwashes humans to get them to act the way it wants. (These failure modes of “having things an AI wants, and being able to give it those things, but not defend yourself” are also alluded to in other comments here, e.g. gwern and Elisabeth’s comments about “the noble wolf” and torture, respectively.)
Putting the entire failure to trade on the ability to communicate seems to understate the issue. Most if not all of the things listed that they ‘could’ do, are things which they could theoretically do with their physical capacities, but not with their cognitive abilities or ability to coordinate within themselves to accomplish a task.
In general, they aren’t able to act with the level of intentionality required to be helpful to us except in cases where those things we want are almost exactly the things they have evolved to do (like bees making honey, as mentioned in another comment).
The ‘failure to communicate’ is therefore in fact a failure to be able to think and act at the required level of flexibility and abstraction, and that seems more likely to carry over to our relations with some theoretical, super advanced AI or civilisation.
Maybe one useful thought experiment is whether we could train a dog-level intelligence to do most of these tasks if it had the actuators of an ant colony, given our good understanding of dog training (~= “communication”) and the fact that dogs still lack a bunch of key cognitive abilities humans have (so dog-human relations are somewhat analogous to human-AI relations).
(Also, ant colonies in aggregate do pretty complex things, so maybe they’re not that far off from dogs? But I’m mostly just thinking of Douglas Hofstadter’s “Aunt Hillary” here :)
My guess is that for a lot of Katja’s proposed trades, you’d only need the ants to have a moderate level of understanding, something like “dog level” or “pretty dumb AI system level”. (e.g. “do thing X in situations where you get inputs Y that were associated with thing-we-actually-care-about Z during the training session we gave you”.)
Definitely true that you’re a more valuable trade partner if you’re smarter. But there are some particularly useful intelligence/comms thresholds that we meet and ants don’t—e.g. the “dog level”, plus some self-awareness stuff, plus not-awful world models in some domains.
Meta: the dog analogy ignores the distinction between training and trading. I’m eliding this here bc it’s hard to know what an ant colony’s “considered opinion” / “reflective endorsement” would mean, let alone an ant’s. but ofc this matters a lot for AGI-human interactions. Consider an AGi that keeps humans around on a “human preserve” out of sentiment, but only cares about certain features of humanity and genetically modifies others out of existence (analogous to training out certain behaviors or engaging in selective breeding), or tortures / brainwashes humans to get them to act the way it wants. (These failure modes of “having things an AI wants, and being able to give it those things, but not defend yourself” are also alluded to in other comments here, e.g. gwern and Elisabeth’s comments about “the noble wolf” and torture, respectively.)