This post is interesting but it does not mention markets, and I argue that markets are intelligent. Not merely a team, or a company, or a civilization. I’m specifically talking about a collection of entities who transact with each other and behave in their own self interest. A market is an organizational technology that allows greater intelligence than could otherwise be achieved.
I agree that markets are themselves mildly intelligent, but I think you are underestimating how weak they are compared to what can exist. eg, imagine a market between 15 housecats (say, imagine they all live nearby behind walls, and they can trade things between each other by dropping them in a chute, if the other cat puts in enough payment treats, or something). anyway, cats each have about 10 trillion synapses in 760 million neurons, vs a single human with about 150 trillion synapses in 100 billion neurons. the market is an enormously limiting information bottleneck compared to a higher-bandwidth communication system, and the adversarial nature makes it hard to transmit information usefully. There’s no way the cats could implement market mechanisms well enough to make use of their neurons to, eg, learn abstract math efficiently. The human can. sure, the smartest cat will end up with the most stuff, and the market will have thereby allowed the most intelligent cat to direct the system. If some cats are better at some things, gains from trade will occur and they can each specialize somehow. Maybe one hunts their enclosure and the other maps the area or something, I’m not really imagining this in much detail. But even then, competitive low bandwidth communication just doesn’t compare favorably to an integrated brain.
AI would be able to beat us because of being a big integrated thing that communicates far better than we can. In the short term it manipulates us like tiktok and youtube (or more accurately, the market—an intelligent, misaligned system—rewards humans who implement AI to manipulate other humans, which is, you know, why tiktok and youtube are big sites), then in the long term it has no need of us because the robots can make more robots and the ai is better at designing AI, not currently true but visibly possible now.
Markets don’t need to ‘transmit information’ at all to the observer, in order to be both useful and intelligent. For example, if aliens, who’ve never seen a single human or single word of human language, came by and inspected the Earth, left, then came back decades later, the changes physically observable to their sensors and associated derivations, would be.
Maybe not in the sense of an active biological intelligence but certainly in the sense such as ‘The pyramids of Giza demonstrates their builder’s intelligence’.
but in order to implement a market, information (trade offers and trades) need to be transmitted. that’s what I was referring to as an information bottleneck.
Sure, but that doesn’t matter to the alien observers, or anyone if they had the equivalent expertise and sensors. They can still gleam a super-intelligent equivalent amount of knowledge.
“or anyone if they had the equivalent expertise and sensors” includes humans on Earth...
e.g. a lone hunter-gather living in a cave for a long period, coming out to survey the world with the latest tools and then going back into their cave and lifestyle with only the results.
https://slatestarcodex.com/2015/12/27/things-that-are-not-superintelligences/
This post is interesting but it does not mention markets, and I argue that markets are intelligent. Not merely a team, or a company, or a civilization. I’m specifically talking about a collection of entities who transact with each other and behave in their own self interest. A market is an organizational technology that allows greater intelligence than could otherwise be achieved.
Please read Nick Bostrom’s “Superintelligence”, it would really help you understand where everyone here has in mind when they talk about AI takeover.
I agree that markets are themselves mildly intelligent, but I think you are underestimating how weak they are compared to what can exist. eg, imagine a market between 15 housecats (say, imagine they all live nearby behind walls, and they can trade things between each other by dropping them in a chute, if the other cat puts in enough payment treats, or something). anyway, cats each have about 10 trillion synapses in 760 million neurons, vs a single human with about 150 trillion synapses in 100 billion neurons. the market is an enormously limiting information bottleneck compared to a higher-bandwidth communication system, and the adversarial nature makes it hard to transmit information usefully. There’s no way the cats could implement market mechanisms well enough to make use of their neurons to, eg, learn abstract math efficiently. The human can. sure, the smartest cat will end up with the most stuff, and the market will have thereby allowed the most intelligent cat to direct the system. If some cats are better at some things, gains from trade will occur and they can each specialize somehow. Maybe one hunts their enclosure and the other maps the area or something, I’m not really imagining this in much detail. But even then, competitive low bandwidth communication just doesn’t compare favorably to an integrated brain.
AI would be able to beat us because of being a big integrated thing that communicates far better than we can. In the short term it manipulates us like tiktok and youtube (or more accurately, the market—an intelligent, misaligned system—rewards humans who implement AI to manipulate other humans, which is, you know, why tiktok and youtube are big sites), then in the long term it has no need of us because the robots can make more robots and the ai is better at designing AI, not currently true but visibly possible now.
Markets don’t need to ‘transmit information’ at all to the observer, in order to be both useful and intelligent. For example, if aliens, who’ve never seen a single human or single word of human language, came by and inspected the Earth, left, then came back decades later, the changes physically observable to their sensors and associated derivations, would be.
Maybe not in the sense of an active biological intelligence but certainly in the sense such as ‘The pyramids of Giza demonstrates their builder’s intelligence’.
but in order to implement a market, information (trade offers and trades) need to be transmitted. that’s what I was referring to as an information bottleneck.
Sure, but that doesn’t matter to the alien observers, or anyone if they had the equivalent expertise and sensors. They can still gleam a super-intelligent equivalent amount of knowledge.
i’m not sure what you’re responding to. I don’t think I made a claim to which the behavior of aliens would be relevant.
“or anyone if they had the equivalent expertise and sensors” includes humans on Earth...
e.g. a lone hunter-gather living in a cave for a long period, coming out to survey the world with the latest tools and then going back into their cave and lifestyle with only the results.