This post is a reply to Eugene Kirpichov’s post on Linkedin. Eugene writes that contributing to general AI and information processing capabilities (including GPUs, general LLM technology, general data processing, etc.) is probably harmful overall because these capabilities effectively increase the speed at which the civilisation is moving but doesn’t affect the trends, and the trends are negative right now because the civilisation is not on a sustainable trajectory, that is, the civilisation doens’t move towards increasing flourishing of all moral patients, human and non-human.
I disagree with Kirpichov’s proposition as is because I think it lacks important nuance in definition of general AI and knowledge production capabilities. I think there are different kinds of general AI capabilities, some, I agree, are probably on net harmful, but others I think are not. I will explain the difference below.
Note that this post is not about general vs. specialised AI, i.e., narrow AI applications. Kirpichov and I agree that specialised AI applications should be judged on the case-by-case basis. Different applications may lie on the range from clearly beneficial, such as AI for community deliberation (see the Collective Intelligence Project), AI for human health (e.g., Slingshot AI, HealthcareAgents), AI for ocean ecosystems modelling and monitoring (e.g., Wildflow AI), or AI for decoding non-human communication (e.g., Earth Species Project), to clearly harmful applications, such as AI for spam, phishing, etc., with a thousand shades of benefit and harm in between.
Yet another dimension is the degree of generality of AI capabilities, from very broadly general, such as GPGPU computing capabilities, to very specialised AI capabilities, applicable only in a single narrow domain. The chances of spilling over some capabilities developed for beneficial applications to other, potentially harmful applications should be estimated. The standard example here is AI for drone navigation and autonomy, if developed for low-footprint delivery of goods, may proliferate into harmful applications like drone warfare and killer drones.
Now, to the question of distinguishing between better or worse general AI capabilities.
AI technologies help to create and test more models faster. I use the word “models” in the broadest sense here, including engineering designs, methods, organisational designs, social technologies, psycho-technologies, legal designs (laws, legal structures for organisations, contracts), industrial standards, etc., along with more standard epistemic theories a.k.a. explanations, as well as theories of ethics.
Models that are not refuted by test or practice become knowledge.
I agree with Kirpichov that currently, the civilisation is not on a sustainable trajectory. The civilisation as a whole lacks a lot of knowledge about how a sustainable civilisational design even looks and how to get there from the current state. As David Deutsch wrote, “all evils are caused by insufficient knowledge”.
AI capabilities could be used to obtain and leverage “good” knowledge: that is, the knowledge that nudges the civilisation towards a sustainable path. However, just as well, AI capabilities could be used to obtain and leverage “bad” knowledge: that is, the knowledge that exploits the flaws in the current design of the civilisation to gain advantage at the high collateral cost for other moral patients.
Therefore, we can infer that AI capabilities that differentially help or incentivise obtaining and leveraging more “good” than “bad” knowledge are probably on net beneficial.
However, how to distinguish between “good” and “bad” knowledge? Can the usage of some knowledge today be “good” today but “bad” tomorrow, or vice versa? How would we know?
I think we can work on this question “backwards”.
“Good” knowledge implies interconnection
When the civilisation is in a sustainable state, all agents whose actions matter for the well-being of any moral patients have to predominantly use “good” knowledge.
By definition of a sustainable civilisation given above, this knowledge should be interconnected with the (subjective/objective) knowledge about all moral patients’ states of flourishing[1], sourced from them directly when possible, e.g. when their own verbal report is available, or if unavailable, “to the best of our knowledge”, that is, using our best methods for inferring flourishing states of mute moral patients.
The phrase knowledge interconnection used above refers to how the models underlying that knowledge have been inferred and tested. “Interconnected inference” means obtaining the joint posterior of the models, and “interconnected testing” means verifying that the two models hold up in practice in interactive scenarios rather than only in isolation. Inference and testing should also not be isolated from each other but rather create a learning loop.
Making the knowledge of every agent (not only humans and AIs but also organisations and states) interconnected with the knowledge about the flourishing states of all moral patients individually would be infeasibly expensive. Imagine that every business or AI agent had to consider the outcomes of all their decisions for every human and animal. Therefore, by necessity, the structures for integrating the knowledge about the states of moral patients and outcomes for them have to be hierarchical[2].
Rafael Kaufmann and I have called these structures that ought to exist to align the civilisation with flourishing of moral patients the Gaia Network. The details of the technical architecture that we proposed for “connected model inference”, federated learning via credit assignment, and trustworthy “connected model testing” (verification/validation) are not important for this post. These details are also still up for a debate. Also, it’s almost certain that if our civilisation will reach a sustainable trajectory, multiple alternative approaches to knowledge connection would coexist.
The big point is that the “good” knowledge embodied in a sustainable civilisation must be interconnected, pretty much by definition of a sustainable civilisation.
Note that the reverse may not be true: a fully interconnected knowledge may not be used towards the flourishing of all patients. As an example, imagine a tight world-wide surveillance regime that doesn’t value the flourishing of the people and animals.
However, I don’t think it means that we should suppress knowledge connection until we figure out how to ensure that knowledge is used benevolently. In opposite, it seems to me that gradual interconnection of knowledge embodied by the economic agents is one of the very few operationalisable strategies for increasing world agents’ circles of concern and care[3].
AI and data processing capabilities that foster knowledge interconnection
From the interim conclusions above, we can posit that the kinds of AI and information processing capabilities that have a propensity to be used for obtaining and using interconnected knowledge are probably on net beneficial.
As is inherent to the discussions of differential technology development (see the recent Michael Nielsen’s notes on the topic), it’s often frustratingly hard even to put a sign on the propensity of this or that capability for “knowledge interconnection”. I cannot do this for general technologies like LLMs, general capabilities like causal reasoning and planning.
For example, LLMs can be deployed (and, in fact, essential) both for “knowledge connection”, such when they are used for semantic knowledge graph mining, and “disconnected” use cases, such as for automating myriads of tasks in the present “disconnected” business ecosystems.
For another example, capabilities for LLM distillation and miniaturisation may on the one hand foster the creation of knowledge graphs and well-structured public data because “they don’t know much by themselves” and therefore have to rely on externalised knowledge, but on the other hand small LLMs make (online) automation much cheaper that accelerates the present harmful trends.
Planning is essential for grounded federated learning and credit assignment, but obviously also the core capability for AI automation that leverages existing “disconnected” knowledge for profit extraction.
Nevertheless, I think I can make at least a few claims relatively confidently.
Capabilities that seem to favor knowledge interconnection:
Federated learning, privacy-preserving multi-party computation, and privacy-preserving machine learning. See Flower AI, OpenMined.org.
Protocols and file formats for data, belief, or claim exchange and validation, such as various blockchain and crypto projects, Solid, ActivityPub, XTDB, or GraphAr.
Semantic knowledge mining and hybrid reasoning on (federated) knowledge graphs and multimodal data, including tabular data. See OpenCog Hyperon.
ML interpretability, including so-called mechanistic interpretability. The usual “theory of impact” of interpretability is ensuring that we know when huge LLMs act or suggest us something for benevolent or nefarious reasons and corrupting LLM’s knowledge of dangerous topics via representation engineering. However, in the context of this post, representation interpretability is an essential piece of the research agenda of cognitive science of flourishing, i.e., understanding the (subjective) states of moral patients, potentially AIs themselves(!), but also animals or humans when the representations are obtained by passing through DNNs video, audio, or other metrics about them, brain-computer interface signals, etc.
Cross-language (such as, English/French) retrieval, search, and semantic knowledge integration. This is especially important for low-online-presence languages. See Cohere for AI.
On the other hand, many stock AI capabilities, such as imitation learning, many (though not all) methods of reinforcement learning, image and video generation[4], online (browser-based) task automation, collaborative filtering, and other capabilities seems to have very little to do with knowledge interconnection. Although it’s probably unfair to also say that most of these AI capabilities favor obtaining and leveraging “isolated” knowledge, either, if we take as the default premise that these capabilities will be mostly used to accelerate the current trends (of leveraging “disconnected” knowledge in the environment where most agents’ circle of concern is rather narrow) rather than change them, we should expect the development of these capabilities to be harmful on net.
Knowledge interconnection and the risk of industrial dehumanisation
This challenges the view that the development AI capabilities that favor knowledge interconnection will be on net beneficial if pursued right now. It corresponds to the possible position that I already mentioned above, namely that any intelligence amplification, even of the “interconnected” flavor, only exacerbates risks until we ensure that the knowledge is deployed stably and benevolently towards humans and other moral patients.
Currently, I conclude the development AI capabilities that favor knowledge interconnection on net reduces the risk of industrial dehumanisation. This is because the economy aligned with the flourishing of humans, animals, and natural ecosystems will be much complex and interconnected (hence, lower entropy[5]) than the pure “machine economy”, at least “locally”/temporarily. This means that AI capabilities favoring knowledge interconnection also differentially favor the development and sustainment of complex “human economy” industries, institutions, and social phenomena, such as healthcare, agriculture, education and enlightenment, family and romance, deliberative or liquid democracy, philosophy and ethics, religion, communities, culture, animal and ecosystem preferences, etc.
In fact, earlier I wrote comments that are very compatible with both this post and the recent Critch’s post calling for differential development of “human economy” industries. This post focuses on AI capabilities and technologies in specific and attempts to find “good” ones among them, but of course I fully support the development of human- and animal-centered AI applications as well.
Nevertheless, I take seriously the “industrial dehumanisation challenge” to the differential knowledge interconnection thesis that I lay out above in this post. I’m not at all sure about my own conclusions. I’m interested in your thoughtful opinions on this subject.
Create more a la carte tools (such as debuggers, observability, modellers, simulators, security analysers, verifiers, AI-first DevOps, CI/CD tools) to empower AI to create, maintain, and explain simple software more reliably and effectively.
Create more real-world-facing software than software-facing software.
Make it easier for system designers and developers to receive and account for the diverse feedback from the real world and the stakeholders.
Spend the software complexity “budget” on the essential complexity of accommodating diverse, interacting users’ and stakeholders’ needs rather than on the accidental complexity of “self-consumed” software.
The simplicity/acc manifesto calls for more focus on AI applications rather than the development of general AI capabilities, and specifically applications that “face the real world” rather than other software and software-derived “virtual” economy elements such as finance.
I guess this manifesto looks somewhat conflicted with the differential knowledge interconnection thesis. Although the last two items of the manifesto are very synergistic with knowledge interconnection for advancing the flourishing of moral patients, the second point about “creating a la carte tools for software engineering” looks perhaps slightly antagonistic to it. So, today I would probably remove it from the manifesto.
I don’t discuss here the disagreements about what subjective or objective states or “life journeys” of moral patients are really desirable between welfarism, utilitarianism, hedonism, eudaimonism, ecocentrism, and other relevant theories of ethics. Without loss of generality, we can assume that moral uncertainty and ethical portfolio views should be applied and the “goodness” of such and such state of such and such moral patient(s) weighted accordingly.
A potential counter-point here, per Andrew Critch’s “My theory of change for working in AI healthcare”, is that image and video generation are mostly used for human entertainment, which is a part of “human economy”, and fostering human economy is preferable to fostering “machine economy”.
Differential knowledge interconnection
Cross-posted from my blog.
This post is a reply to Eugene Kirpichov’s post on Linkedin. Eugene writes that contributing to general AI and information processing capabilities (including GPUs, general LLM technology, general data processing, etc.) is probably harmful overall because these capabilities effectively increase the speed at which the civilisation is moving but doesn’t affect the trends, and the trends are negative right now because the civilisation is not on a sustainable trajectory, that is, the civilisation doens’t move towards increasing flourishing of all moral patients, human and non-human.
I disagree with Kirpichov’s proposition as is because I think it lacks important nuance in definition of general AI and knowledge production capabilities. I think there are different kinds of general AI capabilities, some, I agree, are probably on net harmful, but others I think are not. I will explain the difference below.
Note that this post is not about general vs. specialised AI, i.e., narrow AI applications. Kirpichov and I agree that specialised AI applications should be judged on the case-by-case basis. Different applications may lie on the range from clearly beneficial, such as AI for community deliberation (see the Collective Intelligence Project), AI for human health (e.g., Slingshot AI, HealthcareAgents), AI for ocean ecosystems modelling and monitoring (e.g., Wildflow AI), or AI for decoding non-human communication (e.g., Earth Species Project), to clearly harmful applications, such as AI for spam, phishing, etc., with a thousand shades of benefit and harm in between.
Yet another dimension is the degree of generality of AI capabilities, from very broadly general, such as GPGPU computing capabilities, to very specialised AI capabilities, applicable only in a single narrow domain. The chances of spilling over some capabilities developed for beneficial applications to other, potentially harmful applications should be estimated. The standard example here is AI for drone navigation and autonomy, if developed for low-footprint delivery of goods, may proliferate into harmful applications like drone warfare and killer drones.
Now, to the question of distinguishing between better or worse general AI capabilities.
AI technologies help to create and test more models faster. I use the word “models” in the broadest sense here, including engineering designs, methods, organisational designs, social technologies, psycho-technologies, legal designs (laws, legal structures for organisations, contracts), industrial standards, etc., along with more standard epistemic theories a.k.a. explanations, as well as theories of ethics.
Models that are not refuted by test or practice become knowledge.
I agree with Kirpichov that currently, the civilisation is not on a sustainable trajectory. The civilisation as a whole lacks a lot of knowledge about how a sustainable civilisational design even looks and how to get there from the current state. As David Deutsch wrote, “all evils are caused by insufficient knowledge”.
AI capabilities could be used to obtain and leverage “good” knowledge: that is, the knowledge that nudges the civilisation towards a sustainable path. However, just as well, AI capabilities could be used to obtain and leverage “bad” knowledge: that is, the knowledge that exploits the flaws in the current design of the civilisation to gain advantage at the high collateral cost for other moral patients.
Therefore, we can infer that AI capabilities that differentially help or incentivise obtaining and leveraging more “good” than “bad” knowledge are probably on net beneficial.
However, how to distinguish between “good” and “bad” knowledge? Can the usage of some knowledge today be “good” today but “bad” tomorrow, or vice versa? How would we know?
I think we can work on this question “backwards”.
“Good” knowledge implies interconnection
When the civilisation is in a sustainable state, all agents whose actions matter for the well-being of any moral patients have to predominantly use “good” knowledge.
By definition of a sustainable civilisation given above, this knowledge should be interconnected with the (subjective/objective) knowledge about all moral patients’ states of flourishing[1], sourced from them directly when possible, e.g. when their own verbal report is available, or if unavailable, “to the best of our knowledge”, that is, using our best methods for inferring flourishing states of mute moral patients.
The phrase knowledge interconnection used above refers to how the models underlying that knowledge have been inferred and tested. “Interconnected inference” means obtaining the joint posterior of the models, and “interconnected testing” means verifying that the two models hold up in practice in interactive scenarios rather than only in isolation. Inference and testing should also not be isolated from each other but rather create a learning loop.
Making the knowledge of every agent (not only humans and AIs but also organisations and states) interconnected with the knowledge about the flourishing states of all moral patients individually would be infeasibly expensive. Imagine that every business or AI agent had to consider the outcomes of all their decisions for every human and animal. Therefore, by necessity, the structures for integrating the knowledge about the states of moral patients and outcomes for them have to be hierarchical[2].
Rafael Kaufmann and I have called these structures that ought to exist to align the civilisation with flourishing of moral patients the Gaia Network. The details of the technical architecture that we proposed for “connected model inference”, federated learning via credit assignment, and trustworthy “connected model testing” (verification/validation) are not important for this post. These details are also still up for a debate. Also, it’s almost certain that if our civilisation will reach a sustainable trajectory, multiple alternative approaches to knowledge connection would coexist.
The big point is that the “good” knowledge embodied in a sustainable civilisation must be interconnected, pretty much by definition of a sustainable civilisation.
Note that the reverse may not be true: a fully interconnected knowledge may not be used towards the flourishing of all patients. As an example, imagine a tight world-wide surveillance regime that doesn’t value the flourishing of the people and animals.
However, I don’t think it means that we should suppress knowledge connection until we figure out how to ensure that knowledge is used benevolently. In opposite, it seems to me that gradual interconnection of knowledge embodied by the economic agents is one of the very few operationalisable strategies for increasing world agents’ circles of concern and care[3].
AI and data processing capabilities that foster knowledge interconnection
From the interim conclusions above, we can posit that the kinds of AI and information processing capabilities that have a propensity to be used for obtaining and using interconnected knowledge are probably on net beneficial.
As is inherent to the discussions of differential technology development (see the recent Michael Nielsen’s notes on the topic), it’s often frustratingly hard even to put a sign on the propensity of this or that capability for “knowledge interconnection”. I cannot do this for general technologies like LLMs, general capabilities like causal reasoning and planning.
For example, LLMs can be deployed (and, in fact, essential) both for “knowledge connection”, such when they are used for semantic knowledge graph mining, and “disconnected” use cases, such as for automating myriads of tasks in the present “disconnected” business ecosystems.
For another example, capabilities for LLM distillation and miniaturisation may on the one hand foster the creation of knowledge graphs and well-structured public data because “they don’t know much by themselves” and therefore have to rely on externalised knowledge, but on the other hand small LLMs make (online) automation much cheaper that accelerates the present harmful trends.
Planning is essential for grounded federated learning and credit assignment, but obviously also the core capability for AI automation that leverages existing “disconnected” knowledge for profit extraction.
Nevertheless, I think I can make at least a few claims relatively confidently.
Capabilities that seem to favor knowledge interconnection:
Federated learning, privacy-preserving multi-party computation, and privacy-preserving machine learning. See Flower AI, OpenMined.org.
Federated inference and belief sharing. Examples: prediction markets like Manifold, Digital Gaia.
Protocols and file formats for data, belief, or claim exchange and validation, such as various blockchain and crypto projects, Solid, ActivityPub, XTDB, or GraphAr.
Semantic knowledge mining and hybrid reasoning on (federated) knowledge graphs and multimodal data, including tabular data. See OpenCog Hyperon.
Structured or semantic search such as exa.ai and system.com.
ML interpretability, including so-called mechanistic interpretability. The usual “theory of impact” of interpretability is ensuring that we know when huge LLMs act or suggest us something for benevolent or nefarious reasons and corrupting LLM’s knowledge of dangerous topics via representation engineering. However, in the context of this post, representation interpretability is an essential piece of the research agenda of cognitive science of flourishing, i.e., understanding the (subjective) states of moral patients, potentially AIs themselves(!), but also animals or humans when the representations are obtained by passing through DNNs video, audio, or other metrics about them, brain-computer interface signals, etc.
Datastore federation for retrieval-based LMs.
Cross-language (such as, English/French) retrieval, search, and semantic knowledge integration. This is especially important for low-online-presence languages. See Cohere for AI.
On the other hand, many stock AI capabilities, such as imitation learning, many (though not all) methods of reinforcement learning, image and video generation[4], online (browser-based) task automation, collaborative filtering, and other capabilities seems to have very little to do with knowledge interconnection. Although it’s probably unfair to also say that most of these AI capabilities favor obtaining and leveraging “isolated” knowledge, either, if we take as the default premise that these capabilities will be mostly used to accelerate the current trends (of leveraging “disconnected” knowledge in the environment where most agents’ circle of concern is rather narrow) rather than change them, we should expect the development of these capabilities to be harmful on net.
Knowledge interconnection and the risk of industrial dehumanisation
I agree with Andrew Critch that post-AGI industrial dehumanization is a major extinction risk for humanity.
This challenges the view that the development AI capabilities that favor knowledge interconnection will be on net beneficial if pursued right now. It corresponds to the possible position that I already mentioned above, namely that any intelligence amplification, even of the “interconnected” flavor, only exacerbates risks until we ensure that the knowledge is deployed stably and benevolently towards humans and other moral patients.
Currently, I conclude the development AI capabilities that favor knowledge interconnection on net reduces the risk of industrial dehumanisation. This is because the economy aligned with the flourishing of humans, animals, and natural ecosystems will be much complex and interconnected (hence, lower entropy[5]) than the pure “machine economy”, at least “locally”/temporarily. This means that AI capabilities favoring knowledge interconnection also differentially favor the development and sustainment of complex “human economy” industries, institutions, and social phenomena, such as healthcare, agriculture, education and enlightenment, family and romance, deliberative or liquid democracy, philosophy and ethics, religion, communities, culture, animal and ecosystem preferences, etc.
In fact, earlier I wrote comments that are very compatible with both this post and the recent Critch’s post calling for differential development of “human economy” industries. This post focuses on AI capabilities and technologies in specific and attempts to find “good” ones among them, but of course I fully support the development of human- and animal-centered AI applications as well.
Nevertheless, I take seriously the “industrial dehumanisation challenge” to the differential knowledge interconnection thesis that I lay out above in this post. I’m not at all sure about my own conclusions. I’m interested in your thoughtful opinions on this subject.
Relation to the simplicity/acc manifesto
A few months ago, I’ve drafted the simplicity/acc manifesto:
The simplicity/acc manifesto calls for more focus on AI applications rather than the development of general AI capabilities, and specifically applications that “face the real world” rather than other software and software-derived “virtual” economy elements such as finance.
I guess this manifesto looks somewhat conflicted with the differential knowledge interconnection thesis. Although the last two items of the manifesto are very synergistic with knowledge interconnection for advancing the flourishing of moral patients, the second point about “creating a la carte tools for software engineering” looks perhaps slightly antagonistic to it. So, today I would probably remove it from the manifesto.
I don’t discuss here the disagreements about what subjective or objective states or “life journeys” of moral patients are really desirable between welfarism, utilitarianism, hedonism, eudaimonism, ecocentrism, and other relevant theories of ethics. Without loss of generality, we can assume that moral uncertainty and ethical portfolio views should be applied and the “goodness” of such and such state of such and such moral patient(s) weighted accordingly.
Fields, Chris. “The free energy principle induces compartmentalization.” Biochemical and Biophysical Research Communications (2024): 150070.
Witkowski, Olaf, Thomas Doctor, Elizaveta Solomonova, Bill Duane, and Michael Levin. “Toward an ethics of autopoietic technology: Stress, care, and intelligence.” Biosystems 231 (2023): 104964.
A potential counter-point here, per Andrew Critch’s “My theory of change for working in AI healthcare”, is that image and video generation are mostly used for human entertainment, which is a part of “human economy”, and fostering human economy is preferable to fostering “machine economy”.
See Beren Millidge’s “BCIs and the ecosystem of modular minds”.