This part 1 in a 3-part sequence summarizes my book, The Darwinian Trap (see part 2 here and part 3 here). The book aims to popularize the concept of multipolar traps and establish them as a broader cause area. If you find this series intriguing contact me at kristian@kristianronn.com if you have any input or ideas.
Global coordination stands as arguably the most critical challenge facing humanity today, functioning both as a necessary component for solving existential risks and as a significant barrier to effective mitigation. From nuclear proliferation to artificial intelligence development and climate change, our inability to collaborate effectively on a global scale not only exacerbates these threats but also perpetuates the emergence of new systemic vulnerabilities if left unaddressed.
In this sequence, I will argue that the root of this coordination problem lies in the very mechanisms that shaped our species: natural selection. This evolutionary process, operating as a trial-and-error optimization algorithm, prioritizes immediate survival and reproduction over long-term, global outcomes. As a result, our innate tendencies often favor short-term gains and localized benefits, even when they conflict with the greater good of our species and planet.
The inherent limitations of natural selection in predicting future optimal states have left us ill-equipped to handle global-scale challenges. In a world of finite resources, competition rather than cooperation has often been the more adaptive trait, leading to the emergence of self-interested behaviors that arguably dominate modern societies. This evolutionary legacy manifests in the form of nationalistic tendencies, economic rivalries, dangerous arms races and a general reluctance to sacrifice immediate benefits for long-term collective gains.
Part 1 (the part you are reading): This part explores how evolutionary selection pressures contribute to existential risks, particularly through global resource, power, and intelligence arms races.
Part 2: This part delves into the evolution of cooperation and discusses how survivorship bias might mislead us into thinking cooperation is easier to achieve than it actually is. It suggests that life may be inherently fragile, potentially containing the seeds of its own destruction, which could provide an explanation for the Fermi Paradox.
Part 3: The final part examines strategies to overcome global coordination challenges, both through centralized and decentralized approaches. It introduces the concept of “reputational markets” (a variant of prediction markets) as a tool for supply chain governance, aimed at reshaping the evolutionary trajectory toward cooperation and long-term survival and flourishing.
The Universal Algorithm of Natural Selection
Evolution isn’t just a biological process; it’s a universal optimization algorithm that applies to any type of entity—be it chemicals, groups, countries, companies, or even memes—as long as the system in question fulfills the following three key features:
Variation: In any population, members differ in characteristics. For example, mice may vary in size, speed, and color. In a digital landscape, social media platforms might vary in features, such as content delivery methods or user engagement strategies.
Selection: Different characteristics lead to different survival rates. Brown mice, for instance, may blend into their environment better, avoiding predators. Similarly, when one social media platform introduces features like infinite scroll or outrage algorithms, these tactics drive user engagement, forcing competing platforms to adopt similar features to stay relevant.
Retention: Traits that enhance survival are passed on. Brown mice, being better camouflaged, are more likely to survive and reproduce, increasing their numbers. Similarly, successful but potentially harmful features, like engagement-driven algorithms, become standard across platforms, locking the industry into a cycle of ever-increasing engagement tactics.
Darwinian Demons and Multipolar Traps
Humans, like all life forms, face selection pressures that shape behavior. However, these pressures can sometimes lead to self-serving actions with negative consequences, which I term “Darwinian demons.” I broadly define Darwinian demons as selection pressures that drive short-sighted, goal-oriented behaviors, which, over time, can lead to widespread net-negative consequences.
Alternative terms for this phenomenon include multipolar traps, Moloch, coordination problems, social dilemmas, race to the bottom, and the tragedy of the commons. I chose the term ‘Darwinian Demons’ to emphasize its evolutionary roots and the evolutionary lensI will use to analyze them.
Once you begin to look, examples of Darwinian demons emerge everywhere, both in societies and in nature. Below is a list to help build our intuition. While each example deserves a deeper exploration—something I delve into more fully in my book—these snapshots offer a glimpse into the essence of what a Darwinian demon is.
Nature:
Cancer: The selection pressure for unchecked cellular proliferation leads to the adaptive behavior of uncontrollable growth, with negative outcomes like the destruction of the host organism.
Viruses: The selection pressure to replicate quickly drives the adaptive behavior of causing severe infections, resulting in negative outcomes such as host death and widespread disease.
Parasites: The selection pressure for survival and reproduction encourages the adaptive behavior of exploiting hosts, with negative outcomes like host debilitation and ecosystem destabilization.
Predators: The selection pressure to secure immediate sustenance leads to the adaptive behavior of overhunting, causing negative outcomes such as prey population collapses and ecosystem imbalance.
Algae Bloom: The selection pressure to aggressively multiply drives the adaptive behavior of rapid algae growth, resulting in negative outcomes like oxygen depletion in water bodies, which can lead to dead zones and harm to aquatic life.
Society:
Corporate Fraud: The selection pressure to meet financial targets drives the adaptive behavior of engaging in fraudulent activities, leading to negative outcomes such as massive societal harm when the deceit is uncovered.
Replication crisis: The selection pressure to publish frequently or maximize h-index drives the adaptive behavior of falsifying research results, leading to negative outcomes such as p-value hacking, the replication crisis, and lost trust in academia.
Fake News: The selection pressure to increase ad revenue and social media engagement encourages the adaptive behavior of prioritizing clickbait or even fake news, with negative outcomes including the spread of misinformation and distorted public discourse.
Tax Havens: The selection pressure on small nations to attract capital leads to the adaptive behavior for other countries to also lower taxes, resulting in negative outcomes where important infrastructure, fundamental R&D, and social programs don’t get enough government funding.
Gerrymandering: The selection pressure to secure electoral victories drives the adaptive behavior of manipulating district boundaries, causing negative outcomes such as undermined democratic fairness and public trust.
Beauty Standards: The selection pressure for social media engagement leads to the adaptive behavior of widespread use of AI beauty filters, resulting in negative outcomes like unrealistic beauty standards and mental health issues.
Grade Inflation: The selection pressure for financial gain encourages the adaptive behavior of inflating grades in diploma mills, with negative outcomes including the devaluation of educational credentials and academic integrity.
Aggressive Policing: The selection pressure to meet ticketing quotas results in the adaptive behavior of prioritizing enforcement over community building, leading to negative outcomes like public distrust in the police.
Furthermore, In many cases, Darwinian demons have driven a cultural shift towards the evolution of doublespeak to make adaptive behavior with negative consequences more palatable. Terms like ‘strategic financial engineering’ often mask accounting manipulation; ‘tax planning’ can be a euphemism for aggressive tax avoidance; mass layoffs are disguised as ‘restructuring’; ‘revenue optimization’ may conceal misleading pricing strategies; and paying fines for rule-breaking is reframed as merely the ‘cost of doing business.’ In politics and government, ‘enhanced interrogation techniques’ is a euphemism for torture; ‘collateral damage’ softens the reality of civilian casualties in military operations; ‘right-sizing’ often means cutting public services; and ‘national security measures’ can sometimes cloak actions that infringe on civil liberties.
By popularizing the concept of Darwinian demons, I aim to shift the focus from the simplistic “bad apples” narrative to more systemic explanations rooted in economics and evolutionary biology. This perspective frees individuals from bearing sole responsibility for systemic problems and fosters a deeper understanding of their causes. To tackle society’s greatest challenges, we must look beyond blaming individual apples and examine the orchard that consistently produces them.
Why are the demons so hard to escape?
The core issue isn’t just that it’s often adaptive to act in ways that harm others—it’s that individual agents frequently find themselves trapped in a cycle where harming others is the only rational option. Game theory, and specifically the concept of the tragedy of the commons, helps explain why Darwinian demons are so difficult to overcome. The tragedy of the commons describes scenarios where shared resources are depleted because individuals, acting independently and rationally in their own self-interest, fail to consider the long-term impact on the group. A real-world example is the overfishing in the Grand Banks cod fishery, which led to the collapse of the cod population and devastated the local industry. The outcomes of this dynamic can be illustrated in the following matrix:
Other Players Follow Quota
Other Players Overfish
Player Follows Quota
Sustainable fish stocks for all
Depletion for player, short-term gain for others
Player Overfishes
Depletion for others, short-term gain for player
Rapid depletion for all, long-term losses for all
The key lesson is that rational choices made by individuals don’t always result in the best outcome for the group. This is often referred to as an inefficient Nash equilibrium. In a Nash equilibrium, each player’s strategy is optimized for their individual benefit, assuming others’ strategies remain constant. However, when these individual optimizations lead to a collectively suboptimal result, the equilibrium is considered inefficient.
Without mechanisms to promote and enforce cooperation—like laws enforcing fishing quotas or economic incentives for sustainability—natural selection tends to favor those who defect. When defectors thrive in a mixed population due to selection pressures, more individuals will adapt by choosing defection. Over time, this process erodes the number of cooperators until they eventually all become defectors.
Is Selfishness Really Such a Bad Thing?
In the example of the competing fisheries, the selection pressure for profit prompts the fisheries to behave selfishly in a way that optimizes short-term survival but leads to a depleted fish stock for everyone. And yet there are plenty of other examples where defection can beget positive outcomes—in theory. According to Adam Smith’s principle of the “invisible hand,” when individuals act in their own self-interest, they unknowingly contribute to the economic well-being of the community. This happens through the pursuit of profit, which encourages businesses to produce goods and services that are in demand. In turn, this production meets consumers’ needs and desires, leading to greater overall societal welfare. So is selfishness really such a bad thing? Or as Gordon Gekko famously put it in the movie Wall Street:
“The point is, ladies and gentlemen, that greed, for lack of a better word, is good. Greed is right, greed works. Greed clarifies, cuts through, and captures the essence of the evolutionary spirit. Greed, in all of its forms—greed for life, for money, for love, knowledge—has marked the upward surge of mankind.”
While it’s undeniable that selfishness in a free market has been a tremendous force for good—creating vast wealth and lifting billions out of poverty—we must also critically assess its potential downsides with a clear, unbiased perspective, free from ideological zealousness. Only by doing so can we develop policies that achieve Pareto optimal outcomes.
Take a company that aims to make a positive impact by selling a useful product—say, an innovative new sponge. The company might measure its success in terms of profit, reasoning that the more money it makes, the more sponges it can produce and distribute. However, by focusing primarily on profit, the company may neglect other important goals like employee welfare, consumer protection, or environmental stewardship. While society may benefit from cleaner dishes thanks to the sponge, the company’s disregard for these other priorities could result in a net negative impact.
In other words, in a world with limited resources—a world such as our own— whenwe optimize for one thing while remaining indifferent to another, then we by default actively optimize against the goal to which we’re indifferent. These adverse consequences are what economists call externalities, but the same phenomenon can also be seen in nature. When we optimize for short-term survival in an environment while remaining indifferent to other important goals—such as happiness, health, and well-being—then, by default, we end up optimizing against those values. In his book River Out of Eden, Richard Dawkins eloquently points this out:
“The total amount of suffering per year in the natural world is beyond all decent contemplation. During the minute it takes me to compose this sentence, thousands of animals are being eaten alive; others are running for their lives, whimpering with fear; others are being slowly devoured from within by rasping parasites; thousands of all kinds are dying of starvation, thirst and disease.”
Some even argue that life on Earth, over its 4 billion-year history, has been a net negative, generating more suffering than well-being. Moreover, Darwinian forces pose not only a threat to our well-being but also to our very existence. These zero-sum dynamics drive two existential arms races: for resources and power.
Natural Selection: the Generator Function of Existential Risks
To survive, any agent must secure resources, but this relentless pursuit has led to widespread environmental destruction, potentially setting the stage for a new mass extinction. In a world with finite resources, power is crucial for controlling access to land and resources, a dynamic that has led to the development of nuclear weapons. These weapons, if deployed in global conflict, could annihilate most, if not all, human and animal life.
When plotted over time, the impact of these arms races become increasingly evident. Consider the evolution from stone axes, which took days to fell a tree, to modern harvesters that accomplish the task in seconds. Societies that ruthlessly exploit resources grow more powerful, outcompeting those that live in harmony with nature. As a result, nearly half of all species are at risk of extinction within our lifetime.
Worse still, global competition for resources makes it advantageous to produce ever more powerful weapons. What began with stone axes, capable of killing one person in a minute, has escalated to nuclear bombs, capable of killing billions. This trend predictably led to the genesis of doomsday weapons and continues toward increasingly destructive technologies, such as engineered pandemics.
At some point, these arms races become inevitable; a nation-state cannot simply choose to opt out. As depicted in the movie Oppenheimer, scientists at Los Alamos National Laboratory believed they were in a race against the Germans to develop the first atomic weapon, making inaction untenable.
Similarly, the race towards superintelligence is following the same pattern, with China and the USA on opposing sides. However, AI is not just another technology; it is a technology that will enable the creation of new, more powerful technologies. Human intelligence has historically limited the lethality of the technologies we develop, but AI could potentially remove these limits. AI might enable the creation of new doomsday weapons, where the total destruction of all life on Earth is guaranteed. Such a weapon could act as the ultimate deterrent, creating an arms race to build them before anyone else. This misuse of AI for dangerous weapon manufacturing could occur even before an intelligence explosion or AI takeover scenario, posing a grave threat to humanity.
We Need Civilizational Alignment To Solve Technical Alignment
The technical alignment problem involves ensuring that AI systems consistently operate in line with human values, which may vary depending on the specific use case. For example, in the context of autonomous vehicles, a 2016 study published in Science found that most people agree that vehicles should be programmed to prioritize the welfare of all individuals, including pedestrians, even at the expense of the driver and passengers. However, the same study found that people were three times less likely to purchase such an aligned car themselves.
The lesson, put simply, is this: A solution is only impactful if people actually adopt it—and often they won’t. Even if we successfully solve the technical alignment problem and create AI systems that reliably follow what most humans would consider benevolent intentions, it doesn’t guarantee that AI will no longer pose a threat. The world is still filled with individuals—CEOs driven by profit, military leaders seeking more lethal weapons, and politicians aiming to advance their own agendas—who will be motivated to develop and deploy AI that serves their interests rather than society’s. Much like how most drivers would prefer an autonomous vehicle that prioritizes their own survival over that of others.
Solving the technical AI alignment problem is a noble goal, and a necessary one. But unless we also find ways to solve the societal alignment problem, the quest to make AI universally safe and helpful is bound to fail. To address the societal alignment problem, we must establish robust global systems of governance that incentivize the use of ethically aligned AI. This means creating frameworks where it becomes not just a moral choice but also a practical and advantageous one for companies and governments to adopt AI systems that align with human values. Without such a system, we risk the emergence of regulatory loopholes—akin to tax havens—where certain nations may strip away safety regulations and “human-in-the-loop” requirements in a bid to attract AI enterprises. As AI continues to automate a growing share of the global economy, these regulatory gaps could lead to a race to the bottom, undermining efforts to make AI universally safe and beneficial. Only through coordinated global efforts can we ensure that AI serves the collective good rather than the narrow interests of a few.
In part 2, I’ll challenge the notion that cooperation is an evolutionary inevitability by diving into our evolutionary past and exploring how life’s inherent fragility could be a key to understanding the Fermi Paradox. Then, in Part 3, get ready for an exciting exploration of how the interconnectedness of global value chains and the power of indirect reciprocity and reputation can be harnessed to build a decentralized, tyranny-free global governance system. Stay tuned!
Darwinian Traps and Existential Risks
Global coordination stands as arguably the most critical challenge facing humanity today, functioning both as a necessary component for solving existential risks and as a significant barrier to effective mitigation. From nuclear proliferation to artificial intelligence development and climate change, our inability to collaborate effectively on a global scale not only exacerbates these threats but also perpetuates the emergence of new systemic vulnerabilities if left unaddressed.
In this sequence, I will argue that the root of this coordination problem lies in the very mechanisms that shaped our species: natural selection. This evolutionary process, operating as a trial-and-error optimization algorithm, prioritizes immediate survival and reproduction over long-term, global outcomes. As a result, our innate tendencies often favor short-term gains and localized benefits, even when they conflict with the greater good of our species and planet.
The inherent limitations of natural selection in predicting future optimal states have left us ill-equipped to handle global-scale challenges. In a world of finite resources, competition rather than cooperation has often been the more adaptive trait, leading to the emergence of self-interested behaviors that arguably dominate modern societies. This evolutionary legacy manifests in the form of nationalistic tendencies, economic rivalries, dangerous arms races and a general reluctance to sacrifice immediate benefits for long-term collective gains.
This three-part series summarizes my book: The Darwinian Trap: The Hidden Evolutionary Forces That Explain Our World (and Threaten Our Future).
Part 1 (the part you are reading): This part explores how evolutionary selection pressures contribute to existential risks, particularly through global resource, power, and intelligence arms races.
Part 2: This part delves into the evolution of cooperation and discusses how survivorship bias might mislead us into thinking cooperation is easier to achieve than it actually is. It suggests that life may be inherently fragile, potentially containing the seeds of its own destruction, which could provide an explanation for the Fermi Paradox.
Part 3: The final part examines strategies to overcome global coordination challenges, both through centralized and decentralized approaches. It introduces the concept of “reputational markets” (a variant of prediction markets) as a tool for supply chain governance, aimed at reshaping the evolutionary trajectory toward cooperation and long-term survival and flourishing.
The Universal Algorithm of Natural Selection
Evolution isn’t just a biological process; it’s a universal optimization algorithm that applies to any type of entity—be it chemicals, groups, countries, companies, or even memes—as long as the system in question fulfills the following three key features:
Variation: In any population, members differ in characteristics. For example, mice may vary in size, speed, and color. In a digital landscape, social media platforms might vary in features, such as content delivery methods or user engagement strategies.
Selection: Different characteristics lead to different survival rates. Brown mice, for instance, may blend into their environment better, avoiding predators. Similarly, when one social media platform introduces features like infinite scroll or outrage algorithms, these tactics drive user engagement, forcing competing platforms to adopt similar features to stay relevant.
Retention: Traits that enhance survival are passed on. Brown mice, being better camouflaged, are more likely to survive and reproduce, increasing their numbers. Similarly, successful but potentially harmful features, like engagement-driven algorithms, become standard across platforms, locking the industry into a cycle of ever-increasing engagement tactics.
Darwinian Demons and Multipolar Traps
Humans, like all life forms, face selection pressures that shape behavior. However, these pressures can sometimes lead to self-serving actions with negative consequences, which I term “Darwinian demons.” I broadly define Darwinian demons as selection pressures that drive short-sighted, goal-oriented behaviors, which, over time, can lead to widespread net-negative consequences.
Alternative terms for this phenomenon include multipolar traps, Moloch, coordination problems, social dilemmas, race to the bottom, and the tragedy of the commons. I chose the term ‘Darwinian Demons’ to emphasize its evolutionary roots and the evolutionary lensI will use to analyze them.
Once you begin to look, examples of Darwinian demons emerge everywhere, both in societies and in nature. Below is a list to help build our intuition. While each example deserves a deeper exploration—something I delve into more fully in my book—these snapshots offer a glimpse into the essence of what a Darwinian demon is.
Nature:
Cancer: The selection pressure for unchecked cellular proliferation leads to the adaptive behavior of uncontrollable growth, with negative outcomes like the destruction of the host organism.
Viruses: The selection pressure to replicate quickly drives the adaptive behavior of causing severe infections, resulting in negative outcomes such as host death and widespread disease.
Parasites: The selection pressure for survival and reproduction encourages the adaptive behavior of exploiting hosts, with negative outcomes like host debilitation and ecosystem destabilization.
Predators: The selection pressure to secure immediate sustenance leads to the adaptive behavior of overhunting, causing negative outcomes such as prey population collapses and ecosystem imbalance.
Algae Bloom: The selection pressure to aggressively multiply drives the adaptive behavior of rapid algae growth, resulting in negative outcomes like oxygen depletion in water bodies, which can lead to dead zones and harm to aquatic life.
Society:
Corporate Fraud: The selection pressure to meet financial targets drives the adaptive behavior of engaging in fraudulent activities, leading to negative outcomes such as massive societal harm when the deceit is uncovered.
Replication crisis: The selection pressure to publish frequently or maximize h-index drives the adaptive behavior of falsifying research results, leading to negative outcomes such as p-value hacking, the replication crisis, and lost trust in academia.
Fake News: The selection pressure to increase ad revenue and social media engagement encourages the adaptive behavior of prioritizing clickbait or even fake news, with negative outcomes including the spread of misinformation and distorted public discourse.
Tax Havens: The selection pressure on small nations to attract capital leads to the adaptive behavior for other countries to also lower taxes, resulting in negative outcomes where important infrastructure, fundamental R&D, and social programs don’t get enough government funding.
Gerrymandering: The selection pressure to secure electoral victories drives the adaptive behavior of manipulating district boundaries, causing negative outcomes such as undermined democratic fairness and public trust.
Beauty Standards: The selection pressure for social media engagement leads to the adaptive behavior of widespread use of AI beauty filters, resulting in negative outcomes like unrealistic beauty standards and mental health issues.
Grade Inflation: The selection pressure for financial gain encourages the adaptive behavior of inflating grades in diploma mills, with negative outcomes including the devaluation of educational credentials and academic integrity.
Aggressive Policing: The selection pressure to meet ticketing quotas results in the adaptive behavior of prioritizing enforcement over community building, leading to negative outcomes like public distrust in the police.
Furthermore, In many cases, Darwinian demons have driven a cultural shift towards the evolution of doublespeak to make adaptive behavior with negative consequences more palatable. Terms like ‘strategic financial engineering’ often mask accounting manipulation; ‘tax planning’ can be a euphemism for aggressive tax avoidance; mass layoffs are disguised as ‘restructuring’; ‘revenue optimization’ may conceal misleading pricing strategies; and paying fines for rule-breaking is reframed as merely the ‘cost of doing business.’ In politics and government, ‘enhanced interrogation techniques’ is a euphemism for torture; ‘collateral damage’ softens the reality of civilian casualties in military operations; ‘right-sizing’ often means cutting public services; and ‘national security measures’ can sometimes cloak actions that infringe on civil liberties.
By popularizing the concept of Darwinian demons, I aim to shift the focus from the simplistic “bad apples” narrative to more systemic explanations rooted in economics and evolutionary biology. This perspective frees individuals from bearing sole responsibility for systemic problems and fosters a deeper understanding of their causes. To tackle society’s greatest challenges, we must look beyond blaming individual apples and examine the orchard that consistently produces them.
Why are the demons so hard to escape?
The core issue isn’t just that it’s often adaptive to act in ways that harm others—it’s that individual agents frequently find themselves trapped in a cycle where harming others is the only rational option. Game theory, and specifically the concept of the tragedy of the commons, helps explain why Darwinian demons are so difficult to overcome. The tragedy of the commons describes scenarios where shared resources are depleted because individuals, acting independently and rationally in their own self-interest, fail to consider the long-term impact on the group. A real-world example is the overfishing in the Grand Banks cod fishery, which led to the collapse of the cod population and devastated the local industry. The outcomes of this dynamic can be illustrated in the following matrix:
Other Players Follow Quota
Other Players Overfish
The key lesson is that rational choices made by individuals don’t always result in the best outcome for the group. This is often referred to as an inefficient Nash equilibrium. In a Nash equilibrium, each player’s strategy is optimized for their individual benefit, assuming others’ strategies remain constant. However, when these individual optimizations lead to a collectively suboptimal result, the equilibrium is considered inefficient.
Without mechanisms to promote and enforce cooperation—like laws enforcing fishing quotas or economic incentives for sustainability—natural selection tends to favor those who defect. When defectors thrive in a mixed population due to selection pressures, more individuals will adapt by choosing defection. Over time, this process erodes the number of cooperators until they eventually all become defectors.
Is Selfishness Really Such a Bad Thing?
In the example of the competing fisheries, the selection pressure for profit prompts the fisheries to behave selfishly in a way that optimizes short-term survival but leads to a depleted fish stock for everyone. And yet there are plenty of other examples where defection can beget positive outcomes—in theory. According to Adam Smith’s principle of the “invisible hand,” when individuals act in their own self-interest, they unknowingly contribute to the economic well-being of the community. This happens through the pursuit of profit, which encourages businesses to produce goods and services that are in demand. In turn, this production meets consumers’ needs and desires, leading to greater overall societal welfare. So is selfishness really such a bad thing? Or as Gordon Gekko famously put it in the movie Wall Street:
While it’s undeniable that selfishness in a free market has been a tremendous force for good—creating vast wealth and lifting billions out of poverty—we must also critically assess its potential downsides with a clear, unbiased perspective, free from ideological zealousness. Only by doing so can we develop policies that achieve Pareto optimal outcomes.
Take a company that aims to make a positive impact by selling a useful product—say, an innovative new sponge. The company might measure its success in terms of profit, reasoning that the more money it makes, the more sponges it can produce and distribute. However, by focusing primarily on profit, the company may neglect other important goals like employee welfare, consumer protection, or environmental stewardship. While society may benefit from cleaner dishes thanks to the sponge, the company’s disregard for these other priorities could result in a net negative impact.
In other words, in a world with limited resources—a world such as our own— when we optimize for one thing while remaining indifferent to another, then we by default actively optimize against the goal to which we’re indifferent. These adverse consequences are what economists call externalities, but the same phenomenon can also be seen in nature. When we optimize for short-term survival in an environment while remaining indifferent to other important goals—such as happiness, health, and well-being—then, by default, we end up optimizing against those values. In his book River Out of Eden, Richard Dawkins eloquently points this out:
Some even argue that life on Earth, over its 4 billion-year history, has been a net negative, generating more suffering than well-being. Moreover, Darwinian forces pose not only a threat to our well-being but also to our very existence. These zero-sum dynamics drive two existential arms races: for resources and power.
Natural Selection: the Generator Function of Existential Risks
To survive, any agent must secure resources, but this relentless pursuit has led to widespread environmental destruction, potentially setting the stage for a new mass extinction. In a world with finite resources, power is crucial for controlling access to land and resources, a dynamic that has led to the development of nuclear weapons. These weapons, if deployed in global conflict, could annihilate most, if not all, human and animal life.
When plotted over time, the impact of these arms races become increasingly evident. Consider the evolution from stone axes, which took days to fell a tree, to modern harvesters that accomplish the task in seconds. Societies that ruthlessly exploit resources grow more powerful, outcompeting those that live in harmony with nature. As a result, nearly half of all species are at risk of extinction within our lifetime.
Worse still, global competition for resources makes it advantageous to produce ever more powerful weapons. What began with stone axes, capable of killing one person in a minute, has escalated to nuclear bombs, capable of killing billions. This trend predictably led to the genesis of doomsday weapons and continues toward increasingly destructive technologies, such as engineered pandemics.
At some point, these arms races become inevitable; a nation-state cannot simply choose to opt out. As depicted in the movie Oppenheimer, scientists at Los Alamos National Laboratory believed they were in a race against the Germans to develop the first atomic weapon, making inaction untenable.
Similarly, the race towards superintelligence is following the same pattern, with China and the USA on opposing sides. However, AI is not just another technology; it is a technology that will enable the creation of new, more powerful technologies. Human intelligence has historically limited the lethality of the technologies we develop, but AI could potentially remove these limits. AI might enable the creation of new doomsday weapons, where the total destruction of all life on Earth is guaranteed. Such a weapon could act as the ultimate deterrent, creating an arms race to build them before anyone else. This misuse of AI for dangerous weapon manufacturing could occur even before an intelligence explosion or AI takeover scenario, posing a grave threat to humanity.
We Need Civilizational Alignment To Solve Technical Alignment
The technical alignment problem involves ensuring that AI systems consistently operate in line with human values, which may vary depending on the specific use case. For example, in the context of autonomous vehicles, a 2016 study published in Science found that most people agree that vehicles should be programmed to prioritize the welfare of all individuals, including pedestrians, even at the expense of the driver and passengers. However, the same study found that people were three times less likely to purchase such an aligned car themselves.
The lesson, put simply, is this: A solution is only impactful if people actually adopt it—and often they won’t. Even if we successfully solve the technical alignment problem and create AI systems that reliably follow what most humans would consider benevolent intentions, it doesn’t guarantee that AI will no longer pose a threat. The world is still filled with individuals—CEOs driven by profit, military leaders seeking more lethal weapons, and politicians aiming to advance their own agendas—who will be motivated to develop and deploy AI that serves their interests rather than society’s. Much like how most drivers would prefer an autonomous vehicle that prioritizes their own survival over that of others.
Solving the technical AI alignment problem is a noble goal, and a necessary one. But unless we also find ways to solve the societal alignment problem, the quest to make AI universally safe and helpful is bound to fail. To address the societal alignment problem, we must establish robust global systems of governance that incentivize the use of ethically aligned AI. This means creating frameworks where it becomes not just a moral choice but also a practical and advantageous one for companies and governments to adopt AI systems that align with human values. Without such a system, we risk the emergence of regulatory loopholes—akin to tax havens—where certain nations may strip away safety regulations and “human-in-the-loop” requirements in a bid to attract AI enterprises. As AI continues to automate a growing share of the global economy, these regulatory gaps could lead to a race to the bottom, undermining efforts to make AI universally safe and beneficial. Only through coordinated global efforts can we ensure that AI serves the collective good rather than the narrow interests of a few.
In part 2, I’ll challenge the notion that cooperation is an evolutionary inevitability by diving into our evolutionary past and exploring how life’s inherent fragility could be a key to understanding the Fermi Paradox. Then, in Part 3, get ready for an exciting exploration of how the interconnectedness of global value chains and the power of indirect reciprocity and reputation can be harnessed to build a decentralized, tyranny-free global governance system. Stay tuned!