At first, I thought you were arguing that AGI might be used by some extremists to wipe out most of humanity for some evil and/or stupid reason. Which does seem like a real risk.
Then you went on to point out that someone who thought that was likely might wipe out most of humanity (not including themselves) as a simple survival strategy, since otherwise someone else will wipe them out (along with most other people). As you note, this requires a high level of unconcern for normal moral considerations, which one would think very few people would countenance.
Now comes the strange part… You argue that actually maybe many people would be willing to wipe out most of humanity to save themselves, because… wiping out most of humanity sounds like a pretty good idea!
I’m glad that in the end you seem to still oppose wiping out most of humanity, but I think you have some factual misconceptions about this, and correcting them is a necessary first step to thinking of how to address the problem.
Concerning climate change, you write: “In the absence of any significant technological developments, sober current trajectory predictions seem to me to range from ‘human extinction’ to ‘catastrophic, but survivable’”.
No. Those are not “sober” predictions. They are alarmist claptrap with no scientific basis. You have been lied to. Without getting into details, you might want to contemplate that global temperatures were probably higher than today during the “Holocene Climatic Optimum” around 8000 years ago. That was the time when civilization developed. And temperatures were significantly higher in the previous interglacial, around 120,000 years ago. And the reference point for supposedly-disastrous global warming to come is “pre-industrial” time, which was in the “little ice age”, when low temperatures were causing significant hardship. Now, I know that the standard alarmist response is that it’s the rate of change that matters. But things changed pretty quickly at the end of the last ice age, so this is hardly unprecedented. And you shouldn’t believe the claims made about rates of change in any case—actual science on this question has stagnated for decades, with remarkably little progress being made on reducing the large uncertainty about how much warming CO2 actually causes.
Next, you say that the modern economy is relatively humane “under conditions of growth, which, under current conditions, depends on a growing population and rising consumption. Under stagnant or deflationary conditions it can be expected to become more cutthroat, violent, undemocratic and unjust.”
Certainly, history teaches that a social turn towards violence is quite possible. We haven’t transcended human nature. But the idea that continual growth is needed to keep the economy from deteriorating just has no basis in fact. Capitalist economies can operate perfectly fine without growth. Of course, there’s no guarantee that the economy will be allowed to operate fine. There have been many disastrous economic policies in the past. Again, human nature is still with us, and is complicated. Nobody knows whether social degeneration into poverty and tyranny is more likely with growth or without growth.
Finally, the idea that a world with a small population will be some sort of utopia is also quite disconnected from reality. That wasn’t the way things were historically. And even if it was, it woudn’t be stable, since population will grow if there’s plenty of food, no disease, no violence, etc.
So, I think your first step should be to realize that wiping out most of humanity would not be a good thing. At all. That should make it a lot easier to convince other people not to do it.
Your post reads a bit strangely.
At first, I thought you were arguing that AGI might be used by some extremists to wipe out most of humanity for some evil and/or stupid reason. Which does seem like a real risk.
Then you went on to point out that someone who thought that was likely might wipe out most of humanity (not including themselves) as a simple survival strategy, since otherwise someone else will wipe them out (along with most other people). As you note, this requires a high level of unconcern for normal moral considerations, which one would think very few people would countenance.
Now comes the strange part… You argue that actually maybe many people would be willing to wipe out most of humanity to save themselves, because… wiping out most of humanity sounds like a pretty good idea!
I’m glad that in the end you seem to still oppose wiping out most of humanity, but I think you have some factual misconceptions about this, and correcting them is a necessary first step to thinking of how to address the problem.
Concerning climate change, you write: “In the absence of any significant technological developments, sober current trajectory predictions seem to me to range from ‘human extinction’ to ‘catastrophic, but survivable’”.
No. Those are not “sober” predictions. They are alarmist claptrap with no scientific basis. You have been lied to. Without getting into details, you might want to contemplate that global temperatures were probably higher than today during the “Holocene Climatic Optimum” around 8000 years ago. That was the time when civilization developed. And temperatures were significantly higher in the previous interglacial, around 120,000 years ago. And the reference point for supposedly-disastrous global warming to come is “pre-industrial” time, which was in the “little ice age”, when low temperatures were causing significant hardship. Now, I know that the standard alarmist response is that it’s the rate of change that matters. But things changed pretty quickly at the end of the last ice age, so this is hardly unprecedented. And you shouldn’t believe the claims made about rates of change in any case—actual science on this question has stagnated for decades, with remarkably little progress being made on reducing the large uncertainty about how much warming CO2 actually causes.
Next, you say that the modern economy is relatively humane “under conditions of growth, which, under current conditions, depends on a growing population and rising consumption. Under stagnant or deflationary conditions it can be expected to become more cutthroat, violent, undemocratic and unjust.”
Certainly, history teaches that a social turn towards violence is quite possible. We haven’t transcended human nature. But the idea that continual growth is needed to keep the economy from deteriorating just has no basis in fact. Capitalist economies can operate perfectly fine without growth. Of course, there’s no guarantee that the economy will be allowed to operate fine. There have been many disastrous economic policies in the past. Again, human nature is still with us, and is complicated. Nobody knows whether social degeneration into poverty and tyranny is more likely with growth or without growth.
Finally, the idea that a world with a small population will be some sort of utopia is also quite disconnected from reality. That wasn’t the way things were historically. And even if it was, it woudn’t be stable, since population will grow if there’s plenty of food, no disease, no violence, etc.
So, I think your first step should be to realize that wiping out most of humanity would not be a good thing. At all. That should make it a lot easier to convince other people not to do it.