Consider also that not everyone would believe, upon having the Singularity explained to them, that it would be a good thing.
There comes a point where the One Dial theory and similar acrobatics are just ways to rationalize away the fact that you’re trying to push everyone in a direction that they would hate or consider too dangerous because you personally want to see what is at the end of the road. That’s just good old manipulation, but arguments like the dial thing allow you to feel better about it and think that you’re still pursuing the good option, by narrowing the options down to two so you can pit the one you like against a purposefully made horrible strawman.
I very much agree with you here and in your AGI deployment as an act of aggression post; the overwhelming majority of humans do not want AGI/ASI and its straightforward consequences (total human technological unemployment and concomitant abyssal social/economical disempowerment), regardless of what paradisaical promises (for which there is no recourse if they are not granted: economically useless humans can’t go on strike, etc) are promised them.
The value (this is synonymous with “scarcity”) of human intelligence and labor output has been a foundation of every human social and economic system, from hunter-gatherer groups to highly-advanced technological societies. It is the bedrock onto which humanity has built cooperation, benevolence, compassion, and care. The value of human intelligence and labor output gives humans agency, meaning, decision-making power, and bargaining power towards each other and over corporations / governments. Beneficence flows from this general assumption of human labor value/scarcity.
So far, technological development has left this bedrock intact, even if it’s been bumpy (I was gonna say “rocky” but that’s a mixed metaphor for sure) on the surface. The bedrock’s still been there after the smoke cleared, time and time again. Comparing opponents of AGI/ASI with Luddites or the Unabomber, accusing them of being technophobes, or insinuating that they would have wanted to stop the industrial revolution is wildly specious: unlike every other invention or technological development, successful AGI/ASI development will convert this bedrock into sand. So far, technological development has been wildly beneficial for humanity: technological development that has no need for humans is not likely to hold to that record. The OpenAI mission is literally to create “highly autonomous systems that outperform humans at most economically valuable work”, a flowery way to say “make human labor output worthless”. Fruitful cooperation between AGI/ASI and humans is unlikely to endure since at some point the transaction costs (humans don’t have APIs, are slow, need to sleep/eat/rest, etc) outweigh whatever benefits of cooperation.
There’s been an significant effort to avoid reckoning with or acknowledging these aspects of AGI/ASI (again, AGI/ASI is the explicit goal of AI labs like OpenAI; not autoregressive language models) and those likely (if not explicitly sought out) consequences in public-facing discourse of doomers vs accelerationists. As much as it pains me to come to this conclusion it really does feel like there’s a pervasive gentleman’s agreement to avoid saying “the goal is literally to make systems capable of bringing about total technological unemployment”. This is not aligned with the goals/desires/lives of the overwhelming majority of humanity, and the deception deployed to avoid widespread public realization of this sickens me.
I wrote a handful of comments on the EA forum about this as well.
I have read your comments on the EA forum and the points do resonate with me.
As a layman, I do have a personal distrust with the (what I’d call) anti-human ideologies driving the actors you refer to and agree that a majority of people do as well. It is hard to feel much joy in being extinct and replaced by synthetic beings in probably a way most would characterize as dumb (clippy being the extreme)
I also believe that fundamental changing of the human subjective experience (radical bioengineering or uploading to an extent) in order to erase the ability to suffer in general (not just medical cases like depression) as I have seen being brought up by futurist circles is also akin to death. I think it could possibly be a somewhat literal death, where my conscious experience actually stops if radical changes would occur, but I am completely uneducated and unqualified on how consciousness works.
I think that a hypothetical me, even with my memories, who is physically unable to experience any negative emotions would be philosophically dead. It would be unable to learn nor reflect and its fundamentally different subjective experience would be so radically different from me, and any future biological me should I grow older naturally, that I do not think memories alone would be enough to keep my identity. To my awareness, the majority of people would think similarly and that there is value ascribed to our human nature, including limitations, which has been reinforced by our media and culture. Though whether this attachment is purely a product of coping, I do not know. What I do know is that it is the current reality for every functional human being now and has been for thousands of years. I believe people would prefer sticking with it than relinquishing it for vague promises of ascended consciousness. I think this is somewhat supported by my subjective observation that to a lot of people who want a posthuman existence and what it entails, their end goal seems to often come back to creating simulations they themselves can live in normally.
I’m curious though if you have any hopes for the situation regarding the nebulous motivations of some AGI researchers, especially as AI and its risks have recently started becoming “mainstream”. Do you expect to see changes and their views challenged? My question is loaded, but it seems you are already invested in its answer.
I’m curious though if you have any hopes for the situation regarding the nebulous motivations of some AGI researchers, especially as AI and its risks have recently started becoming “mainstream”. Do you expect to see changes and their views challenged? My question is loaded, but it seems you are already invested in its answer.
I think there’s a case to be made for AGI/ASI development and deployment as a “hostis humani generis” act; and others have made the case as well. I am confused (and let’s be honest, increasingly aghast) as to why AI doomers rarely try to press this angle in their debates/public-facing writings.
To me it feels like AI doomers have been asleep on sentry duty, and I’m not exactly sure why. My best guesses look somewhat like “some level of agreement with the possible benefits of AGI/ASI” or “a belief that AGI/ASI is overwhelmingly inevitable and so it’s better not to show any sign of adversariality towards those developing it, so as to best influence them to mind safety”, but this is quite speculative on my part. I think LW/EA stuff inculcates in many a grievous and pervasive fear of upsetting AGI accelerationists/researchers/labs (fear of retaliatory paperclipping? fear of losing mostly illusory leverage and influence? getting memed into the idea that AGI/ASI is inevitable and unstoppable?).
I feel like this foundational dissonance makes AI doomers come across as confused fawny wordcels or hectoring cultists whenever they face AGI accelerationists / AI risk deniers (who in contrast tend to come across as open/frank/honest/aligned/of action/assertive/doers/etc). This vibe is really not conducive to convincing people of the risks/consequences of AGI/ASI.
I do have hopes but they feel kinda gated on “AI doomers” being many orders of magnitudes more honest, unflinchingly open, and unflatteringly frank about the ideologies that motivate AGI/ASI researchers and the intended/likely consequences of their success—even if “alignment/control” gets solved—of total technological unemployment and consequential social/economic human disempowerment, instead of continuing to treat AGI/ASI as some sort of neutral(if not outright necessary)-but-highly-risky technology like rockets or nukes or recombinant DNA technology. Also gated on explicitly countering the contentions that AGI/ASI—even if aligned—is inevitable/necessary/good or that China is a viable contender in this omnicidal race or that we need AGI/ASI to fight climate change or asteroids or pandemics or all the other (sorry for being profane) bullshit that gets trotted out to justify AGI/ASI development. And gated on explicitly saying that AGI/ASI accelerationists are transhumanist fundamentalists who are willing to sacrifice the entire human species on the altar of their ideology.
I don’t think AGI/ASI is inherently inevitable, but as long as AI doomers shy away from explaining that the AGI/ASI labs are specifically seeking (and likely soonish succeeding) to build systems strong enough to turn the yet-unbroken—from hunter-gatherer bands to July 2023 -- bedrock (“human labor is irreplaceably valuable”) assumption of human society into fine sand; I think there’s little hope of stopping AGI/ASI development.
Yup. These precise points were also the main argument of my other post on a post-AGI world, the benevolence of the butcher.
Also due to the AI discourse I’ve actually ended up learning more about the original Luddites and, hear hear, they actually weren’t the fanatical, reactionary anti-technology ignorant peasants that popular history mainly portrays them as. They were mostly workers who were angry about the way the machines were being used, not to make labour easier and safer, but to squeeze more profit out of less skilled workers to make lower quality products which in the end left almost everyone involved worse off except for the ones who owned the factories. That’s I think something we can relate to even now, and I’d say is even more important in the case of AGI. The risk that it simply ends up being owned by the few who create it leading thus to a total concentration of the productive power of humanity isn’t immaterial, in fact it looks like the default outcome.
The risk that it simply ends up being owned by the few who create it leading thus to a total concentration of the productive power of humanity isn’t immaterial, in fact it looks like the default outcome.
Yes, this is why I’ve been frustrated (and honestly aghast, given timelines) at the popular focus on AI doom and paperclips rather than the fact that this is the default (if not nigh-unavoidable) outcome of AGI/ASI, even if “alignment” gets solved. Comparisons with industrialization and other technological developments are specious because none of them had the potential to do anything close to this.
I think the doom narrative is still worth bringing up because this is what these people are risking for all of us in the pursuit of essentially conquering the world and/or personal immortality. That’s the level of insane supervillainy that this whole situation actually translates to. Just because they don’t think they’ll fail doesn’t mean they’re not likely to.
I’m also disappointed that the political left is dropping the ball so hard on opposing AI, turning to either contradictory “it’s really stupid, just a stochastic parrot, and also threatens our jobs somehow” statements, or focusing on details of its behaviour. There’s probably something deeper to say about capitalists openly making a bid to turn labour itself into capital.
First, let me say I appreciate you expressing your viewpoint and it does strike an emotional chord with me. With that said,
Wouldn’t an important invention such as the machine gun or obviously fission weapons fit your argument pattern? You could make a reasonable case that, like a world with technological unemployment, worlds where humans are cheap to slaughter is overall worse. That if you could coordinate with the world powers at that time to agree to an “automatic weapon moratorium” it would resort in a better world.
The problem is Kaiser Wilhelm and other historical leaders are going to say “suuurrrreee”, agree to the deal, and you already know the nasty surprise any power honoring such a deal will face on the battlefield. (Or Stalin would have said “sureee” to such a deal on fission weapons, and we can assume would immediately renege and test the devices in secret, only announcing their existence with a preemptive first strike on the enemies of the USSR).
What’s different now? Is there a property about AGI/ASI that makes such international agreements more feasible?
To add one piece of information that may not be well known: I work on inference accelerator ASICs and they are significantly simpler than GPUs. A large amount of Nvidias stack isn’t actually necessary if pure AI perf/training is your goal. So the only real bottleneck to monitor AI accelerators is that wafer processing equipment currently comes exclusively from asml for the highest end equipment, creating a monitorable supply chain for now. All bets are off if major superpowers build their own domestic equivalents, which they would be strongly incentivized to do in worlds where we know AGI is possible and have built working examples.
Wouldn’t an important invention such as the machine gun or obviously fission weapons fit your argument pattern? You could make a reasonable case that, like a world with technological unemployment, worlds where humans are cheap to slaughter is overall worse. That if you could coordinate with the world powers at that time to agree to an “automatic weapon moratorium” it would resort in a better world.
The problem is Kaiser Wilhelm and other historical leaders are going to say “suuurrrreee”, agree to the deal, and you already know the nasty surprise any power honoring such a deal will face on the battlefield. (Or Stalin would have said “sureee” to such a deal on fission weapons, and we can assume would immediately renege and test the devices in secret, only announcing their existence with a preemptive first strike on the enemies of the USSR).
I might be misunderstanding your point but I wasn’t trying to argue that it’s easy (or even feasible) to make robust international agreements not to develop AGI.
The machine gun and nuclear weapons don’t, AFAICT, fit my argument pattern. Powerful weapons like those certainly make humans easier to slaughter on industrial scales, but since humans are necessary to keep economies and industries and militaries running, military/political leaders have robust incentives to prevent large-scale slaughter of their own citizens and soldiers (and so do their adversaries for their own people). Which OK, this can get done by deterrence or arms-control agreement but it’s also started arms races, preemptive strikes, and wars hot and cold. Nevertheless, the bedrock of “human labor/intelligence is valuable/scarce” creates strong restoring forces towards “don’t senselessly slaughter tons of people”. It is possible to create robust-ish (pretty sure Russia’s cheating with them Novichoks) international agreements against weapons that are better at senseless civilian slaughter than at achieving military objectives, chemical weapons are the notable case.
The salient threat to me isn’t “AGI gives us better ways to kill people” (society has been coping remarkably well with better ways to kill people, up to and including a fleet of portable stars that can be dispatched to vaporize cities in the time it took me to write this comment), the salient threat to me (which seems inherent to the development of AGI/ASI) is “AGI renders the overwhelming majority of humanity economically/socially irrelevant, and therefore the overwhelming majority of humanity loses all agency, meaning, decision-making power, and bargaining power, and is vulnerable to inescapable and abyssal oppression if not outright killing because there’s no longer any robust incentives to keep them alive/happy/productive”.
I agree technological unemployment is a huge potential problem. Though like always the actual problem is aging. I think what people miss is they think of tasks to be done as a fixed pool, you don’t need more than 1 vehicle per person or less, or 1 dwelling, or n hours per year of medical care, or food etc. And neglect how AGI clearly cannot be trusted to do many things regardless of capabilities, there would need to be a fleet of human overseers armed with advanced tools.
It’s just what do you do for a 50 year old truck driver, expecting them to retrain to be an O’Neil colony construction supervisor doesn’t make sense unless you can treat their aging and restore neural plasticity.
Which is itself an immense megaproject not being done. Bet aging research would go a lot faster if we had the functional equivalent of a billion people working on it, and all billion are informed as to everyone else’s research outcomes.
Where I was going for in the analogy was much simpler. You don’t get a choice. In the immediate term, agreeing to not build machine guns and honoring it means you face a rat tat tat when it matters most. Similar for fission weapons, obviously your enemy is going to build a nuclear arsenal and try to vaporize all your key cities in a surprise attack.
The issues you mention happen long term. In the short term you can use AGI to automate many key tasks and become vastly more economically and militarily powerful.
I agree technological unemployment is a huge potential problem. Though like always the actual problem is aging.
I think this is a typical LW bias. No, I don’t enjoy the idea of death. But I would rather live a long and reasonably happy life in a human friendly world and then die when I am old than starve to death as one of the 7.9 billion casualties of the AGI Wars. The idea that there’s some sliver of a chance that in some future immortality is on the table for you, personally, is a delusion. I think life extension is very possible, and true immortality is not. But as things are either would only be on the table for, like, the CEOs of the big AI companies who got their biomarkers registered as part of the alignment protocol so that their product obeys them. Not for you. You’re the peasant whose blood, if necessary, cyber-Elizabeth Bathory will use for her rejuvenation rituals.
That’s never happened historically and aging treatments isn’t immortality, it’s just approximately a life expectancy of 10k years. Do you know who is richer than any CEO you name? Medicare. I bet they would like to stop paying all these medical bills, which would be the case if treated patients had the approximate morbidity rate of young adults.
You also need such treatments to be given at large scales to find and correct the edge cases. A rejuvenation treatment “beta tester” is exactly what it sounds, you will have a higher risk of death but get earlier access. Going to need a lot of beta testers.
The rational, data driven belief is that aging is treatable and that ASI systems with the cognitive capacity to take into account more variables than humans are mentally capable of could be built to systematically attack the problem. Doesn’t mean it will help anyone alive today, there are no guarantees. Because automated systems found whatever treatments are possible, automated systems can deliver the same treatments at low cost.
If you don’t think this is a reasonable conclusion, perhaps you could go into your reasoning. Arguments like you made above are unconvincing.
While it is true that certain esoteric treatments for aging like young blood transfusions are inherently limited in who can benefit, they don’t even work that well and de aged hemopoietic stem cells can be generated in automated laboratories and would be a real treatment everyone can benefit.
The wealthy are not powerful enough to “hoard” treatments, because Medicare et al represent the government, which has a monopoly on violence and incentives to not allow such hoarding.
The wealthy are not powerful enough to “hoard” treatments, because Medicare et al represent the government, which has a monopoly on violence and incentives to not allow such hoarding.
That’s naive. If a private has obedient ASI, they also have a monopoly on violence now. If labour has become superfluous, states have lost all incentive to care about the opinion of people.
I think worlds with the tools to treat most causes of human death ranks strictly higher than a world without those tools. In the same way that a world with running water ranks above worlds without it. Even today not everyone benefits from running water. If you could go back in time would you campaign against developing pipes and pumps because you believed only the rich would ever have running water? (Which was true for a period of time)
I would campaign against lead pipes and support the goths in destroying Rome which likely improved human futures over an alternative of widespread lead piping.
Running water doesn’t create the conditions to permanently disempower almost everyone, AGI does. What I’m talking about isn’t a situation in which initially only the rich benefit but then the tech gets cheaper and trickles down. It’s a permanent trap that destroys democracy and capitalism as we know them.
Consider also that not everyone would believe, upon having the Singularity explained to them, that it would be a good thing.
There comes a point where the One Dial theory and similar acrobatics are just ways to rationalize away the fact that you’re trying to push everyone in a direction that they would hate or consider too dangerous because you personally want to see what is at the end of the road. That’s just good old manipulation, but arguments like the dial thing allow you to feel better about it and think that you’re still pursuing the good option, by narrowing the options down to two so you can pit the one you like against a purposefully made horrible strawman.
I very much agree with you here and in your AGI deployment as an act of aggression post; the overwhelming majority of humans do not want AGI/ASI and its straightforward consequences (total human technological unemployment and concomitant abyssal social/economical disempowerment), regardless of what paradisaical promises (for which there is no recourse if they are not granted: economically useless humans can’t go on strike, etc) are promised them.
The value (this is synonymous with “scarcity”) of human intelligence and labor output has been a foundation of every human social and economic system, from hunter-gatherer groups to highly-advanced technological societies. It is the bedrock onto which humanity has built cooperation, benevolence, compassion, and care. The value of human intelligence and labor output gives humans agency, meaning, decision-making power, and bargaining power towards each other and over corporations / governments. Beneficence flows from this general assumption of human labor value/scarcity.
So far, technological development has left this bedrock intact, even if it’s been bumpy (I was gonna say “rocky” but that’s a mixed metaphor for sure) on the surface. The bedrock’s still been there after the smoke cleared, time and time again. Comparing opponents of AGI/ASI with Luddites or the Unabomber, accusing them of being technophobes, or insinuating that they would have wanted to stop the industrial revolution is wildly specious: unlike every other invention or technological development, successful AGI/ASI development will convert this bedrock into sand. So far, technological development has been wildly beneficial for humanity: technological development that has no need for humans is not likely to hold to that record. The OpenAI mission is literally to create “highly autonomous systems that outperform humans at most economically valuable work”, a flowery way to say “make human labor output worthless”. Fruitful cooperation between AGI/ASI and humans is unlikely to endure since at some point the transaction costs (humans don’t have APIs, are slow, need to sleep/eat/rest, etc) outweigh whatever benefits of cooperation.
There’s been an significant effort to avoid reckoning with or acknowledging these aspects of AGI/ASI (again, AGI/ASI is the explicit goal of AI labs like OpenAI; not autoregressive language models) and those likely (if not explicitly sought out) consequences in public-facing discourse of doomers vs accelerationists. As much as it pains me to come to this conclusion it really does feel like there’s a pervasive gentleman’s agreement to avoid saying “the goal is literally to make systems capable of bringing about total technological unemployment”. This is not aligned with the goals/desires/lives of the overwhelming majority of humanity, and the deception deployed to avoid widespread public realization of this sickens me.
I wrote a handful of comments on the EA forum about this as well.
I have read your comments on the EA forum and the points do resonate with me.
As a layman, I do have a personal distrust with the (what I’d call) anti-human ideologies driving the actors you refer to and agree that a majority of people do as well. It is hard to feel much joy in being extinct and replaced by synthetic beings in probably a way most would characterize as dumb (clippy being the extreme)
I also believe that fundamental changing of the human subjective experience (radical bioengineering or uploading to an extent) in order to erase the ability to suffer in general (not just medical cases like depression) as I have seen being brought up by futurist circles is also akin to death. I think it could possibly be a somewhat literal death, where my conscious experience actually stops if radical changes would occur, but I am completely uneducated and unqualified on how consciousness works.
I think that a hypothetical me, even with my memories, who is physically unable to experience any negative emotions would be philosophically dead. It would be unable to learn nor reflect and its fundamentally different subjective experience would be so radically different from me, and any future biological me should I grow older naturally, that I do not think memories alone would be enough to keep my identity. To my awareness, the majority of people would think similarly and that there is value ascribed to our human nature, including limitations, which has been reinforced by our media and culture. Though whether this attachment is purely a product of coping, I do not know. What I do know is that it is the current reality for every functional human being now and has been for thousands of years. I believe people would prefer sticking with it than relinquishing it for vague promises of ascended consciousness. I think this is somewhat supported by my subjective observation that to a lot of people who want a posthuman existence and what it entails, their end goal seems to often come back to creating simulations they themselves can live in normally.
I’m curious though if you have any hopes for the situation regarding the nebulous motivations of some AGI researchers, especially as AI and its risks have recently started becoming “mainstream”. Do you expect to see changes and their views challenged? My question is loaded, but it seems you are already invested in its answer.
I think there’s a case to be made for AGI/ASI development and deployment as a “hostis humani generis” act; and others have made the case as well. I am confused (and let’s be honest, increasingly aghast) as to why AI doomers rarely try to press this angle in their debates/public-facing writings.
To me it feels like AI doomers have been asleep on sentry duty, and I’m not exactly sure why. My best guesses look somewhat like “some level of agreement with the possible benefits of AGI/ASI” or “a belief that AGI/ASI is overwhelmingly inevitable and so it’s better not to show any sign of adversariality towards those developing it, so as to best influence them to mind safety”, but this is quite speculative on my part. I think LW/EA stuff inculcates in many a grievous and pervasive fear of upsetting AGI accelerationists/researchers/labs (fear of retaliatory paperclipping? fear of losing mostly illusory leverage and influence? getting memed into the idea that AGI/ASI is inevitable and unstoppable?).
It seems to me like people whose primary tool of action/thinking/orienting is some sort of scientific/truth-finding rational system will inevitably lose against groups of doggedly motivated, strategically+technically competent, cunning unilateralists who gleefully use deceit / misdirection to prevent normies from catching on to what they’re doing and motivated by fundamentalist pseudo-religious impulses (“the prospect of immortality, of solving philosophy”).
I feel like this foundational dissonance makes AI doomers come across as confused fawny wordcels or hectoring cultists whenever they face AGI accelerationists / AI risk deniers (who in contrast tend to come across as open/frank/honest/aligned/of action/assertive/doers/etc). This vibe is really not conducive to convincing people of the risks/consequences of AGI/ASI.
I do have hopes but they feel kinda gated on “AI doomers” being many orders of magnitudes more honest, unflinchingly open, and unflatteringly frank about the ideologies that motivate AGI/ASI researchers and the intended/likely consequences of their success—even if “alignment/control” gets solved—of total technological unemployment and consequential social/economic human disempowerment, instead of continuing to treat AGI/ASI as some sort of neutral(if not outright necessary)-but-highly-risky technology like rockets or nukes or recombinant DNA technology. Also gated on explicitly countering the contentions that AGI/ASI—even if aligned—is inevitable/necessary/good or that China is a viable contender in this omnicidal race or that we need AGI/ASI to fight climate change or asteroids or pandemics or all the other (sorry for being profane) bullshit that gets trotted out to justify AGI/ASI development. And gated on explicitly saying that AGI/ASI accelerationists are transhumanist fundamentalists who are willing to sacrifice the entire human species on the altar of their ideology.
I don’t think AGI/ASI is inherently inevitable, but as long as AI doomers shy away from explaining that the AGI/ASI labs are specifically seeking (and likely soonish succeeding) to build systems strong enough to turn the yet-unbroken—from hunter-gatherer bands to July 2023 -- bedrock (“human labor is irreplaceably valuable”) assumption of human society into fine sand; I think there’s little hope of stopping AGI/ASI development.
Yup. These precise points were also the main argument of my other post on a post-AGI world, the benevolence of the butcher.
Also due to the AI discourse I’ve actually ended up learning more about the original Luddites and, hear hear, they actually weren’t the fanatical, reactionary anti-technology ignorant peasants that popular history mainly portrays them as. They were mostly workers who were angry about the way the machines were being used, not to make labour easier and safer, but to squeeze more profit out of less skilled workers to make lower quality products which in the end left almost everyone involved worse off except for the ones who owned the factories. That’s I think something we can relate to even now, and I’d say is even more important in the case of AGI. The risk that it simply ends up being owned by the few who create it leading thus to a total concentration of the productive power of humanity isn’t immaterial, in fact it looks like the default outcome.
Yes, this is why I’ve been frustrated (and honestly aghast, given timelines) at the popular focus on AI doom and paperclips rather than the fact that this is the default (if not nigh-unavoidable) outcome of AGI/ASI, even if “alignment” gets solved. Comparisons with industrialization and other technological developments are specious because none of them had the potential to do anything close to this.
I think the doom narrative is still worth bringing up because this is what these people are risking for all of us in the pursuit of essentially conquering the world and/or personal immortality. That’s the level of insane supervillainy that this whole situation actually translates to. Just because they don’t think they’ll fail doesn’t mean they’re not likely to.
I’m also disappointed that the political left is dropping the ball so hard on opposing AI, turning to either contradictory “it’s really stupid, just a stochastic parrot, and also threatens our jobs somehow” statements, or focusing on details of its behaviour. There’s probably something deeper to say about capitalists openly making a bid to turn labour itself into capital.
First, let me say I appreciate you expressing your viewpoint and it does strike an emotional chord with me. With that said,
Wouldn’t an important invention such as the machine gun or obviously fission weapons fit your argument pattern? You could make a reasonable case that, like a world with technological unemployment, worlds where humans are cheap to slaughter is overall worse. That if you could coordinate with the world powers at that time to agree to an “automatic weapon moratorium” it would resort in a better world.
The problem is Kaiser Wilhelm and other historical leaders are going to say “suuurrrreee”, agree to the deal, and you already know the nasty surprise any power honoring such a deal will face on the battlefield. (Or Stalin would have said “sureee” to such a deal on fission weapons, and we can assume would immediately renege and test the devices in secret, only announcing their existence with a preemptive first strike on the enemies of the USSR).
What’s different now? Is there a property about AGI/ASI that makes such international agreements more feasible?
To add one piece of information that may not be well known: I work on inference accelerator ASICs and they are significantly simpler than GPUs. A large amount of Nvidias stack isn’t actually necessary if pure AI perf/training is your goal. So the only real bottleneck to monitor AI accelerators is that wafer processing equipment currently comes exclusively from asml for the highest end equipment, creating a monitorable supply chain for now. All bets are off if major superpowers build their own domestic equivalents, which they would be strongly incentivized to do in worlds where we know AGI is possible and have built working examples.
I might be misunderstanding your point but I wasn’t trying to argue that it’s easy (or even feasible) to make robust international agreements not to develop AGI.
The machine gun and nuclear weapons don’t, AFAICT, fit my argument pattern. Powerful weapons like those certainly make humans easier to slaughter on industrial scales, but since humans are necessary to keep economies and industries and militaries running, military/political leaders have robust incentives to prevent large-scale slaughter of their own citizens and soldiers (and so do their adversaries for their own people). Which OK, this can get done by deterrence or arms-control agreement but it’s also started arms races, preemptive strikes, and wars hot and cold. Nevertheless, the bedrock of “human labor/intelligence is valuable/scarce” creates strong restoring forces towards “don’t senselessly slaughter tons of people”. It is possible to create robust-ish (pretty sure Russia’s cheating with them Novichoks) international agreements against weapons that are better at senseless civilian slaughter than at achieving military objectives, chemical weapons are the notable case.
The salient threat to me isn’t “AGI gives us better ways to kill people” (society has been coping remarkably well with better ways to kill people, up to and including a fleet of portable stars that can be dispatched to vaporize cities in the time it took me to write this comment), the salient threat to me (which seems inherent to the development of AGI/ASI) is “AGI renders the overwhelming majority of humanity economically/socially irrelevant, and therefore the overwhelming majority of humanity loses all agency, meaning, decision-making power, and bargaining power, and is vulnerable to inescapable and abyssal oppression if not outright killing because there’s no longer any robust incentives to keep them alive/happy/productive”.
I agree technological unemployment is a huge potential problem. Though like always the actual problem is aging. I think what people miss is they think of tasks to be done as a fixed pool, you don’t need more than 1 vehicle per person or less, or 1 dwelling, or n hours per year of medical care, or food etc. And neglect how AGI clearly cannot be trusted to do many things regardless of capabilities, there would need to be a fleet of human overseers armed with advanced tools.
It’s just what do you do for a 50 year old truck driver, expecting them to retrain to be an O’Neil colony construction supervisor doesn’t make sense unless you can treat their aging and restore neural plasticity.
Which is itself an immense megaproject not being done. Bet aging research would go a lot faster if we had the functional equivalent of a billion people working on it, and all billion are informed as to everyone else’s research outcomes.
Where I was going for in the analogy was much simpler. You don’t get a choice. In the immediate term, agreeing to not build machine guns and honoring it means you face a rat tat tat when it matters most. Similar for fission weapons, obviously your enemy is going to build a nuclear arsenal and try to vaporize all your key cities in a surprise attack.
The issues you mention happen long term. In the short term you can use AGI to automate many key tasks and become vastly more economically and militarily powerful.
I think this is a typical LW bias. No, I don’t enjoy the idea of death. But I would rather live a long and reasonably happy life in a human friendly world and then die when I am old than starve to death as one of the 7.9 billion casualties of the AGI Wars. The idea that there’s some sliver of a chance that in some future immortality is on the table for you, personally, is a delusion. I think life extension is very possible, and true immortality is not. But as things are either would only be on the table for, like, the CEOs of the big AI companies who got their biomarkers registered as part of the alignment protocol so that their product obeys them. Not for you. You’re the peasant whose blood, if necessary, cyber-Elizabeth Bathory will use for her rejuvenation rituals.
That’s never happened historically and aging treatments isn’t immortality, it’s just approximately a life expectancy of 10k years. Do you know who is richer than any CEO you name? Medicare. I bet they would like to stop paying all these medical bills, which would be the case if treated patients had the approximate morbidity rate of young adults.
You also need such treatments to be given at large scales to find and correct the edge cases. A rejuvenation treatment “beta tester” is exactly what it sounds, you will have a higher risk of death but get earlier access. Going to need a lot of beta testers.
The rational, data driven belief is that aging is treatable and that ASI systems with the cognitive capacity to take into account more variables than humans are mentally capable of could be built to systematically attack the problem. Doesn’t mean it will help anyone alive today, there are no guarantees. Because automated systems found whatever treatments are possible, automated systems can deliver the same treatments at low cost.
If you don’t think this is a reasonable conclusion, perhaps you could go into your reasoning. Arguments like you made above are unconvincing.
While it is true that certain esoteric treatments for aging like young blood transfusions are inherently limited in who can benefit, they don’t even work that well and de aged hemopoietic stem cells can be generated in automated laboratories and would be a real treatment everyone can benefit.
The wealthy are not powerful enough to “hoard” treatments, because Medicare et al represent the government, which has a monopoly on violence and incentives to not allow such hoarding.
That’s naive. If a private has obedient ASI, they also have a monopoly on violence now. If labour has become superfluous, states have lost all incentive to care about the opinion of people.
I think worlds with the tools to treat most causes of human death ranks strictly higher than a world without those tools. In the same way that a world with running water ranks above worlds without it. Even today not everyone benefits from running water. If you could go back in time would you campaign against developing pipes and pumps because you believed only the rich would ever have running water? (Which was true for a period of time)
I would campaign against lead pipes and support the goths in destroying Rome which likely improved human futures over an alternative of widespread lead piping.
Running water doesn’t create the conditions to permanently disempower almost everyone, AGI does. What I’m talking about isn’t a situation in which initially only the rich benefit but then the tech gets cheaper and trickles down. It’s a permanent trap that destroys democracy and capitalism as we know them.