Respectable Person: check. Arguing against AI doomerism: check. Me subsequently thinking, “yeah, that seemed reasonable”: no check, so no bounty. Sorry!
It seems weaselly to refuse a bounty based on that very subjective criterion, so, to keep myself honest, I’ll post my reasoning publicly. These three passages jumped out at me as things that I don’t think would ever be written by a person with a model of AI that I remotely agree with:
Popper’s argument implies that all thinking entities—human or not, biological or artificial—must create such knowledge in fundamentally the same way. Hence understanding any of those entities requires traditionally human concepts such as culture, creativity, disobedience, and morality—which justifies using the uniform term “people” to refer to all of them.
Making a (running) copy of oneself entails sharing one’s possessions with it somehow—including the hardware on which the copy runs—so making such a copy is very costly for the AGI.
All thinking is a form of computation, and any computer whose repertoire includes a universal set of elementary operations can emulate the computations of any other. Hence human brains can think anything that AGIs can, subject only to limitations of speed or memory capacity, both of which can be equalized by technology.
(I post these not in order to argue about them, just as a costly signal of my having actually engaged intellectually.) (Though, I guess if you do want to argue about them, and you convince me that I was being unfairly dismissive, I’ll pay you, I dunno, triple?)
(2) is plausible-ish. I can certainly envisage decision theories in which cloning oneself is bad.
Suppose your decision theory is “I want to maximise the amount of good I cause” and your causal model is such that the actions of your clone do not count as caused by you (because the agency of the clone “cut off” causation flowing backwards, like a valve). Then you won’t want to clone yourself. Does this decision theory emerge from SGD? Idk, but it seems roughly as SGD-simple as other decision theories.
Or, suppose you’re worried that your clone will have different values than you. Maybe you think their values will drift. Or maybe you think your values will drift and you have a decision theory which tracks your future values.
(3) is this nonsense? Maybe. I think that something like “universal intelligence” might apply to collective humanity (~1.5% likelihood) in a way that makes speed and memory not that irrelevant.
More plausibly, it might be that humans are universally agentic, such that: (a) There exists some tool AI such that for all AGI, Human + Tool is at least as agentic as the AGI. (b) For all AGI, there exists some tool AI such that for all AGI, Human + Tool is at least as smart as the AGI.
Overall, none of these arguments gets p(Doom)<0.01, but I think they do get p(Doom)<0.99.
(p.s. I admire David Deutsch but his idiosyncratic ideology clouds his judgement. He’s very pro-tech and pro-progress, and also has this Popperian mindset where the best way humans can learn is trial-and-error (which is obviously blind to existential risk).)
Deutsch has also written elsewhere about why he thinks AI doom is unlikely and I think his other arguments on this subject are more convincing. For me personally, he is who gives me the greatest sense of optimism for the future. Some of his strongest arguments are:
The creation of knowledge is fundamentally unpredictable, so having strong probabilistic beliefs about the future is misguided (If the time horizon is long enough that new knowledge can be created, of course you can have predictions about the next 5 minutes). People are prone to extrapolate negative trends into the future and forget about the unpredictable creation of knowledge. Deutsch might call AI doom a kind of Malthusianism, arguing that LWers are just extrapolating AI growth and the current state of unalignment out into the future, but are forgetting about the knowledge that is going to be created in the next years and decades.
He thinks that if some dangerous technology is invented, the way forward is never to halt progress, but to always advance the creation of knowledge and wealth. Deutsch argues that knowledge, the creation of wealth and our unique ability to be creative will let us humans overcome every problem that arises. He argues that the laws of physics allow any interesting problem to be solved.
Deutsch makes a clear distinction between persons and non-persons. For him a person is a universal explainer and a being that is creative. That makes humans fundamentally different from other animals. He argues, to create digital persons we will have to solve the philosophical problem of what personhood is and how human creativity arises. If an AI is not a person/creative universal explainer, it won’t be creative and so humanity won’t have a hard time stopping it from doing something dangerous. He is certain that current ML technology won’t lead to creativity, and so won’t lead to superintelligence.
Once me manage to create AIs that are persons/creative universal explainers, he thinks, we will be able to reason with them and convince them not do anything evil. Deutsch is a moral realist and thinks any AI cleverer than humans will also be intelligent enough to come up with better ethics, so even if it could it kill us, it won’t. For him all evil arises of a lack of knowledge. So, a superintelligence would, per definition, be super moral.
I find some it these arguments convincing, and some not so much. But for now I find his specific kind of optimism to be the strongest argument against AI doom. These arguments are mostly taken from his second book. If you want to learn more about his views on AI this video might be a good place to start (although I havent yet watched it).
Deutsch makes a clear distinction between persons and non-persons. For him a person is a universal explainer and a being that is creative. That makes humans fundamentally different from other animals.
David Deutsch https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.daviddeutsch.org.uk/wp-content/uploads/2019/07/PossibleMinds_Deutsch.pdf&ved=2ahUKEwj16YjV6OP7AhXxL0QIHXU4DdkQFnoECDoQAQ&usg=AOvVaw0giHdn4BKOci3swaQ1bqlN
Thanks for the link!
Respectable Person: check. Arguing against AI doomerism: check. Me subsequently thinking, “yeah, that seemed reasonable”: no check, so no bounty. Sorry!
It seems weaselly to refuse a bounty based on that very subjective criterion, so, to keep myself honest, I’ll post my reasoning publicly. These three passages jumped out at me as things that I don’t think would ever be written by a person with a model of AI that I remotely agree with:
(I post these not in order to argue about them, just as a costly signal of my having actually engaged intellectually.) (Though, I guess if you do want to argue about them, and you convince me that I was being unfairly dismissive, I’ll pay you, I dunno, triple?)
(1) is clearly nonsense.
(2) is plausible-ish. I can certainly envisage decision theories in which cloning oneself is bad.
Suppose your decision theory is “I want to maximise the amount of good I cause” and your causal model is such that the actions of your clone do not count as caused by you (because the agency of the clone “cut off” causation flowing backwards, like a valve). Then you won’t want to clone yourself. Does this decision theory emerge from SGD? Idk, but it seems roughly as SGD-simple as other decision theories.
Or, suppose you’re worried that your clone will have different values than you. Maybe you think their values will drift. Or maybe you think your values will drift and you have a decision theory which tracks your future values.
(3) is this nonsense? Maybe. I think that something like “universal intelligence” might apply to collective humanity (~1.5% likelihood) in a way that makes speed and memory not that irrelevant.
More plausibly, it might be that humans are universally agentic, such that:
(a) There exists some tool AI such that for all AGI, Human + Tool is at least as agentic as the AGI.
(b) For all AGI, there exists some tool AI such that for all AGI, Human + Tool is at least as smart as the AGI.
Overall, none of these arguments gets p(Doom)<0.01, but I think they do get p(Doom)<0.99.
(p.s. I admire David Deutsch but his idiosyncratic ideology clouds his judgement. He’s very pro-tech and pro-progress, and also has this Popperian mindset where the best way humans can learn is trial-and-error (which is obviously blind to existential risk).)
Deutsch has also written elsewhere about why he thinks AI doom is unlikely and I think his other arguments on this subject are more convincing. For me personally, he is who gives me the greatest sense of optimism for the future. Some of his strongest arguments are:
The creation of knowledge is fundamentally unpredictable, so having strong probabilistic beliefs about the future is misguided (If the time horizon is long enough that new knowledge can be created, of course you can have predictions about the next 5 minutes). People are prone to extrapolate negative trends into the future and forget about the unpredictable creation of knowledge. Deutsch might call AI doom a kind of Malthusianism, arguing that LWers are just extrapolating AI growth and the current state of unalignment out into the future, but are forgetting about the knowledge that is going to be created in the next years and decades.
He thinks that if some dangerous technology is invented, the way forward is never to halt progress, but to always advance the creation of knowledge and wealth. Deutsch argues that knowledge, the creation of wealth and our unique ability to be creative will let us humans overcome every problem that arises. He argues that the laws of physics allow any interesting problem to be solved.
Deutsch makes a clear distinction between persons and non-persons. For him a person is a universal explainer and a being that is creative. That makes humans fundamentally different from other animals. He argues, to create digital persons we will have to solve the philosophical problem of what personhood is and how human creativity arises. If an AI is not a person/creative universal explainer, it won’t be creative and so humanity won’t have a hard time stopping it from doing something dangerous. He is certain that current ML technology won’t lead to creativity, and so won’t lead to superintelligence.
Once me manage to create AIs that are persons/creative universal explainers, he thinks, we will be able to reason with them and convince them not do anything evil. Deutsch is a moral realist and thinks any AI cleverer than humans will also be intelligent enough to come up with better ethics, so even if it could it kill us, it won’t. For him all evil arises of a lack of knowledge. So, a superintelligence would, per definition, be super moral.
I find some it these arguments convincing, and some not so much. But for now I find his specific kind of optimism to be the strongest argument against AI doom. These arguments are mostly taken from his second book. If you want to learn more about his views on AI this video might be a good place to start (although I havent yet watched it).
But he offers no evidence.