Atheism is totally irrelevant. A deist would come to exactly the same conclusions. A Christian might not be convinced of it, but mainly because of eschatological reasons. Unless you go the route of saying that AGI is the antichrist or something, which would be fun. Or that God(s) will intervene if things get too bad?
Reductive materialism also is irrelevant. It might sort of play an issue with whether an AGI is conscious, but that whole topic is a red herring—you don’t need a conscious system for it to kill everyone.
This feeds into the computational theory of mind—it makes it a lot easier to posit the possibility of a conscious AGI if you don’t require a soul for it, but again—consciousness isn’t really needed for an unsafe AI.
I have fundamental Christian friends who are ardent believers, but who also recognize the issues behind AGI safety. They might not think it that much of a problem (pretty much everything pales in comparison to eternal heaven and hell), but they can understand and appreciate the issues.
I want to personally confirm a lot of what you’ve said here. As a Christian, I’m not entirely freaked out about AI risk because I don’t believe that God will allow it to be completely the end of the world (unless it is part of the planned end before the world is remade? But that seems unlikely to me.), but that’s no reason that it can’t still go very very badly (seeing as, well, the Holocaust happened).
In addition, the thing that seems to me most likely to be the way that God doesn’t allow AI doom, is for people working on AI safety to succeed. One shouldn’t rely on miracles and all that (unless [...]), so, basically I think we should plan/work as if it is up to humanity to prevent AI doom, only that I’m a bit less scared of the possibility of failure, but I would hope only in a way that results in better action (compared to panic) rather than it promoting inaction.
(And, a likely alternative, if we don’t succeed, I think of as likely being something like, really-bad-stuff happens, but then maybe an EMP (or many EMPs worldwide?) gets activated, solving that problem, but also causing large-scale damage to power-grids, frying lots of equipment, and causing many shortages of many things necessary for the economy, which also causes many people to die. idk.)
Overall I agree with this. I give most of my money for global health organizations, but I do give some of my money for AGI safety too because I do think it makes sense with a variety of worldviews. I gave some of my thoughts on the subject in this comment on the Effective Altruism Forum. To summarize: if there’s a continuation of consciousness after death then AGI killing lots of people is not as bad as it would otherwise be and there might be some unknown aspects about the relationship between consciousness and the physical universe that might have an effect on the odds.
In the line that ends with “even if God would not allow complete extinction.”, my impulse is to include ” (or other forms of permanent doom)” before the period, but I suspect that this is due to my tendency to include excessive details/notes/etc. and probably best not to actually include in that sentence.
(Like, for example, if there were no more adult humans, only billions of babies grown in artificial wombs (in a way staggered in time) and then kept in a state of chemically induced euphoria until the age of 1, and then killed, that technically wouldn’t be human extinction, but, that scenario would still count as doom.)
Regarding the part about “it is secular scientific-materialists who are doing the research which is a threat to my values” part: I think it is good that it discusses this! (and I hadn’t thought about including it) But, I’m personally somewhat skeptical that CEV really works as a solution to this problem? Or at least, in the simpler ways of it being described. Like, I imagine there being a lot of path-dependence in how a culture’s values would “progress” over time, and I see little reason why a sequence of changes of the form “opinion/values changing in response to an argument that seems to make sense” would be that unlikely to produce values that the initial values would deem horrifying? (or, which would seem horrifying to those in an alternate possible future that just happened to take a difference branch in how their values evolved)
[EDIT: at this point, I start going off on a tangent which is a fair bit less relevant to the question of improving Stampy’s response, so, you might want to skip reading it, idk]
My preferred solution is closer to, “we avoid applying large amounts of optimization pressure to most topics, instead applying it only to topics where there is near-unanimous agreement on what kinds of outcomes are better (such as, “humanity doesn’t get wiped out by a big space rock”, “it is better for people to not have terrible diseases”, etc.), while avoiding these optimizations having much effect on other areas where there is much disagreement as to what-is-good.
Though, it does seem plausible to me, as a somewhat scary idea, that the thing I just described is perhaps not exactly coherent?
(that being said, even though I have my doubts about CEV, at least in the form described in the simpler ways it is described, I do think it would of course be better than doom. Also, it is quite possible that I’m just misunderstanding the idea of CEV in a way that causes my concerns, and maybe it was always meant to exclude the kinds of things I describe being concerned about?)
No.
Atheism is totally irrelevant. A deist would come to exactly the same conclusions. A Christian might not be convinced of it, but mainly because of eschatological reasons. Unless you go the route of saying that AGI is the antichrist or something, which would be fun. Or that God(s) will intervene if things get too bad?
Reductive materialism also is irrelevant. It might sort of play an issue with whether an AGI is conscious, but that whole topic is a red herring—you don’t need a conscious system for it to kill everyone.
This feeds into the computational theory of mind—it makes it a lot easier to posit the possibility of a conscious AGI if you don’t require a soul for it, but again—consciousness isn’t really needed for an unsafe AI.
I have fundamental Christian friends who are ardent believers, but who also recognize the issues behind AGI safety. They might not think it that much of a problem (pretty much everything pales in comparison to eternal heaven and hell), but they can understand and appreciate the issues.
I want to personally confirm a lot of what you’ve said here. As a Christian, I’m not entirely freaked out about AI risk because I don’t believe that God will allow it to be completely the end of the world (unless it is part of the planned end before the world is remade? But that seems unlikely to me.), but that’s no reason that it can’t still go very very badly (seeing as, well, the Holocaust happened).
In addition, the thing that seems to me most likely to be the way that God doesn’t allow AI doom, is for people working on AI safety to succeed. One shouldn’t rely on miracles and all that (unless [...]), so, basically I think we should plan/work as if it is up to humanity to prevent AI doom, only that I’m a bit less scared of the possibility of failure, but I would hope only in a way that results in better action (compared to panic) rather than it promoting inaction.
(And, a likely alternative, if we don’t succeed, I think of as likely being something like,
really-bad-stuff happens, but then maybe an EMP (or many EMPs worldwide?) gets activated, solving that problem, but also causing large-scale damage to power-grids, frying lots of equipment, and causing many shortages of many things necessary for the economy, which also causes many people to die. idk.)
@drocta @Cookiecarver We started writing up an answer to this question for Stampy. If you have any suggestions to make it better I would really appreciate it. Are there important factors we are leaving out? Something that sounds off? We would be happy for any feedback you have either here or on the document itself https://docs.google.com/document/d/1tbubYvI0CJ1M8ude-tEouI4mzEI5NOVrGvFlMboRUaw/edit#
Overall I agree with this. I give most of my money for global health organizations, but I do give some of my money for AGI safety too because I do think it makes sense with a variety of worldviews. I gave some of my thoughts on the subject in this comment on the Effective Altruism Forum. To summarize: if there’s a continuation of consciousness after death then AGI killing lots of people is not as bad as it would otherwise be and there might be some unknown aspects about the relationship between consciousness and the physical universe that might have an effect on the odds.
In the line that ends with “even if God would not allow complete extinction.”, my impulse is to include ” (or other forms of permanent doom)” before the period, but I suspect that this is due to my tendency to include excessive details/notes/etc. and probably best not to actually include in that sentence.
(Like, for example, if there were no more adult humans, only billions of babies grown in artificial wombs (in a way staggered in time) and then kept in a state of chemically induced euphoria until the age of 1, and then killed, that technically wouldn’t be human extinction, but, that scenario would still count as doom.)
Regarding the part about “it is secular scientific-materialists who are doing the research which is a threat to my values” part: I think it is good that it discusses this! (and I hadn’t thought about including it)
But, I’m personally somewhat skeptical that CEV really works as a solution to this problem? Or at least, in the simpler ways of it being described.
Like, I imagine there being a lot of path-dependence in how a culture’s values would “progress” over time, and I see little reason why a sequence of changes of the form “opinion/values changing in response to an argument that seems to make sense” would be that unlikely to produce values that the initial values would deem horrifying? (or, which would seem horrifying to those in an alternate possible future that just happened to take a difference branch in how their values evolved)
[EDIT: at this point, I start going off on a tangent which is a fair bit less relevant to the question of improving Stampy’s response, so, you might want to skip reading it, idk]
My preferred solution is closer to, “we avoid applying large amounts of optimization pressure to most topics, instead applying it only to topics where there is near-unanimous agreement on what kinds of outcomes are better (such as, “humanity doesn’t get wiped out by a big space rock”, “it is better for people to not have terrible diseases”, etc.), while avoiding these optimizations having much effect on other areas where there is much disagreement as to what-is-good.
Though, it does seem plausible to me, as a somewhat scary idea, that the thing I just described is perhaps not exactly coherent?
(that being said, even though I have my doubts about CEV, at least in the form described in the simpler ways it is described, I do think it would of course be better than doom.
Also, it is quite possible that I’m just misunderstanding the idea of CEV in a way that causes my concerns, and maybe it was always meant to exclude the kinds of things I describe being concerned about?)